XGBoost — Kaggle জেতার অস্ত্র

XGBoost-এর mathematical heart — second-order optimization। Practical impact বহুমুখী।

First-order vs second-order:

(১) GBM (first-order):

• $F_{t+1} = F_t - \eta \nabla L$।
• Linear approximation।
• Step size $\eta$ — fixed, manual।
• Slow convergence in curved regions।

(২) XGBoost (second-order):

• $\Delta F = -G/(H+\lambda)$।
• Quadratic approximation।
• Step size — adaptive, automatic।
• Newton's method analog।

Hessian-এর role:

• Loss curvature — কত fast bend।
• Flat region (low H) — large step OK।
• Sharp region (high H) — small step careful।
• Adaptive step — overshoot prevent।

Impact-1: Faster convergence:

• Newton's method — quadratic convergence।
• Gradient descent — linear convergence।
• Practice — 30-50% fewer iterations।
• Less computation, better accuracy।

Impact-2: Better split finding:

• Gain formula — Hessian-aware।
• Sample weight per Hessian — different importance।
• Important samples — more precise splits।
• Decision boundary sharper।

Impact-3: Loss-agnostic framework:

• Custom loss — provide gradient + hessian।
• Same engine works।
• Quantile regression, Tweedie, Poisson — extensions easy।

Impact-4: Numerical stability:

• $\lambda$ in denominator — divide-by-zero prevention।
• Regularization built-in।
• Production reliable।

Hessian for common losses:

• Squared loss: $g = \hat{y} - y$, $H = 1$ (constant)।
• Log loss: $g = p - y$, $H = p(1-p)$।
• Multi-class: diagonal Hessian per class।

Constant Hessian implication:

• Squared loss-এ $H = 1$ — XGBoost reduces to GBM-like।
• $w^* = -G/(n+\lambda)$ — mean residual / regularized।
• Slight improvement over GBM-এর mean।

Variable Hessian implication (log loss):

• $H = p(1-p)$ — boundary near uncertain samples max।
• Confident samples — low Hessian — discounted।
• Boundary samples — high Hessian — leverage বেশি।
• Decision boundary refinement smoother।

Trade-offs:

• Hessian computation — extra compute (negligible)।
• Hessian storage — memory cost।
• Custom loss — Hessian derivation harder।

When second-order doesn't help:

• Highly non-convex loss।
• Hessian unstable (extreme values)।
• $\lambda$ misspecified।

Connection to other methods:

• L-BFGS — quasi-Newton for parameter optimization।
• Natural Gradient — Fisher information-based।
• K-FAC, Shampoo — modern NN second-order।
• Common theme: curvature-aware updates।

Empirical evidence:

• Chen & Guestrin (২০১৬) paper benchmarks।
• Higgs Boson — 30% faster, 2% better accuracy।
• YearPredictMSD — 20% improvement।
• Consistent gains across domains।

মূল উপলব্ধি: Second-order — convex optimization-এর gold standard। XGBoost — boosting-এ first principled application। Math + engineering = dominance।

XGBoost — algorithm + system co-design। Engineering equally critical।

(১) Sparsity-aware split finding:

• Real-world data — sparse (one-hot, missing)।
• Naive: zero treat as value — incorrect।
• XGBoost: missing/zero default direction learn।
• Best gain choice — automatic।
• Memory savings — sparse storage।
• 50× speedup on sparse data।

(২) Histogram-based split:

• Original: প্রতি unique value-এ split try।
• Continuous feature — millions split candidates।
• Histogram: feature values bucket (256 default)।
• Per-bucket gradient sum precompute।
• Split candidates — bucket boundary।
• Approximate but fast (10-100×)।

(৩) Column-block parallel:

• Sort-based split — pre-sort columns।
• Each column — independent block।
• Multi-thread — parallel split search।
• Reuse across iterations।
• Linear scaling with cores।

(৪) Cache-aware access:

• Modern CPU cache hierarchy।
• Sequential access — fast।
• Random access — slow।
• Block size tuned to cache।
• Cache miss minimize।

(৫) Out-of-core learning:

• Data > RAM scenarios।
• Block sharding to disk।
• Streaming reads।
• Sparsity-aware compression।
• 10-100GB datasets feasible।

(৬) Approximate split finding:

• Local — within node।
• Global — once, used everywhere।
• Speed-accuracy tradeoff।

(৭) GPU support:

• 2017+ — full GPU acceleration।
• Histogram on GPU।
• 10-50× speedup।
• Multi-GPU distributed।

Comparison with competitors:

• LightGBM: additional histogram tricks (GOSS, EFB)।
• CatBoost: ordered boosting (different angle)।
• sklearn HistGBM: XGBoost-inspired।

Real-world impact:

• Higgs ১১M samples — 30 minutes (XGBoost) vs hours (sklearn GBM)।
• KDDCup ২০১৫ winner — XGBoost critical।
• Production billions samples — feasible।

মূল উপলব্ধি: "Engineering matters as much as math" — Tianqi Chen's philosophy। Modern ML — system + algorithm co-design।

XGBoost tuning — production ML-এর recurring challenge। Strategy choice critical।

Hyperparameter landscape:

• ৬-৮টি critical knobs।
• Continuous + discrete mix।
• Interactions exist (max_depth × learning_rate)।
• Search space ১০⁶+।

(১) Manual tuning:

Approach:

• Start with defaults।
• One-at-a-time vary।
• Domain expert intuition।
• Iterative refinement।

Pros:

• Fast (few experiments)।
• Interpretable progress।
• Debug-friendly।

Cons:

• Subjective।
• Local optimum trap।
• Missing interactions।
• Inconsistent across people।

(২) Grid Search:

Approach: All combinations exhaustive।

Pros: Reproducible, complete coverage।

Cons: Curse of dimensionality (5 params × 5 values = 3125 runs)।

(৩) Random Search:

Approach: Random combinations sample।

Pros: Bergstra & Bengio — comparable to grid in less time।

Cons: No learning from history।

(৪) Bayesian Optimization:

Approach:

• Surrogate model (GP) — performance predict।
• Acquisition function — next point pick।
• History from previous runs।
• Sequential exploration-exploitation।

Pros:

• Sample-efficient — fewer runs।
• Handles continuous gracefully।
• Theoretically grounded।

Cons:

• Sequential — slow for parallel hardware।
• Setup complexity।
• GP hyperparameter own।

(৫) Hyperband / BOHB:

Approach: Multi-fidelity — bad early kill, good train more।

Pros: 10-100× speedup over random।

Cons: Implementation complexity।

(৬) Population Based Training (PBT):

Approach: Genetic-like, online during training।

Pros: No retrain, schedule discover।

Cons: Compute-heavy।

Production recommendation:

• (১) Manual + sensible defaults: baseline ২ hours।
• (২) Random Search: ৫০-১০০ runs broad coverage।
• (৩) Bayesian (Optuna): ১০০-৫০০ runs refinement।
• (৪) Final manual sanity check।

Tools:

• Optuna: Bayesian, easy use, popular।
• Hyperopt: classic Bayesian।
• Ray Tune: distributed, flexible।
• scikit-optimize: simple, sklearn-like।

Pitfall-1: Validation contamination:

• Same validation hyperparameter selection।
• Over-fit to validation।
• Solution: nested CV, separate test set।

Pitfall-2: Tuning before features:

• Wasted effort if features change।
• Solution: feature engineering first, then tune।

Pitfall-3: Ignoring early stopping:

• $n\_estimators$ tune — wasteful।
• Solution: early stopping always।

Pitfall-4: Single seed:

• Variance across seeds — noisy comparison।
• Solution: multi-seed averaging।

Pitfall-5: Local optimum:

• Manual tuning — myopic।
• Solution: global search method।

Pitfall-6: Compute budget mismatch:

• Too few runs — wrong answer।
• Too many — wasted compute।
• Solution: convergence monitoring।

Pitfall-7: Default search space:

• Generic ranges — domain miss।
• Solution: domain prior knowledge integrate।

Practical workflow:

import optuna
def objective(trial):
    params = {
        'max_depth': trial.suggest_int('max_depth', 3, 10),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'subsample': trial.suggest_float('subsample', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0),
        'reg_lambda': trial.suggest_float('reg_lambda', 0.01, 10, log=True),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
    }
    model = xgb.XGBClassifier(n_estimators=2000, **params,
                              early_stopping_rounds=50, random_state=0)
    model.fit(Xt, yt, eval_set=[(Xv, yv)], verbose=False)
    return model.best_score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=200)
print(study.best_params)

Bangladesh production:

• Compute budget often limited।
• Quick win: defaults + early stopping।
• Long-term: Optuna pipeline।
• Track all experiments (MLflow)।

মূল উপলব্ধি: Tuning — art + science। Automated Bayesian optimization gold standard। Manual baseline + automated refinement — production sweet spot।

XGBoost dominance peak ২০১৬-২০১৮। Modern landscape diverse, nuanced।

২০১৪-২০১৮ — XGBoost age:

• ২০১৫-এ Kaggle competitions ৬০%+ XGBoost-based।
• "Just XGBoost it" meme।
• Ensemble of XGBoost variants।
• Higgs Boson, KDDCup wins।

২০১৭+ — LightGBM challenger:

• Microsoft (২০১৭) — Histogram speedup further।
• GOSS — gradient sampling।
• EFB — exclusive feature bundling।
• Leaf-wise growth।
• 10× faster, similar accuracy।
• Big data শক্তি।

২০১৮+ — CatBoost specialist:

• Yandex (২০১৭) — categorical native।
• Ordered boosting — leak-free।
• Symmetric trees — fast inference।
• Categorical-heavy data — winner।

২০১৯+ — Deep tabular:

• TabNet (Google ২০২০) — attention-based।
• NODE (Yandex) — neural oblivious decision ensemble।
• SAINT — self-attention।
• FT-Transformer — feature tokenization।
• TabPFN (২০২৩) — meta-learning prior।

২০২৪ benchmark realities:

(১) Kaggle 2023 winners:

• ৪০% LightGBM।
• ৩০% XGBoost।
• ২০% CatBoost।
• ১০% deep tabular (mainly hybrids)।
• "GBDT family" still dominant।

(২) Production landscape:

• FAANG — XGBoost/LightGBM mainstream।
• Russia (Yandex, Sber) — CatBoost preferred।
• Startups — practical: any GBDT।
• Research — deep tabular emerging।

(৩) Performance comparison:

• Small data (<10K): XGBoost wins।
• Mid (10K-1M): LightGBM wins (speed)।
• Categorical-heavy: CatBoost wins।
• Large (1M+): LightGBM / GPU XGBoost।
• Specialty cases: Deep tabular spot strength।

XGBoost current strengths:

• Maturity: 10-year battle-tested।
• Documentation: extensive।
• Bug-free: stable production।
• Cross-platform: Python, R, Java, Scala।
• Distributed: Spark/Flink integration।
• Educational: reference implementation।

XGBoost current weaknesses:

• Speed: LightGBM ahead।
• Categorical: CatBoost ahead।
• Memory: heavy compared to LightGBM।
• GPU efficiency: RAPIDS RF often faster।

Deep tabular reality check:

• Benchmark — Grinsztajn et al. (২০২২)।
• ৪৫টি tabular dataset।
• GBDT consistently competitive বা better।
• Deep tabular — specific niches।
• Reasons: tree-friendly geometry, robustness, sample efficiency।

When to use what:

• XGBoost: default, mature pipeline, mid-data।
• LightGBM: speed critical, big data।
• CatBoost: categorical-heavy, less tuning।
• TabPFN: very small data (<1000)।
• TabNet/SAINT: interpretability + tabular DL preferred।

Future trends:

• (১) Foundation models for tabular।
• (২) Multi-modal (tabular + text + image)।
• (৩) AutoML standard।
• (৪) Hardware acceleration (TPU)।
• (৫) Causal inference integration।

Bangladesh context:

• Data sizes typical small-medium।
• Skill availability — XGBoost most documented।
• Production deployment — XGBoost mature।
• Future: LightGBM growing share।

মূল উপলব্ধি: XGBoost still excellent, but landscape diverse। Tabular ML mature field — multiple tools each strength। Choose by use case, not hype।