পাঠ ২৫ · ৪৫-এর মধ্যে · মডিউল ৩

Home / AI Courses / Machine Learning / LightGBM & CatBoost

LightGBM ও CatBoost

LightGBM & CatBoost — modern boosting

৭ মিনিট পড়া মাঝারি · Intermediate lightgbm + catboost

এই পাঠে যা শিখবেন

LightGBM-এর তিন signature feature — GOSS, EFB, leaf-wise growth
CatBoost-এর ordered boosting — target encoding leak প্রতিরোধ
Symmetric (oblivious) trees — fast inference
XGBoost vs LightGBM vs CatBoost — কখন কোনটি
Practical setup ও hyperparameter best practice

১ · LightGBM — Microsoft-এর speed champion

Ke et al. (২০১৭) — XGBoost-এর histogram approach আরও push। Three signature contributions:

(ক) Leaf-wise growth

XGBoost: level-wise — সব leaves একই depth-এ grow।
LightGBM: leaf-wiseLeaf-wise Growthপ্রতিবার সব leaves-এর মধ্যে যেটির gain সর্বোচ্চ — সেটি split। অসমভাবে grow হয়। Symmetric tree-এর তুলনায় deeper, কিন্তু accuracy বেশি। সমস্যা — small data-এ overfit prone। — সর্বোচ্চ gain leaf split।
Asymmetric tree — কিছু path গভীর, কিছু shallow।
Same number of leaves → leaf-wise lower training loss।
Trade-off: small data-এ overfit prone।

(খ) GOSS — Gradient-based One-Side Sampling

Idea: large-gradient samples — under-trained, important।
Small-gradient samples — well-trained, less informative।
Top $a$% large-gradient samples রাখুন; bottom $(1-a)$% থেকে random $b$% নিন।
Gain calculation-এ — small-gradient sample-গুলোকে $(1-a)/b$ multiplier দিয়ে scale up।
Distribution unbiased preserve, কম sample।
2-3× speedup with minimal accuracy loss।

(গ) EFB — Exclusive Feature Bundling

Sparse features (one-hot)-এ — অনেক features একসাথে non-zero না।
Conflict-free features bundle করে ১টি feature-এ merge।
$O(n \cdot d) \to O(n \cdot d_{\text{bundle}})$।
NLP, recommender — ১০-১০০× feature reduction।

২ · CatBoost — Yandex-এর categorical king

Prokhorenkova et al. (২০১৭/২০১৮) — categorical-heavy data-এর জন্য targeted। Two signature contributions:

(ক) Ordered Boosting — target leak prevention

Standard target encoding — category-এর mean target replace। সমস্যা: training-এ target-এর data leak — overfit।

Ordered boostingOrdered BoostingCatBoost-এর leak-free target encoding scheme। Random permutation, প্রতি sample-এর জন্য — শুধু "past" samples-এর statistics। Online learning analog।:

Random permutation $\sigma$ of training samples।
প্রতি sample $i$-এর জন্য — শুধু $\sigma(j) < \sigma(i)$ samples-এ statistics calculate।
Test-এ — পুরো training set-এ statistics।
Multiple permutations — ensemble diversity।

Result — categorical encoding-এ data leak-free, native handling। NB ও fintech-এ অসাধারণ।

(খ) Symmetric (oblivious) trees

প্রতি depth-এ — সব node-এ একই split (feature, threshold)।
Tree balanced — leaf indexing simple।
Inference: bit operations — extremely fast।
Less expressive (per-tree) — তবু overfit-resistant।
Embedded device deploy ideal।

৩ · Comparison — কখন কোনটি

দিক	XGBoost	LightGBM	CatBoost
Speed	Medium	Fastest	Medium-Slow
Memory	High	Lowest	Medium
Categorical	One-hot/encode	Native (basic)	Native (best)
Tuning needed	High	Medium	Low
Small data	Good	Overfit prone	Robust
Large data	Slow	Excellent	Good
Ecosystem	Largest	Large	Smaller

Rule of thumb:

Tabular data, mid-size — LightGBM default।
Categorical heavy (e-commerce, NLP categorical) — CatBoost।
Mature pipeline, small data — XGBoost।
Very large (১০M+) — LightGBM / Distributed।

Level-wise (XGBoost) — depth-balanced, robust। Leaf-wise (LightGBM) — gain-greedy, faster convergence কিন্তু overfit prone। num_leaves careful tune লাগে।

৪ · Python — LightGBM

Python · lightgbm

import lightgbm as lgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
Xt, Xv, yt, yv = train_test_split(X, y, test_size=0.3, random_state=0)

model = lgb.LGBMClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    num_leaves=31,            # 2^max_depth-এর বদলে
    max_depth=-1,             # unlimited (num_leaves cap)
    min_child_samples=20,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_lambda=1.0,
    random_state=0,
    n_jobs=-1
)

model.fit(Xt, yt,
          eval_set=[(Xv, yv)],
          callbacks=[lgb.early_stopping(50), lgb.log_evaluation(0)])

print(f"Best iter: {model.best_iteration_}")
print(f"Val acc:   {model.score(Xv, yv):.4f}")

num_leaves — LightGBM-এর primary complexity knob (max_depth-এর বদলে)। Default ৩১। ছোট data-এ ৭-১৫ better।

৫ · Python — CatBoost

Python · catboost

from catboost import CatBoostClassifier
from sklearn.model_selection import train_test_split
import pandas as pd

# Toy categorical-heavy dataset
import numpy as np
np.random.seed(0)
n = 1000
df = pd.DataFrame({
    'district': np.random.choice(['Dhaka', 'Chittagong', 'Sylhet', 'Rajshahi'], n),
    'product':  np.random.choice(['phone', 'tablet', 'laptop', 'tv'], n),
    'age':      np.random.randint(18, 70, n),
    'income':   np.random.randint(10000, 100000, n),
})
y = (df['income'] / 1000 + (df['district'] == 'Dhaka') * 5
     + np.random.randn(n) * 3 > 30).astype(int)

cat_cols = ['district', 'product']
Xt, Xv, yt, yv = train_test_split(df, y, test_size=0.3, random_state=0)

model = CatBoostClassifier(
    iterations=1000,
    learning_rate=0.05,
    depth=6,
    cat_features=cat_cols,
    early_stopping_rounds=50,
    verbose=0,
    random_state=0
)
model.fit(Xt, yt, eval_set=(Xv, yv))
print(f"Val acc: {model.score(Xv, yv):.4f}")

cat_features — categorical column names পাস করেই হয়ে গেল। One-hot, label encoding — কিছুই দরকার নেই। Internal-এ ordered target statistics।

৬ · GOSS — গাণিতিক explanation

Naïve sampling — small-gradient samples drop করলে distribution biased হয়। GOSS-এর fix:

Samples-কে |gradient| অনুযায়ী sort।
Top $a \cdot n$ samples (large gradient) — সব রাখা।
বাকি $(1-a) \cdot n$ থেকে — random $b \cdot (1-a) \cdot n$ samples নেওয়া।
Gain calculation-এ — bottom samples-এর contribution $(1-a)/b$ multiplier দিয়ে scale up।

Result: split decision unbiased, computation $a + b(1-a)$ fraction।

৭ · CatBoost — categorical encoding-এর "leak" সমস্যা

Standard target encoding:

$$\hat{x}_{i,c} = \mathbb{E}[y \mid x_c = \text{category}_i]$$

Training-এ — sample-এর own target উপস্থিত mean-এ। Test-এ এটা available নয় → train-test mismatch, overfit।

CatBoost solution — ordered:

Permutation $\sigma$ — random ordering।
Sample $i$-এর encoding — শুধু $\sigma(j) < \sigma(i)$ samples-এ statistics।
Multiple permutations → ensemble।
"Online learning" analog — past-only।

No leak, accurate, native categorical। CatBoost-এর USP।

৮ · কোথায় এগুলো fail

LightGBM:

Small data (<১০K) — leaf-wise overfit prone।
num_leaves না tune করলে — accuracy খারাপ।
Categorical native সাপোর্ট basic — high-cardinality-এ CatBoost।

CatBoost:

Numeric-only data — XGBoost/LightGBM আরো faster।
Memory consumption বেশি (multiple permutation)।
Smaller community — debug সহায়তা কম।

Production-এ — তিনটিই baseline-এ run করুন, validation score compare। "Best on paper" ≠ "best on your data"। Empirical decision।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ Leaf-wise growth XGBoost-এর তুলনায় faster convergence — কেন? এর dark side কী, এবং practitioner কীভাবে তা mitigate করেন?

Leaf-wise growth — LightGBM-এর genius ও pitfall দুটোই।

Level-wise (XGBoost):

প্রতি depth-এ — সব leaves একসাথে split।
"Breadth-first" expansion।
Symmetric tree — easy parallelize।
Fixed maximum depth।

Leaf-wise (LightGBM):

সব leaves-এর gain compute।
সর্বোচ্চ gain-এর leaf — split।
"Best-first" expansion।
Asymmetric tree — কিছু path long, কিছু short।

Why faster convergence:

(১) Loss reduction maximize:

Same number of nodes — leaf-wise নিজে বাছে best location।
Level-wise — wasteful splits (low-gain leaves)।
Per node — leaf-wise efficient।

(২) Adaptive depth:

Complex pattern area — deep grow।
Simple area — shallow।
Resource focused।

(৩) Empirical observation:

Same num_leaves — leaf-wise lower training loss।
Same accuracy — fewer trees।
2-3× speedup typical।

Dark side — overfitting:

(১) Deep narrow paths:

Leaf-wise — depth 30+ একটি path।
৫টি sample isolation।
Memorization easy।

(২) Small data sensitive:

$n < 10000$ — leaf-wise overfit prone।
XGBoost level-wise robust।

(৩) Validation crucial:

Default num_leaves=31 — many cases overfit।
Careful tuning required।

Mitigation strategies:

(১) num_leaves limit:

Default 31 — moderate data ভাল।
Small data: 7-15।
Large data: 63-127।
Rule: $2^{\text{max\_depth}} - 1$।

(২) max_depth সাহায্য:

Default -1 (unlimited)।
5-10 set — runaway growth prevent।
num_leaves + max_depth দু'টোই control।

(৩) min_child_samples:

Leaf-এ minimum samples।
Default 20।
Small data: 5-10।
Noise robustness।

(৪) Regularization:

reg_alpha (L1), reg_lambda (L2)।
min_split_gain — minimum split improvement।

(৫) Subsample + colsample:

Stochastic boost — diversity।
0.7-0.9 typical।

(৬) Early stopping:

50-100 patience।
Validation U-shape catch।

(৭) Cross-validation:

K-fold reliable evaluation।
Single split misleading।

When level-wise (XGBoost) better:

Small data (<5000)।
High-noise labels।
Less hyperparameter expertise।
Robust default needed।

Hybrid solutions:

LightGBM with tight num_leaves।
Mimic level-wise behavior।
Best of both — small data robust + leaf-wise speed।

মূল উপলব্ধি: Algorithm choice ≠ default trust। Leaf-wise — power, careful tune required। Documentation read, validation rigorous।

প্র ০২ Target encoding-এ leak সমস্যা — সহজ ভাষায় কী, কেন CatBoost-এর ordered approach এত বুদ্ধিমান, এবং অন্য encoding-এর সাথে comparison?

Target encoding leak — production ML-এর silent killer। CatBoost-এর ordered approach elegant solution।

Target encoding basic:

Categorical → mean target।
"Dhaka district" → 0.45 (default rate)।
"Sylhet district" → 0.20।
Information-rich encoding।

The leak problem:

Train sample i — Dhaka, target=1।
Encoding "Dhaka" mean — i-এর target included।
Self-referential!
Training accuracy artificially high।
Test-এ — generalization fail।

Severity:

Rare categories (low count) — extreme leak।
Single sample category → encoding = target।
Tree memorize trivially।
Validation pessimistic।

Existing solutions:

(১) Out-of-fold (CV) encoding:

K-fold split data।
Per-fold encoding — other folds-এ statistics।
Standard practice (Kaggle)।
Computationally expensive।

(২) Smoothing:

Bayesian — prior + likelihood।
$\hat{x} = \frac{n \cdot m + \alpha \cdot \text{global}}{n + \alpha}$।
Rare category — global toward।
Leak reduce, not eliminate।

(৩) Leave-one-out:

Per-sample — সবাই except self mean।
Naive O(n²)।
Trick: total sum minus self।
One pass possible।

CatBoost ordered approach:

(১) Random permutation:

Training samples shuffle।
Index-এ time order impose।

(২) Past-only encoding:

Sample i — শুধু $j < i$ permutation-এ statistics।
"Future" data ignore।
Online learning analog।

(৩) Multiple permutations:

Different orderings।
Per-permutation encoding ভিন্ন।
Ensemble robustness।
Order bias mitigated।

Why elegant:

Mathematically — leak proof।
Empirically — best categorical handling।
No CV overhead।
Native ordered boosting integration।

Comparison with one-hot:

One-hot — high dimensional, sparse।
Tree split inefficient।
High cardinality — explosion।
CatBoost ordered — single feature, dense।

Comparison with label encoding:

Label — arbitrary integer (Dhaka=0, Sylhet=1)।
Tree splits — meaningless ordering।
"district < 2" — odd condition।
CatBoost — semantically meaningful encoding।

Comparison with embedding:

Embedding — learned dense vector।
Powerful but data-hungry।
NN-based, expensive।
CatBoost — tree-friendly alternative।

Practical performance:

High-cardinality (1000+ categories) — CatBoost wins।
Low-cardinality (<10) — one-hot competitive।
Mid (50-500) — sweet spot CatBoost।

Bangladesh use cases:

E-commerce: product_id (10K+ categories)।
Fintech: merchant_id (1M+)।
Telecom: phone_brand (500+)।
Healthcare: ICD codes।
সব CatBoost ideal।

Edge cases:

Time-ordered data — natural permutation।
Geographic — careful with leakage।
Tiny data — CatBoost overkill।

মূল উপলব্ধি: Target encoding leak — silent feature engineering pitfall। CatBoost ordered — production-grade solution। Other libraries-এ — manual CV-based encoding still viable।

প্র ০৩ একজন ML engineer-এর জন্য — XGBoost vs LightGBM vs CatBoost — কোন approach সবার আগে শেখা উচিত? Career-perspective।

ML career — tools choice strategic। Boosting library mastery — practical impact বিশাল।

Recommendation: XGBoost first, then LightGBM:

(১) XGBoost — pedagogical advantage:

Most teaching material — XGBoost first।
Documentation extensive — Chinese, English।
Concept clarity — ground truth।
Mature, stable।

(২) Career market:

Job postings — XGBoost mention 70%+।
"XGBoost experience" common requirement।
Bangladesh market — XGBoost dominant।

(৩) Concept transferability:

XGBoost master — LightGBM/CatBoost easy।
Underlying GBM theory — same।
Hyperparameter intuition — transferable।

(৪) Production readiness:

XGBoost — Kafka/Spark/Flink integration।
Multi-language deployment।
Battle-tested।

Learning path suggestion:

Phase 1 (১-২ মাস) — XGBoost foundation:

Concept understanding — math, regularization, hyperparameter।
10 Kaggle competitions — practice।
Production scenarios।

Phase 2 (২ সপ্তাহ) — LightGBM transition:

Speed advantages explore।
num_leaves tuning learn।
GOSS, EFB concepts।

Phase 3 (১ সপ্তাহ) — CatBoost niche:

Categorical-heavy data।
Ordered boosting concept।
When prefer over alternatives।

Phase 4 (continuous) — modern landscape:

HistGradientBoosting (sklearn)।
Optuna AutoML।
NGBoost, TabPFN।

Career tracks:

(১) Generalist ML engineer:

XGBoost + LightGBM dual।
Tabular data dominant skill।
Bangladesh market most common।

(২) Kaggle competitor:

All three master।
Ensemble of multiple।
Time intensive but high reward।

(৩) Big data engineer:

LightGBM + Spark integration।
Distributed training।
Scale focus।

(৪) E-commerce/recommendation:

CatBoost emphasis।
Categorical handling crucial।
Yandex/JD/Alibaba style।

(৫) Research:

All three benchmark।
Custom modifications।
Cutting edge keep up।

Common mistake:

"Latest = best" — false।
One library deep > many shallow।
Theory > library specifics।

Skills beyond library:

Feature engineering: 50% of ML success।
Validation: CV, time-series splits।
Hyperparameter: Bayesian optimization।
Production: deployment, monitoring।
SHAP: interpretation।
Domain: business context।

Bangladesh job market:

Top requirement: XGBoost + scikit-learn।
Bonus: LightGBM, CatBoost।
Senior: tuning, deployment, MLOps।
Lead: business impact, team mentoring।

Learning resources:

XGBoost docs — well-structured।
Kaggle notebooks — practical।
Optuna documentation — tuning।
SHAP book — interpretation।

মূল উপলব্ধি: XGBoost foundation, LightGBM modern, CatBoost niche। Concept master — library variations easy। Career — depth + breadth balance।

প্র ০৪ আপনি Daraz-এর জন্য customer churn model বানাচ্ছেন — ১০ million customers, ৫০টি features (২০টি high-cardinality categorical)। কোন boosting library ও কেন?

Real-world e-commerce ML — boosting choice strategic। Daraz scenario typical edge cases।

Problem characteristics:

Scale: ১০M customers — large।
Features: ৫০টি — moderate dimensionality।
Categorical: ২০টি high-cardinality (district 8, product 100K, brand 5K)।
Numerical: ৩০টি (purchase frequency, amount, time)।
Imbalance: churn rate 10-20%।
Deployment: daily scoring।

Library evaluation:

(১) XGBoost:

Pros:

Mature, stable।
SHAP excellent।
Bangladesh team familiarity।

Cons:

Categorical handling — one-hot blow up (100K product → 100K columns)।
Memory demand high।
Slow on 10M।
Target encoding leak risk।

(২) LightGBM:

Pros:

Speed — 10× XGBoost।
Memory efficient।
Native categorical (basic)।
10M scale ভাল।

Cons:

High-cardinality categorical — basic handling।
Leaf-wise overfit risk।
Tuning more required।

(৩) CatBoost:

Pros:

Categorical native — ordered boosting leak-free।
2 high-cardinality features ideal।
Less tuning। robust default।
Symmetric tree fast inference।

Cons:

Speed — slower than LightGBM।
Memory — multiple permutation।
Bangladesh team — less familiarity।

Recommendation: CatBoost primary:

(১) High-cardinality categorical decisive:

20 categorical features-এ CatBoost-এর native handling unmatched।
Product-id, brand-id — leak-free encoding।
Other libraries — manual CV target encoding overhead।

(২) Robustness:

Less hyperparameter tuning।
Production-ready out of box।
Less production debugging।

(৩) Inference speed:

Daily scoring 10M customers।
Symmetric tree fast।
Acceptable inference time।

(৪) Interpretability:

SHAP support।
Per-customer explanation possible।

Implementation strategy:

Phase 1 — Feature engineering:

Time-based: days_since_last_order, recency-frequency-monetary।
Categorical: district, product_category, brand, payment_method।
Behavioral: cart_abandonment_rate, return_rate, discount_usage।
Demographic: age_group, gender, urban_rural।

Phase 2 — Baseline (CatBoost):

from catboost import CatBoostClassifier
cat_features = ['district', 'product_category', 'brand', ...]
model = CatBoostClassifier(
    iterations=1000,
    learning_rate=0.05,
    depth=6,
    cat_features=cat_features,
    early_stopping_rounds=50,
    auto_class_weights='Balanced',  # imbalance handle
)

Phase 3 — Cross-validation:

Time-based split (last 3 months test)।
Stratified — class balance।
Per-segment evaluation (district-wise)।

Phase 4 — Tuning:

Optuna 100 trials।
depth, learning_rate, l2_leaf_reg।
cat_features encoding type।

Phase 5 — Production:

Daily retraining schedule।
SHAP per-customer explanation cache।
Drift monitoring।
A/B test against current system।

Backup plan — LightGBM:

If CatBoost speed inadequate।
Manual ordered target encoding pipeline।
Hash-based categorical।

Ensemble option:

CatBoost + LightGBM weighted average।
Diverse model errors decorrelate।
Production cost 2×, accuracy +1-2%।

Risks:

Memory: 10M × 50 features — careful sampling/streaming।
Concept drift: seasonal sales pattern change।
Cold start: new customer features sparse।
Bias: demographic fairness audit।

Bangladesh-specific:

Eid spike — pattern shift।
COD vs digital — payment behavior।
Mobile vs desktop — device interaction।
Bengali product names — text features।

Success metrics:

AUC-PR (imbalanced)।
Recall at top 10% (campaign target)।
Calibration plot।
Lift chart।
Business impact: retention rate change।

Long-term:

Embedding for products (NN-based)।
Sequence models (RNN/Transformer)।
Multi-task learning (churn + lifetime value)।

মূল উপলব্ধি: Library choice — data character driven। Daraz scenario — CatBoost ideal due to categorical dominance। Process > tool.

অনুশীলন

হিসাব করুন: GOSS — top $a = 0.2$, bottom থেকে $b = 0.1$। ১০০০ samples-এ — কতগুলো sample retained, computational saving কত?
- Top: $0.2 \times 1000 = 200$।
- Bottom random: $0.1 \times 800 = 80$।
- Total: ২৮০ samples (২৮%)।
- Computational saving: ~৭২%।
- Accuracy loss minimal — distribution unbiased।

Compare: Same dataset-এ XGBoost, LightGBM, CatBoost — train time + accuracy compare।

import time
for name, model in [('xgb', xgb.XGBClassifier()),
                    ('lgb', lgb.LGBMClassifier()),
                    ('cat', CatBoostClassifier(verbose=0))]:
    t0 = time.time()
    model.fit(Xt, yt)
    print(f"{name}: {time.time()-t0:.1f}s, acc {model.score(Xv,yv):.4f}")

চিন্তা: high-cardinality categorical (১০,০০০ unique values) — XGBoost-এ কী problem? CatBoost কীভাবে handle?

XGBoost: one-hot — ১০K columns explosion, sparse, slow। Label encoding — semantic-less ordering। Manual target encoding leak risk। CatBoost: ordered boosting native — no leak, dense, fast।

এই পাঠে যা শিখবেন

১ · LightGBM — Microsoft-এর speed champion

(ক) Leaf-wise growth

(খ) GOSS — Gradient-based One-Side Sampling

(গ) EFB — Exclusive Feature Bundling

২ · CatBoost — Yandex-এর categorical king

(ক) Ordered Boosting — target leak prevention

(খ) Symmetric (oblivious) trees

৩ · Comparison — কখন কোনটি

৪ · Python — LightGBM

৫ · Python — CatBoost

৬ · GOSS — গাণিতিক explanation

৭ · CatBoost — categorical encoding-এর "leak" সমস্যা

৮ · কোথায় এগুলো fail

ভাবনার প্রশ্ন

অনুশীলন

আরও পড়ুন