Out-of-Bag Error

OOB-test relationship — extensive research। Practical reliability strong, কিন্তু caveats আছে।

Theoretical foundation:

• Breiman (১৯৯৬) — bagging context-এ OOB consistent estimator।
• $n \to \infty$, $B \to \infty$ — OOB error → true generalization error।
• Bias term $O(1/n)$ — small data-এ visible।

Empirical validation:

• Bylander (২০০২) — extensive UCI benchmark।
• OOB ও 10-fold CV correlation 0.99+।
• Bias প্রায় শূন্য (random direction)।
• RMSE difference <1%।

Bias direction — slightly pessimistic:

• OOB এ — প্রতি tree ০.৬৩ data train।
• Final model ১.০ data train।
• OOB কম-data model approximate।
• Pessimism small (1-2%)।

0.632 estimator — Efron correction:

• Pessimism counteract করে।
• $\hat{\text{err}}_{0.632} = 0.368 \cdot \text{err}_{\text{train}} + 0.632 \cdot \text{err}_{\text{OOB}}$।
• 0.632+ — overfit-aware version।
• Modern RF-এ rarely used; raw OOB ভাল enough।

কখন OOB unreliable:

(১) Small dataset:

• $n < 100$ — OOB high variance।
• প্রতি sample শুধু ৩৭ trees-এ OOB।
• Vote noisy।
• Repeated CV বেশি stable।

(২) Class imbalance:

• Minority class — OOB-এ underrepresented।
• Per-class metric noisy।
• Stratified CV preferred।

(৩) Temporal data:

• Bootstrap order ignore।
• Time-correlated samples — leak।
• OOB optimistic — past predict past।
• Walk-forward validation দরকার।

(৪) Group structure:

• Multiple samples per patient/user।
• Bootstrap patient-level group ভাঙে।
• Solution: GroupKFold।

Empirical case studies:

• Genomic data: small $n$, large $p$ — OOB stable।
• Image: RF rarely used, irrelevant।
• Tabular finance: OOB excellent।
• Time series: OOB misleading।

Best practice:

• OOB — primary monitor (free)।
• CV — confirmation (slower)।
• Held-out test — final validation।
• Production — long-term performance tracking।

OOB diagnostic value:

• Train accuracy ১০০%, OOB ৯০% — typical RF।
• Train ১০০%, OOB ৬০% — overfit warning।
• OOB plateau — $B$ enough।

মূল উপলব্ধি: OOB — practical "free lunch"। ৯৫% RF use case-এ sufficient। Edge case-এ — supplement, replace।

Hyperparameter selection — OOB practical workhorse, কিন্তু subtle considerations আছে।

OOB-based tuning workflow:

• Different hyperparameter সেট — RF train।
• প্রতিটির OOB score record।
• Best OOB score-এর hyperparameter বাছুন।
• Final model — সম্পূর্ণ data-এ retrain।

সুবিধা:

(১) Computational saving:

• 5-fold CV — ৫× retraining।
• OOB — single training।
• Grid search-এ ৫× speedup।

(২) Full data usage:

• CV — fold-এ ৮০% data train।
• OOB — সব data-এ train, ৩৭% per-tree validation।
• Larger effective training।

(৩) Variance lower:

• প্রতি sample-এ many trees vote।
• CV — fold-এ once test।
• OOB smoother estimate।

সমস্যা:

(১) Hyperparameter scope limited:

• OOB শুধু RF-এ — bagging-only।
• Boosting (XGBoost), neural net — CV/early stopping।

(২) $B$ depend:

• Small $B$ — OOB noisy।
• $n\_estimators$ tune করতে — OOB ব্যবহার করলে cyclic dependency।
• $B$ fixed করে অন্য parameter tune।

(৩) Class imbalance:

• OOB stratified না।
• Minority class metric noisy।
• StratifiedKFold preferred।

(৪) Search space coverage:

• OOB — fast, broader search সম্ভব।
• CV — costly, careful selection।
• OOB-CV combined — broad OOB, focused CV।

Compare with CV:

• OOB: ১× cost, slightly pessimistic।
• 5-fold CV: ৫× cost, bias direction unbiased।
• 10-fold CV: ১০× cost, lowest variance।
• Repeated CV: N×k× cost, gold standard।

Practical strategy:

• (১) Initial search: wide, OOB।
• (২) Refine: narrow, 5-fold CV।
• (৩) Final: held-out test।
• Computational ও statistical balance।

Specific hyperparameter:

• $n\_estimators$: OOB plateau দেখুন। ১০০-৫০০ usually।
• $max\_depth$: OOB sweep — overfit detect।
• $min\_samples\_leaf$: OOB validate noise robustness।
• $max\_features$: sqrt vs log2 vs fraction — OOB compare।

Code pattern:

best_params = None
best_oob = 0
for max_depth in [5, 10, 15, None]:
    for max_features in ['sqrt', 'log2', 0.3]:
        rf = RandomForestClassifier(
            n_estimators=200,
            max_depth=max_depth,
            max_features=max_features,
            oob_score=True,
            random_state=0,
            n_jobs=-1
        ).fit(X, y)
        if rf.oob_score_ > best_oob:
            best_oob = rf.oob_score_
            best_params = (max_depth, max_features)

মূল উপলব্ধি: OOB tuning — RF-এ practical। CV — universal। Hybrid often best।

০.৩৬৮ — bootstrap statistics-এর iconic number। এর গণিত elegant এবং deep।

Derivation:

• $n$ samples-এ একটি specific sample।
• প্রতি draw-এ — চয়ন না-হওয়ার probability $1 - 1/n$।
• $n$ independent draws — none chose probability $(1 - 1/n)^n$।

Limit:

$$\lim_{n \to \infty} \left(1 - \frac{1}{n}\right)^n = \frac{1}{e} \approx 0.3679$$

• $e$ — Euler's number, mathematically deep।
• Compound interest, decay process — সব পরিচিত।

Convergence rate:

• $n = 10$ — $0.349$।
• $n = 100$ — $0.366$।
• $n = 1000$ — $0.368$।
• $n = 10000$ — $0.368$।
• $n = 50$+ — limit-এর ১% within।

Why $1/e$?

• $e^x$-এর Taylor expansion।
• $(1 - 1/n)^n = \exp(n \log(1 - 1/n)) \approx \exp(-1) = 1/e$।
• Probability theory's Poisson distribution-এর সাথে সম্পর্কিত।

Practical implications:

(১) OOB samples ~৩৭%:

• Per tree — ৩৭% data unseen।
• Sufficient validation set।
• "Free" CV-like effect।

(২) Effective training set:

• Per tree — ৬৩% unique samples।
• Some samples 2x, 3x, 4x repeated।
• Diversity yet adequate signal।

(৩) Bias-variance:

• ৬৩% data-এ trained tree — slightly more bias।
• OOB error pessimistic (slight)।
• 0.632 estimator correction।

(৪) Diversity engineering:

• Different bootstraps — ভিন্ন ৬৩% samples।
• Trees see different "world"।
• Decorrelation natural।

Connection to other ML:

• Stochastic Gradient Descent: mini-batch sampling — bootstrap analog।
• Dropout: ৫০% activation drop — similar diversity idea।
• Bayesian inference: bootstrap → posterior approximation।

Variants:

• Subsampling: without replacement, fixed fraction।
• Bayesian bootstrap: Dirichlet weights।
• Block bootstrap: time series-এ।
• Stratified bootstrap: class imbalance।

OOB sample size:

• $n = 1000$ → ~৩৭০ OOB per tree।
• ৫০০ trees → প্রতি sample ~১৮৫ trees-এ OOB।
• Robust prediction।
• Statistical confidence।

Historical note:

• Efron (১৯৭৯) — bootstrap-এর introduction।
• Originally — confidence interval estimation।
• Breiman (১৯৯৬) — bagging-এ extension।
• Modern RF — direct application।

মূল উপলব্ধি: $1/e$ — coincidence নয়। Bootstrap-এর core mathematical structure। Understanding গভীর করে practitioner-কে।

Class imbalance — medical/finance ML-এর recurring scenario। OOB strength ও weakness দু'টোই বের।

Class imbalance OOB-এ — challenges:

(১) Bootstrap class proportion:

• Original — ১% disease, ৯৯% healthy।
• Bootstrap — same ratio (random)।
• Per-tree — ১% disease (~১ disease, ~১০০ healthy in 100 sample)।
• Tree learn — predict majority always।

(২) OOB minority underrepresentation:

• Disease ১০০ — OOB-এ ~৩৭ disease per tree।
• Sufficient validation কিন্তু noisy।
• Per-class metric high variance।

(৩) Accuracy misleading:

• OOB accuracy ৯৯% — predict all healthy।
• Disease detection ০%।
• "Good model" appearance, useless reality।

OOB কি reliable — answer: depends:

(ক) Overall metric — reliable:

• Accuracy, AUC — OOB-এ unbiased।
• Trend (improving/degrading) — clear।

(খ) Per-class — unreliable:

• Recall (sensitivity) noisy।
• Precision noisy।
• Confidence interval wide।
• Stratified validation দরকার।

Alternatives ও solutions:

(১) Stratified Cross-Validation:

• StratifiedKFold — class proportion preserve।
• Per-fold reliable minority metric।
• $5 \times$ cost, but worth।

(২) Class weights:

• class_weight='balanced' — sklearn।
• Loss function — minority weighted up।
• Tree split — minority focus।

(৩) Balanced bagging:

• BalancedRandomForestClassifier (imblearn)।
• Each bootstrap — equal class proportion।
• Better minority learning।

(৪) SMOTE / oversampling:

• Synthetic minority generation।
• Pre-RF preprocessing।
• Bootstrap diversity preserve।
• Pipeline-এ integrate।

(৫) Threshold tuning:

• Default ০.৫ — accuracy-based।
• F1 / cost-aware threshold।
• OOB probability calibration check।

(৬) Anomaly detection framework:

• Isolation Forest — bagging variant।
• One-class SVM।
• Rare event modeling।

(৭) Ensemble of ensembles:

• RF + balanced approach + cost-sensitive।
• Multiple model averaging।
• Robust।

Recommended workflow:

• (১) EDA: imbalance ratio quantify।
• (২) Stratified split: train/test/holdout।
• (৩) Class weights / SMOTE: training pipeline।
• (৪) Stratified CV: validation।
• (৫) Per-class metrics: precision, recall, F1, AUC।
• (৬) Cost-aware threshold: business cost integrate।
• (৭) OOB: overall trend monitoring।

Bangladesh medical ML scenario:

• Cancer screening — ১% prevalence।
• Sepsis detection — ৫% prevalence।
• Diabetic retinopathy — ১০-২০% prevalence।
• Each different strategy।

Specific advice:

• Recall (sensitivity) primary — false negative critical।
• Precision secondary — false positive — additional test।
• F2 score — recall weighted।
• Decision threshold — clinician input।

Validation rigor:

• Multi-site validation।
• Temporal validation (different seasons, years)।
• Subgroup analysis (age, gender, district)।
• Calibration check।

মূল উপলব্ধি: Class imbalance — algorithm only-এর বিষয় নয়, end-to-end pipeline। OOB diagnostic, not silver bullet। Domain-aware metric selection critical।