পাঠ ৪১ · ৪৫-এর মধ্যে · মডিউল ৫

Home / AI Courses / Machine Learning / Hyperparameter Tuning

Hyperparameter tuning

Hyperparameter optimization — grid, random, Bayesian

৭ মিনিট পড়া মাঝারি · Intermediate sklearn / Optuna

এই পাঠে যা শিখবেন

Hyperparameter বনাম parameter — মৌলিক পার্থক্য
Grid, random, Bayesian — তিন paradigm
Cross-validation ও nested CV — proper evaluation
Optuna — modern Python library, TPE
Bangladesh credit scoring case study

১ · Hyperparameter কী, parameter-এর সাথে পার্থক্য

ML model-এ দু'ধরনের "knob" আছে:

Parameter: training-এ data থেকে শেখা — linear regression-এর coefficient $\boldsymbol{\beta}$, neural network weights।
Hyperparameter: training-এর আগে fix — learning rate, regularization $\lambda$, tree depth, $k$ in K-NN, $C$ in SVM।

Hyperparameter কে data থেকে শেখানো যায় না (একই data দিয়ে ভাল cross-validation overfit)। তাই hyperparameter optimization (HPO) — separate validation/CV দিয়ে।

কেন hyperparameter critical

XGBoost-এর default-এ ০.৭০ accuracy, tuned হলে ০.৮৫ — same model, ভিন্ন hyperparameter। Production-এ একটা ভাল-tuned simple model untuned complex model-এর চেয়ে ভাল হতে পারে। Tuning-ই অর্ধেক ML engineering।

২ · Grid search

প্রতিটি hyperparameter-এর কয়েকটি value list। সব combination evaluate। Exhaustive কিন্তু expensive।

$C \in \{0.1, 1, 10, 100\}$, $\gamma \in \{0.001, 0.01, 0.1, 1\}$, kernel $\in \{$rbf, poly$\}$ → ৪×৪×২ = ৩২ combination। প্রতিটিতে $k$-fold CV।

সমস্যা: "curse of dimensionality" — ৫টি hyperparameter, প্রতিটিতে ৫ value → ৩১২৫ combination।

৩ · Random search

Bergstra-Bengio (২০১২) — random search প্রায়ই grid-এর চেয়ে ভাল। কারণ:

Real performance প্রায়ই ১-২টি hyperparameter-এ sensitive।
Grid — irrelevant hyperparameter-এ wasteful evaluation।
Random — important dimension-এ broader coverage।

Continuous hyperparameter (learning rate)-এর জন্য log-uniform: $10^{-5}$ থেকে $10^0$ — uniform-এ exponentially distributed।

৪ · Bayesian optimization

Mockus (১৯৭৪)-এর কাজ — Snoek et al. (২০১২) ML-এর জন্য popularize।

Idea: previous trial থেকে শিখে — কোন hyperparameter combination next-best, সেটাই probe। সাধারণত two component:

Surrogate model: hyperparameter → CV score-এর mapping। সাধারণত Gaussian Process বা Random Forest। Posterior + uncertainty দু'টিই।
Acquisition function: next কোথায় try করব। Expected Improvement (EI), Upper Confidence Bound (UCB), Probability of Improvement (PI)।

Loop: surrogate fit → acquisition maximize → trial run → update surrogate → repeat। Sample-efficient — ২০-৫০ trial-এ random search-এর ১০০-এর সমান।

৫ · TPE (Tree-structured Parzen Estimator)

Hyperopt, Optuna-এর default। Bergstra et al. (২০১১) — instead of GP, two density estimate:

$l(x)$ = good trial-এর density (top-যেমন ২০%)।
$g(x)$ = bad trial-এর density।
Sample where $l(x)/g(x)$ high।

Categorical hyperparameter, conditional dependency — TPE handle ভাল। Most modern systems এই algorithm।

৬ · Cross-validation pitfalls

Single split overfit: validation accuracy চেয়ে test accuracy generally কম। তাই $k$-fold CV essential।
Data leak: preprocessing (StandardScaler) test data-এ fit হলে test info leak। Pipeline-এ wrap (L42)।
Multiple comparison: ১০০০ hyperparameter try → best লুকিয়ে চান্সে ভাল। Nested CV — outer test, inner tune।
Time-series: random fold ভুল — temporal leak। TimeSeriesSplit ব্যবহার।

৭ · Nested cross-validation

Outer loop ($k=5$): train+val ৪ fold, test ১। Inner loop ($k=3$): train ২ fold, val ১। Inner-এ tuning, outer-এ evaluation। Per-fold best hyperparameter ভিন্ন হতে পারে — final model-এর জন্য entire data-তে tuning।

২০ trial-এর তুলনা — Grid uniform but wasteful, Random broader coverage, Bayesian smart concentrate around optimum (red circle)।

৮ · sklearn-এ Grid + Random search

Python · sklearn

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import randint, uniform

X, y = make_classification(n_samples=1000, n_features=20,
                            n_informative=10, random_state=0)

# Grid search
grid = {"n_estimators": [50, 100, 200],
        "max_depth": [5, 10, None],
        "min_samples_split": [2, 5, 10]}
gs = GridSearchCV(RandomForestClassifier(random_state=0),
                  grid, cv=3, n_jobs=-1, scoring="roc_auc")
gs.fit(X, y)
print(f"Grid best: {gs.best_params_}")
print(f"Grid score: {gs.best_score_:.4f}")

# Random search
random_dist = {"n_estimators": randint(50, 300),
               "max_depth":     randint(3, 20),
               "min_samples_split": randint(2, 20),
               "max_features": uniform(0.1, 0.9)}
rs = RandomizedSearchCV(RandomForestClassifier(random_state=0),
                        random_dist, n_iter=30, cv=3,
                        n_jobs=-1, scoring="roc_auc",
                        random_state=0)
rs.fit(X, y)
print(f"\nRandom best: {rs.best_params_}")
print(f"Random score: {rs.best_score_:.4f}")

Random search প্রায়ই grid-এর সমান বা better — fewer trial-এ। Real-world ML — random search standard baseline।

৯ · Optuna — modern HPO

Python · Optuna

# pip install optuna
import optuna
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

def objective(trial):
    n_estimators = trial.suggest_int("n_estimators", 50, 300)
    max_depth    = trial.suggest_int("max_depth", 3, 20)
    min_split    = trial.suggest_int("min_samples_split", 2, 20)
    max_features = trial.suggest_float("max_features", 0.1, 1.0)

    clf = RandomForestClassifier(
        n_estimators=n_estimators, max_depth=max_depth,
        min_samples_split=min_split, max_features=max_features,
        random_state=0, n_jobs=-1)

    scores = cross_val_score(clf, X, y, cv=3, scoring="roc_auc")
    return scores.mean()

study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=30, show_progress_bar=False)

print(f"Optuna best: {study.best_params}")
print(f"Optuna score: {study.best_value:.4f}")

Optuna TPE — random-এর চেয়ে দ্রুত converge। Conditional hyperparameter, pruning (early stop), distributed — Optuna-এর strength।

Hyperparameter tuning-এ overfit করা সম্ভব — ১০০০ trial best CV pick করলে validation set-এ অস্বাভাবিক ভাল দেখাবে কিন্তু prod-এ disappoint। Nested CV বা separate held-out test set absolute essential।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ Random search সাধারণত grid-এর চেয়ে ভাল কেন? Bergstra-Bengio paper-এর মূল insight?

২০১২-এর landmark paper — common sense আঘাত করে।

Effective dimensionality:

Hyperparameter important-এর সংখ্যা usually কম (~১-২)।
Other hyperparameter-এর performance-এ marginal effect।
"Important" varies by dataset।

Grid problem:

৫ hyperparameter, প্রতিতে ৪ value → ১০২৪ trial।
Important হয়তো $\eta$ ও $\lambda$।
Grid-এ each value ৪ × ৪ × ৪⁻³ = ২৫৬ trial-এ replicate (অন্য hyperparameter cross)।
Effective unique value per important: 4।

Random advantage:

২০ random trial — ২০ unique value per important hyperparameter।
Important dimension-এ ৫× resolution।

Mathematical:

Grid: marginal coverage = $n^{1/d}$।
Random: marginal coverage = $n$।
$d$ বড় হলে random win।

Caveats:

Random "lucky" — variance high।
Reproducibility — seed control।
Multiple run important।

When grid better:

Few hyperparameters (1-2)।
All hyperparameter equally important।
Discrete only (categorical kernel choice)।

Practical takeaway:

3+ hyperparameter — random > grid।
Continuous range — random or Bayesian।
Log-uniform distribution for learning rate, $\lambda$।

Beyond random:

Bayesian — informed search।
Hyperband — successive halving।
BOHB — Bayesian + Hyperband hybrid।
Population-based training — neural network specific।

Bergstra-Bengio impact:

HPO field reset — random as baseline।
Bayesian methods accelerated।
NeurIPS publication legend।

মূল উপলব্ধি: Default-এ random search; specific need-এ grid বা Bayesian। "Always grid" — outdated practice।

প্র ০২ Nested CV vs single CV — কেন nested এত important? Computational cost-এর সাথে balance?

ML practice-এ subtle but critical।

Single CV problem:

Hyperparameter tune CV-এ best score।
Best score-এর hyperparameter generalize estimate?
BIASED — same data tune ও evaluate।
Optimistic bias।

Multiple comparison:

১০০০ hyperparameter try, best pick।
Best hyperparameter "lucky" on this CV split।
New data-এ regress।
Cawley-Talbot (২০১০) — selection bias paper।

Nested CV:

Outer fold: held-out test।
Inner fold: hyperparameter tune।
Outer score = unbiased generalization estimate।

Cost:

Outer 5 × Inner 3 × hyperparameter trials।
5× compute over single CV।

When essential:

Many hyperparameter try (>100)।
Small dataset।
Publishing results।
Production deployment evaluation।

When skip:

Few hyperparameter (e.g., default check)।
Large held-out test set available।
Compute-constrained, exploratory phase।

Alternative — train/val/test:

Single split: train (60%), val (20%), test (20%)।
Tune on val।
Final evaluate on test (untouched)।
Less compute, less reliable।

Bootstrap nested:

Bootstrap outer instead of K-fold।
Confidence interval naturally।
Computationally heavier।

Practical workflow:

Quick development — single CV।
Final candidate — nested CV।
Production model — train on full data with chosen hyperparameter।
Held-out test — final unbiased report।

sklearn nested CV:

cross_val_score(GridSearchCV(...), ...) — nested।
Outer cv from cross_val_score, inner from GridSearchCV।

Bangladesh case:

Credit scoring — small data critical。 nested CV essential।
Image classification — large data, hold-out OK।
Medical diagnosis — nested CV + external validation।

মূল উপলব্ধি: Single CV development quick; nested CV publication-grade। Compute budget-এ trade-off। External validation always paramount।

প্র ০৩ Bayesian optimization — surrogate model কতদূর accurate? Failure mode?

BO powerful but specific failure mode আছে।

Surrogate options:

Gaussian Process: classical, smooth response, $O(n^3)$।
Random Forest: SMAC, scalable, discrete-friendly।
TPE: Optuna default, density-based।
Neural network: deep BO, complex landscape।

Strengths:

Sample efficient — 50 trial > 1000 random often।
Uncertainty-aware — explore vs exploit balance।
Continuous + categorical hybrid।

Failure modes:

(১) High dimensional:

20+ hyperparameter — GP struggle।
Curse of dimensionality।
Random additive structure tricks।

(২) Discrete + continuous:

GP designed for continuous।
Discrete via embedding/encoding lossy।
Random Forest surrogate better।

(৩) Conditional:

"If kernel=RBF, then γ matters"।
Tree-structured space।
TPE handle natively, GP not।

(৪) Noisy objective:

CV score variance high (small data)।
Surrogate confused noise vs signal।
Multiple CV trials, average।

(৫) Long evaluations:

Each trial hour+।
BO sequential — total time long।
Parallel BO — multiple trial simultaneously।

(৬) Multi-objective:

Accuracy vs latency trade।
Pareto front search।
Standard BO single objective।

Practical mitigations:

Log-transform hyperparameter (learning rate, $\lambda$)।
Constrained search range domain knowledge।
Initial random warmup (10-20 trial)।
Pruning (Hyperband) — early stop bad trial।
Multi-fidelity — coarse evaluation first।

Library ecosystem:

Optuna — TPE default, modern API।
Hyperopt — TPE pioneer।
BayesOpt, GPyOpt — pure GP।
Ray Tune — distributed, integrated।
SMAC3 — RF surrogate।

Modern advances:

BOHB — Bayesian + Hyperband।
BOTORCH — PyTorch-based, modular।
AutoML systems — meta-learning, transfer।
Population-based training — async parallelism।

Bangladesh case:

XGBoost credit scoring — Optuna 100 trial, well।
Neural network — Hyperband + BO।
Resource-constrained — random + best practice tuning।

মূল উপলব্ধি: BO sample-efficient but fragile। Use library mature (Optuna), monitor convergence, fallback to random। AutoML-এর foundation BO কিন্তু modern BO library ই production-ready।

প্র ০৪ Bangladesh-এ একটি bank credit scoring model XGBoost — full HPO pipeline ডিজাইন।

Real production scenario — credit risk-এ tuning critical।

Setup:

Bank: BRAC Bank, Eastern Bank।
Data: 50K loan history, 30 features।
Target: default within 12 month (binary)।
Imbalance: 5% default, 95% repaid (L43)।

(১) Data split:

Time-based split — 2018-2022 train, 2023 test।
Within train: 5-fold time-aware CV।
Test held-out untouched।

(২) Pipeline:

Preprocessing — sklearn Pipeline (L42)।
Categorical encoding — target encoding।
Numeric — StandardScaler (XGBoost optional)।
Imbalance — class_weight, SMOTE during training।

(৩) Hyperparameters:

n_estimators: 100-1500 (with early stopping)।
learning_rate: 0.01-0.3 (log-uniform)।
max_depth: 3-10।
subsample, colsample_bytree: 0.5-1.0।
reg_alpha, reg_lambda: 1e-3 to 10 (log)।
scale_pos_weight: imbalance handle।

(৪) Search strategy:

Optuna with TPE।
200 trials।
Early-stop pruning per trial (Hyperband)।
Distributed — multiple workers।

(৫) Objective:

AUC primary।
Recall@FPR=10% secondary (regulator concern)।
Multi-objective Optuna।

(৬) Validation:

Time-aware CV (TimeSeriesSplit)।
Outer test — once at end।
Calibration check (Brier score)।
Sub-population fairness — gender, region, age।

(৭) Monitoring:

Validation curve — overfit detect।
Feature importance — instability tracker।
Hyperparameter parallel coordinate plot।
Optuna study persistent storage।

(৮) Deploy:

Best hyperparameter on full data train।
Calibration (Platt scaling)।
Model card documentation।
Production scoring API।
A/B test against current scorecard।

(৯) Compliance:

Bangladesh Bank guideline adherence।
Model risk management framework।
Bias audit — protected class।
Audit trail — every hyperparameter trial logged।

(১০) Continuous improvement:

Quarterly retrain।
HPO incremental — new data, warm start।
Drift monitoring।
Champion-challenger framework।

(১১) Common pitfalls avoided:

Data leak — target encoding inside CV।
Time leak — strict temporal split।
Overfit HPO — nested CV, hold-out test।
Imbalance distort — proper metric (AUC, not accuracy)।

(১২) Resource:

200 trial × 5 fold × ~30 sec/fit = ~8 hour।
Multi-core — 2 hour।
Cloud compute — manageable।

মূল উপলব্ধি: Production HPO 30% algorithm, 70% engineering — pipeline integrity, time-aware validation, fairness check, compliance। Bangladesh banking-এ — tuning competitive advantage, not commodity।

অনুশীলন

হিসাব করুন: ৫টি hyperparameter, প্রতিটিতে ১০ value। Grid search কত trial? Random ৫০ trial-এ কতগুলো unique value per hyperparameter (expected)?
- Grid: 10⁵ = 100,000 trials।
- Random 50 trials → প্রতিটি hyperparameter-এ ~50 unique value (continuous), ~9-10 (discrete 10 value)।
- Random many more unique value per dimension!

Optuna-তে চেষ্টা: Iris dataset SVM tuning।

import optuna
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

X, y = load_iris(return_X_y=True)

def objective(trial):
    C = trial.suggest_float("C", 1e-3, 1e3, log=True)
    gamma = trial.suggest_float("gamma", 1e-4, 1e1, log=True)
    kernel = trial.suggest_categorical("kernel", ["rbf", "poly"])
    clf = SVC(C=C, gamma=gamma, kernel=kernel, random_state=0)
    return cross_val_score(clf, X, y, cv=3, scoring="accuracy").mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=30, show_progress_bar=False)
print(study.best_params, study.best_value)

ভাবুন: Bangladesh-এ একটি startup-এ ML engineer সীমিত compute। কোন HPO strategy বেছে নেবেন কেন?
- Tier 1: Default + sensible manual tweaks — quick baseline।
- Tier 2: Random 20-50 trial — broad exploration।
- Tier 3: Optuna TPE 100 trial — focused refinement।
- Pruning: Hyperband-Optuna integration — bad trial early stop।
- Cache: CV split fix, intermediate result store।
- Cloud spot: AWS spot, GCP preemptible — cheap।
- Domain priors: XGBoost lr 0.05-0.1 sensible — narrow range।
- Avoid: exhaustive grid early; nested CV per iteration।