পাঠ ২১ · ৩০-এর মধ্যে · মডিউল ৩

Feature selection

Feature selection
৭ মিনিট পড়া মাঝারি · Intermediate Filter, wrapper, embedded

এই পাঠে যা শিখবেন

  • Feature selection-এর তিন family — filter, wrapper, embedded
  • Correlation, mutual information, RFE, Lasso, tree importance
  • Selection-এ leakage এড়ানোর pattern
  • "Garbage feature" vs "redundant feature" পার্থক্য
  • Bangladesh project-এ practical workflow

১ · কেন feature selection

Feature selectionFeature Selectionসব feature থেকে সবচেয়ে relevant subset বাছা — যাতে model simpler, faster, and often more accurate হয়। Filter, wrapper, embedded — তিন family। ML-এর "less is more" principle। Cause:

  • Curse of dimensionality: high-D-এ generalization কঠিন।
  • Overfitting: noisy feature মডেলকে memorize করায়।
  • Train time: kth feature add → time linear-quadratic বাড়ে।
  • Interpretability: ১,০০০ feature-এর model ব্যাখ্যা অসম্ভব।
  • Production cost: প্রতি feature compute, store, monitor — infrastructure।
  • Latency: real-time inference-এ feature collection time-critical।

Bangladesh fintech-এ একটি transaction-time scoring 100ms-এর কম দরকার। ৫০০ feature compute করলে সম্ভব না। ৩০-৫০ select করতে হয়।

২ · তিন family

Selection-এর তিন approach

১) Filter: মডেল-independent — feature-target relationship measure।
২) Wrapper: মডেল-train করে subset compare।
৩) Embedded: মডেল training-এর সাথেই selection।

৩ · Filter methods

প্রতিটি feature-এ target-এর সাথে statistical metric। Threshold-এ কম-গুরুত্বপূর্ণ drop।

  • Correlation (Pearson): numeric-numeric। $|r| < 0.05$ — drop।
  • Spearman correlation: rank-based; non-linear monotonic।
  • Mutual Information: non-linear too; categorical ও numeric দু'টোতে।
  • Chi-square test: categorical-categorical।
  • ANOVA F-test: categorical target, numeric feature।
  • Variance threshold: low-variance (constant-প্রায়) drop।

Mutual Information:

$$I(X; Y) = \sum_{x, y} p(x, y) \log \frac{p(x, y)}{p(x) p(y)}$$

$I = 0$ — independent। $I > 0$ — কিছু relationship। Sklearn-এ mutual_info_classif।

সুবিধা: দ্রুত, scalable, model-agnostic।
সীমা: univariate — feature-feature interaction miss। দু'টি feature individually weak কিন্তু together strong — drop হয়ে যেতে পারে।

৪ · Wrapper methods

Subset দিয়ে model train, performance দিয়ে compare। Computationally expensive কিন্তু accurate।

  • Forward selection: empty থেকে শুরু — প্রতিটি step-এ best feature add।
  • Backward elimination: all থেকে শুরু — প্রতিটি step-এ worst drop।
  • Recursive Feature Elimination (RFE): model train, lowest importance drop, repeat।

Sklearn: RFE(estimator, n_features_to_select=k)। Cross-validated version: RFECV।

Cost: $n$ feature, $k$ select → $O(n \cdot k \cdot \text{train\_time})$।

৫ · Embedded methods

(ক) Lasso (L1 regularization):

$$\min_w \|y - Xw\|_2^2 + \lambda \|w\|_1$$

L1 penalty কিছু coefficient exactly ০-এ pull করে। সেই feature automatically dropped।

Sklearn: Lasso(alpha=0.1)। Ridge (L2) — coefficient shrink করে কিন্তু ০ করে না।

(খ) Tree feature_importance:

  • প্রতিটি feature-এ training-এ split-এ contribution।
  • RandomForest, XGBoost, LightGBM সবারই .feature_importances_ attribute।
  • Threshold-এ low-importance drop।

(গ) SHAP values:

  • Game-theory based explanation।
  • Each feature-এ row-level contribution।
  • Global feature ranking।

সুবিধা: model-aware; interactions ধরে; production-ready।
Caveat: tree importance correlated feature-এ unstable; SHAP slow।

"Feature importance" interpretation সাবধানে: tree-এ correlated দু'টি feature থাকলে — importance arbitrary split। কোনটা important সেটা runs-এ vary।

৬ · Multicollinearity ও VIF

দু'টি feature highly correlated হলে — redundant। Linear model coefficient unstable। VIF (Variance Inflation Factor):

$$VIF_i = \frac{1}{1 - R^2_i}$$

যেখানে $R^2_i$ — feature $i$-কে অন্য সব দিয়ে predict-এর R²। VIF > ১০ — drop consider।

৭ · Leakage-এ সাবধানতা

  • Selection train-set-এ: full data-তে selection → test info leak।
  • Cross-validation pipeline: প্রতিটি fold-এ selection নতুন।
  • Time-series: walk-forward selection।
  • Hyperparameter (e.g., k in SelectKBest): nested CV-এ tune।
ভাবুন BUET admission। ১,০০,০০০ ছাত্র, ৫,০০০ কে admit করতে হবে। প্রতিটি ছাত্রের ১০০ পরীক্ষার score আছে। সব use করলে — overwhelmed। Filter → top 30 score (correlation with success); Wrapper → এই 30 থেকে best 10 combination; Embedded → ML model আপনাকে বলে দিচ্ছে কোন 5 ultimately matter। Production-এ এই 5-ই সংরক্ষিত।
Feature selection — three-stage workflow Filter → wrapper → embedded ১ · Filter corr, MI, chi² দ্রুত — ১০০০→১০০ ২ · Wrapper RFE / forward accurate — ১০০→৩০ ৩ · Embedded Lasso / tree imp final — ৩০→১০ Final feature set — ১০টি, model-ready explainable, fast, robust 📌 প্রতিটি stage train-এ; test-এ apply ⚠ Leakage এড়াতে — Pipeline-এ wrap model-agnostic cross-validated model-aware
Three-stage feature selection — wide → narrow funnel।

৮ · Filter — sklearn কোড

Python · sklearn
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, mutual_info_classif, f_classif

X, y = load_breast_cancer(return_X_y=True)
feature_names = load_breast_cancer().feature_names

# Top 10 features by mutual information
selector = SelectKBest(mutual_info_classif, k=10)
X_new = selector.fit_transform(X, y)

selected = [feature_names[i] for i in selector.get_support(indices=True)]
print(f"Total features: {X.shape[1]}")
print(f"Selected: {X_new.shape[1]}")
print(f"\nTop 10:")
for f, score in sorted(zip(feature_names, selector.scores_), key=lambda t: -t[1])[:10]:
    print(f"  {f}: {score:.3f}")

    
Mutual information non-linear relationship ধরে — Pearson correlation-এর চেয়ে general।

৯ · Lasso embedded selection

Python · sklearn
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
names = load_diabetes().feature_names
X_scaled = StandardScaler().fit_transform(X)

lasso = LassoCV(cv=5, random_state=0).fit(X_scaled, y)

print(f"Best alpha: {lasso.alpha_:.4f}")
print(f"\nCoefficients:")
for n, c in zip(names, lasso.coef_):
    flag = "✓" if abs(c) > 0.001 else "✗"
    print(f"  {flag} {n:10s}: {c:+.2f}")

n_selected = (np.abs(lasso.coef_) > 0.001).sum()
print(f"\nSelected: {n_selected}/{len(names)}")

    
Lasso L1 penalty কিছু coefficient ০-এ shrink করে — সেই feature automatically dropped। Alpha CV-তে chosen।

১০ · Tree feature importance

Python · sklearn
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)
names = load_breast_cancer().feature_names

rf = RandomForestClassifier(n_estimators=100, random_state=0).fit(X, y)

# Sort by importance
importance = sorted(zip(names, rf.feature_importances_), key=lambda t: -t[1])
print("Top 10 features by RF importance:")
for n, imp in importance[:10]:
    print(f"  {n:30s}: {imp:.4f}")

    
Random Forest-এ "worst perimeter", "worst concave points" সাধারণত top — domain-এ malignant tumor-এর strong indicator। Production-এ এই top-K রাখুন।

১১ · Bangladesh ML pipeline-এ feature selection

  • bKash fraud (5000 features): filter (chi², MI) → 200; RFE → 50; XGBoost feature importance → 30 production।
  • Daraz CTR: mostly tree-importance based — embedded efficient।
  • Pathao surge: Lasso-cv pre-selected; linear model production।
  • Microfinance loan default: domain-driven (income, history, demography); selection refinement।

১২ · Common pitfalls

  • Selection on full data: leakage; train-only।
  • Univariate-only: interaction-rich data-এ poor।
  • Importance trust: correlated feature-এ unstable।
  • Domain ignoring: stat-only — business knowledge integrate করুন।
  • Future leakage: "customer_lifetime_value" target predict-এ — definition-এ leak।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ "Filter univariate" এবং "interaction" — দু'টো feature individually unimportant কিন্তু together important। Concrete example দিন। কীভাবে detect?

এটি ML-এর সবচেয়ে subtle phenomenon — synergy বা interaction effect।

Concrete example ১: XOR pattern

  • $x_1 \in \{0, 1\}$, $x_2 \in \{0, 1\}$।
  • $y = x_1 \oplus x_2$ (XOR)।
  • Truth table:
  • (0,0) → 0, (0,1) → 1, (1,0) → 1, (1,1) → 0।
  • $\text{corr}(x_1, y) = 0$, $\text{corr}(x_2, y) = 0$।
  • Univariate filter — both drop।
  • কিন্তু combined — perfectly predict।

Concrete example ২: Fraud detection

  • "transaction_hour" alone — fraud rate uniform।
  • "is_first_time_recipient" alone — slight elevated।
  • Combined: "first-time recipient at 3am" — fraud rate ১০x।
  • Univariate miss; interaction crucial।

Concrete example ৩: Healthcare

  • Drug A — overall benefit zero (mean across population)।
  • Drug A + gene marker B — strong benefit।
  • "Personalized medicine" rests on interactions।

Bangladesh example: Microfinance default

  • "Income" alone — moderate predictor।
  • "Loan size" alone — moderate predictor।
  • "Income / Loan-size ratio" — strong predictor।
  • Univariate miss; engineered interaction key।

Detection methods:

(১) Tree-based importance

  • Random Forest, XGBoost interaction natively capture।
  • Feature importance still reflect both।
  • Tree split — interaction implicitly modeled।

(২) SHAP interaction values

  • SHAP feature-pair contribution explicitly।
  • shap.TreeExplainer.shap_interaction_values()।
  • Top interaction pairs identify।

(৩) Friedman's H-statistic

  • Specifically interaction strength measure।
  • 0 = no interaction, 1 = pure interaction।
  • sklearn.inspection.partial_dependence 2D।

(৪) Polynomial features (manual)

  • PolynomialFeatures(degree=2, interaction_only=True)।
  • Pairwise multiplications create।
  • Explosion: $n$ feature → $n^2$ — careful।

(৫) Domain-driven feature engineering

  • Best approach often।
  • "income/expense ratio" engineered।
  • "hour × is_weekend" interaction।

Recommendation pipeline:

  1. Domain expert review — likely interactions list।
  2. Engineer key interactions explicitly।
  3. Tree model train — auto-detect rest।
  4. SHAP analysis — surprising interactions reveal।
  5. Refine feature engineering iteratively।

Univariate filter — কোথায় usefulful:

  • Initial screening — 5,000 → 500।
  • Feature-wise garbage detection।
  • Computational efficiency।
  • But never sole criterion।

মূল উপলব্ধি: Univariate quick screen কাজে লাগে কিন্তু sufficient না। Tree model + SHAP interaction-aware। Best — domain knowledge দিয়ে interaction explicit বানানো।

প্র ০২ Lasso-এর L1 vs Ridge-এর L2 — coefficient-এর behavior কেন আলাদা? কেন L1 zero তৈরি করে কিন্তু L2 না?

এটি ML-এর সবচেয়ে elegant geometric insight গুলোর একটা। Optimization landscape বুঝলে — intuitive।

Loss formulation:

Lasso: $L = \|y - Xw\|_2^2 + \lambda \|w\|_1 = \sum (y_i - \hat{y}_i)^2 + \lambda \sum |w_j|$

Ridge: $L = \|y - Xw\|_2^2 + \lambda \|w\|_2^2 = \sum (y_i - \hat{y}_i)^2 + \lambda \sum w_j^2$

Constraint formulation (equivalent):

Lasso: minimize MSE subject to $\sum |w_j| \le t$ (L1 ball)।

Ridge: minimize MSE subject to $\sum w_j^2 \le t$ (L2 ball)।

Geometric picture:

  • MSE contours — ellipses on (w₁, w₂) plane।
  • Lasso constraint — diamond (rotated square)।
  • Ridge constraint — circle।
  • Optimal w — যেখানে contour first touches constraint।

Why diamond produces zeros:

  • Diamond-এর "corner" axis-এ — যেমন (t, 0) corner।
  • Ellipse এই corner-এ touch করার সম্ভাবনা বেশি (corner sharp)।
  • Touch on axis = one coordinate exactly 0।
  • 2D-এ এই argument; high-D-এ corner আরও বেশি, zero-feature বেশি।

Why circle does not produce zeros:

  • Circle smooth — কোনো corner নেই।
  • Ellipse touch generic point-এ — both coordinates non-zero।
  • Coefficient shrink হয় কিন্তু ০ হয় না।

Mathematical proof — subdifferential:

  • $|w|$-এর derivative ০-এ undefined; subdifferential $[-1, 1]$।
  • Optimum-এ KKT condition: gradient ∈ subdifferential।
  • Range $[-1, 1]$ allow $w^* = 0$ — exact sparsity।
  • $w^2$-এর derivative ০-এ ০ — exact sparsity rare (only by chance)।

Practical implications:

Lasso:

  • Automatic feature selection।
  • Sparse model — production-friendly।
  • High-D-এ ($n < p$): Lasso essential।
  • Issue: correlated feature-এ random selection (ভাল feature drop, similar-correlated keep)।

Ridge:

  • All feature retain — small weight।
  • Multicollinearity handle ভাল (stable coefficient)।
  • Smooth regularization।
  • Issue: feature selection না।

Elastic Net — best of both:

$$L = \text{MSE} + \lambda_1 \|w\|_1 + \lambda_2 \|w\|_2^2$$

  • L1 + L2 mixed।
  • Sparsity + stability।
  • Sklearn: ElasticNet(alpha=0.1, l1_ratio=0.5)।
  • Correlated feature group together select।

Hyperparameter tuning:

  • $\lambda$ (alpha): $0 \to \infty$।
  • $\lambda = 0$: no regularization।
  • $\lambda \to \infty$: all weights → 0।
  • CV-তে select: LassoCV।

Bayesian interpretation:

  • Lasso = MAP estimate with Laplace prior (sharp peak at 0)।
  • Ridge = MAP with Gaussian prior।
  • Laplace's sharp peak → coefficients pulled to 0।

Common applications:

  • Genomics: 20K gene → handful predictive। Lasso natural।
  • NLP: 100K word features → top hundred। Lasso।
  • Bangladesh fintech: 5K transaction features → 30 production। Lasso first cut।

Best practices:

  • Always scale features before Lasso/Ridge (penalty fair)।
  • CV for alpha।
  • Lasso path visualize: lasso_path() — alpha varies, see coefficients enter/exit।
  • Stability selection: bootstrap + Lasso → reproducible feature set।

মূল উপলব্ধি: L1 vs L2 — geometry-driven behavior। Lasso natural feature selector; Ridge stabilizer। Elastic Net balance। Production-এ — Lasso first cut, then refine।

প্র ০৩ "Tree feature_importance" সাবধানে use করতে বলা হয়। কোন কোন বিকৃতি? Permutation importance ও SHAP কেন better?

Random Forest-এর built-in feature importance — quick কিন্তু flawed। Production-grade interpretability-এ সাবধানতা।

Default tree importance — Mean Decrease Impurity (MDI):

  • প্রতিটি split-এ Gini/entropy reduction।
  • সব tree-এ sum এ feature-এর contribution।
  • Sklearn-এ .feature_importances_ default।

সমস্যা ১: High-cardinality bias

  • Many-unique-value feature-এ split-options বেশি।
  • Inflated importance — even if no real signal।
  • Numerical feature continuous → many split options।
  • Categorical feature few values → low split options।
  • Numeric feature artificially favored।

Concrete example:

  • Add a column "random_int_1_to_1000" — pure noise।
  • RF importance often top-10!
  • Cardinality fakes signal।

সমস্যা ২: Correlated features

  • X1 ও X2 highly correlated; both predictive।
  • Tree randomly chooses one for split।
  • Importance distributed arbitrarily।
  • "X1 important" run-1, "X2 important" run-2 — same data, different result।

সমস্যা ৩: Training data only

  • MDI training-set splits-এ based।
  • Generalization-এর reflection না।
  • Overfitting-prone feature inflated importance।

Permutation importance — alternative:

  1. Train model।
  2. Test set-এ prediction performance baseline measure।
  3. একটি feature-এর values randomly permute।
  4. Performance drop measure।
  5. Drop = importance।

Permutation importance-এর সুবিধা:

  • Test-set based — generalization reflect।
  • Cardinality-bias কম।
  • Model-agnostic — যেকোনো model-এ apply।
  • Sklearn: permutation_importance()।

Permutation-এর সমস্যা:

  • Correlated feature-এ — permute X1, X2-ও deteriorate (still correlated to permuted)। Importance underestimate।
  • Computationally expensive (per feature, per repeat)।
  • Interpretation: "if this feature randomly garbage" — not "if removed"।

SHAP — Shapley values:

  • Game-theory: cooperative game-এ player-এর fair contribution।
  • প্রতিটি row-এ feature contribution prediction-এ।
  • Summing across rows → global importance।
  • Mathematically principled।

SHAP-এর সুবিধা:

  • Local + global interpretation।
  • Interaction effects capture।
  • Direction (positive/negative) shown।
  • Audit-friendly — regulatory standard।
  • Visualizations (waterfall, beeswarm, dependence plot)।

SHAP-এর সমস্যা:

  • Computationally expensive (TreeSHAP fast for trees)।
  • Correlated feature still tricky (different background distribution issue)।
  • Library: shap।

Comparison summary:

  • MDI (default): fast, biased, training-set।
  • Permutation: slower, less biased, test-set।
  • SHAP: slowest, most principled, local+global।

Production recommendation:

  • Quick screening: MDI।
  • Decision-making: Permutation (test set)।
  • Audit/compliance: SHAP।
  • Combine: cross-check three methods।

Bangladesh examples:

  • Bangladesh Bank-এর regulatory loan-default model — SHAP audit।
  • BFIU-এ AML model — explainability mandate।
  • Daraz CTR — MDI quick, SHAP for product team explanation।

Common mistake:

  • "Top-3 RF importance — these features matter" — without permutation check।
  • Cardinality-biased decision।
  • Stakeholder explanation misleading।

মূল উপলব্ধি: Default tree importance fast কিন্তু biased। Permutation + SHAP combination — robust feature insight। Interpretability ML ethics-এর foundation।

প্র ০৪ "Future leakage feature" বলতে কী বোঝায়? bKash/Daraz-এর context-এ ৩-৪ উদাহরণ দিন।

Feature leakage — production ML-এর সবচেয়ে devastating bug। Model দেখায় ৯৫% accuracy, production-এ ৬০% — কারণ training-এ "future" দেখা।

Definition:

  • Feature যা target-এর সাথে সম্পর্কিত প্রক্রিয়ার পরে compute হয়।
  • Inference time-এ unavailable।
  • Training-এ available — leakage সৃষ্টি।

Categories:

(১) Time leakage

  • Feature created after target event।
  • Production-এ "এখন থেকে" আমরা future জানি না।

(২) Target leakage

  • Feature target-কে directly বা proxy-তে contain।
  • "is_blocked_due_to_fraud" feature → "is_fraud" target predict — circular।

(৩) Train-test leakage

  • Test data train-এ peek (encoding, scaling)।
  • আগের পাঠে discussed।

bKash example ১: "n_failed_login_attempts"

  • Fraud prediction-এ — ৯৯% accuracy।
  • কিন্তু — failed login fraud detection trigger করে; security team account flag করে।
  • "n_failed_login" historical = pre-flag, modern = post-flag mixed।
  • Training data-তে (post-fraud entries-এ ৫০ failed login) but production prediction time-এ (০ failed login) — vastly different distribution।
  • Solution: only use pre-fraud-flag failed logins।

bKash example ২: "customer_complaint_count"

  • Fraud detection-এ "had complaint" feature।
  • Customer complaint = fraud detect করে ফেলে fast track।
  • Complaint = consequence, not cause।
  • Model "complaint exists → fraud" — but at scoring time, no complaint yet।

bKash example ৩: "lifetime_transaction_count"

  • Churn prediction-এ feature।
  • Lifetime = entire history, including AFTER target prediction window।
  • Churned customer-এর "lifetime" small (closed early)।
  • Active customer-এর "lifetime" growing।
  • Trivial classifier।
  • Solution: time-aware aggregation (last 30 days, fixed window)।

Daraz example ১: "review_count"

  • Conversion prediction-এ feature।
  • Review post-purchase লেখা হয়।
  • "আমি কি কিনব?" predict-এর সময় — review count ০ (not yet purchased)।
  • Training-এ: purchased → reviewed → review_count high → conversion target = 1।
  • Trivial pattern।

Daraz example ২: "delivery_time"

  • Customer satisfaction prediction।
  • Order placement-এ scoring; delivery_time at-time unknown।
  • Use: estimated_delivery_time at order-time, not actual।

Daraz example ৩: "return_status"

  • Refund prediction-এ "return_status = returned"।
  • Tautological — returned items are returned।
  • Training accuracy 100%, no business value।

Daraz example ৪: "next_login_time"

  • Engagement prediction-এ — future login time encode।
  • Inference-এ unavailable।
  • Subtle if not careful with timestamps।

Detection methods:

(১) Suspiciously high accuracy

  • Baseline AUC 0.7, model 0.99 — investigate।
  • Real-world tasks rarely achieve perfection।

(২) Feature importance review

  • Top feature definition examine।
  • "How is this computed? When?"
  • Domain expert validate।

(৩) Time-based train-test split

  • Random split — leakage hides।
  • Time-based: train pre-2024, test 2024+।
  • Production-mimic — leakage exposed।

(৪) Production scoring simulation

  • "Score this hypothetical row" — feature available কি?
  • Each feature timestamp manually verify।

(৫) Forward-only feature engineering

  • Each feature-এর creation-time documented।
  • Target-time strictly before feature-time required।
  • Feature store enforce।

Prevention:

  • Snapshot data at scoring-time।
  • Time-aware feature store।
  • Cohort-based training: "users joined Jan-Jun 2024, target = active in Dec 2024"।
  • Pre-window aggregations: "transactions in last 7 days BEFORE target window"।

Tools:

  • Feast, Tecton — feature store with time-travel।
  • MLflow — feature lineage track।
  • Custom: timestamp-aware ETL।

মূল উপলব্ধি: Feature leakage = "tomorrow's news in today's headline"। Detection-এ চোখ-চক্ষু skeptical থাকতে হয়। প্রতিটি feature-এ "এটা কখন available?" প্রশ্ন। Senior ML engineer-এর mark — leakage paranoia।

অনুশীলন

  1. SelectKBest: Sklearn-এর Iris-এ mutual information দিয়ে top-2 feature বাছুন। কোনগুলো? কেন?
    from sklearn.datasets import load_iris
    from sklearn.feature_selection import SelectKBest, mutual_info_classif
    X, y = load_iris(return_X_y=True)
    sel = SelectKBest(mutual_info_classif, k=2).fit(X, y)
    print(load_iris().feature_names)
    print(sel.get_support())

    সাধারণত "petal length" ও "petal width" — species-classification-এ এই দু'টি সবচেয়ে discriminative। Sepal-এর variability species-এ overlap বেশি।

  2. Lasso path: Diabetes dataset-এ Lasso fit করে কোন alpha-এ কত feature ০-এ — visualize।
    from sklearn.datasets import load_diabetes
    from sklearn.linear_model import lasso_path
    import numpy as np
    X, y = load_diabetes(return_X_y=True)
    alphas, coefs, _ = lasso_path(X, y, alphas=np.logspace(-3, 1, 20))
    print("Alpha → number of zero coefficients")
    for a, c in zip(alphas, coefs.T):
        print(f"  {a:.4f}: {(np.abs(c) < 1e-5).sum()}")

    Alpha বাড়ানোর সাথে সাথে — zero coefficient সংখ্যা বাড়ে। CV-তে optimal alpha।

  3. Permutation importance: RandomForest-এ default vs permutation importance compare করুন। কোনটি বেশি stable?
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.inspection import permutation_importance
    from sklearn.datasets import load_breast_cancer
    X, y = load_breast_cancer(return_X_y=True)
    rf = RandomForestClassifier(random_state=0).fit(X, y)
    print("MDI top 5:", sorted(zip(rf.feature_importances_, range(30)), reverse=True)[:5])
    perm = permutation_importance(rf, X, y, n_repeats=10, random_state=0)
    print("Perm top 5:", sorted(zip(perm.importances_mean, range(30)), reverse=True)[:5])

    সাধারণত top-features দু'টিতে similar; কিন্তু rank ও magnitude আলাদা। Cardinality-affected feature MDI-তে inflated।

আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

কোড রানার কাজ না করলে? Google Colab ব্যবহার করুন।
পূর্ববর্তী পাঠ
পাঠ ২০ · Feature scaling