Feature selection
এই পাঠে যা শিখবেন
- Feature selection-এর তিন family — filter, wrapper, embedded
- Correlation, mutual information, RFE, Lasso, tree importance
- Selection-এ leakage এড়ানোর pattern
- "Garbage feature" vs "redundant feature" পার্থক্য
- Bangladesh project-এ practical workflow
১ · কেন feature selection
Feature selectionFeature Selectionসব feature থেকে সবচেয়ে relevant subset বাছা — যাতে model simpler, faster, and often more accurate হয়। Filter, wrapper, embedded — তিন family। ML-এর "less is more" principle। Cause:
- Curse of dimensionality: high-D-এ generalization কঠিন।
- Overfitting: noisy feature মডেলকে memorize করায়।
- Train time: kth feature add → time linear-quadratic বাড়ে।
- Interpretability: ১,০০০ feature-এর model ব্যাখ্যা অসম্ভব।
- Production cost: প্রতি feature compute, store, monitor — infrastructure।
- Latency: real-time inference-এ feature collection time-critical।
Bangladesh fintech-এ একটি transaction-time scoring 100ms-এর কম দরকার। ৫০০ feature compute করলে সম্ভব না। ৩০-৫০ select করতে হয়।
২ · তিন family
১) Filter: মডেল-independent — feature-target relationship measure।
২) Wrapper: মডেল-train করে subset compare।
৩) Embedded: মডেল training-এর সাথেই selection।
৩ · Filter methods
প্রতিটি feature-এ target-এর সাথে statistical metric। Threshold-এ কম-গুরুত্বপূর্ণ drop।
- Correlation (Pearson): numeric-numeric। $|r| < 0.05$ — drop।
- Spearman correlation: rank-based; non-linear monotonic।
- Mutual Information: non-linear too; categorical ও numeric দু'টোতে।
- Chi-square test: categorical-categorical।
- ANOVA F-test: categorical target, numeric feature।
- Variance threshold: low-variance (constant-প্রায়) drop।
Mutual Information:
$$I(X; Y) = \sum_{x, y} p(x, y) \log \frac{p(x, y)}{p(x) p(y)}$$
$I = 0$ — independent। $I > 0$ — কিছু relationship। Sklearn-এ mutual_info_classif।
সুবিধা: দ্রুত, scalable, model-agnostic।
সীমা: univariate — feature-feature interaction miss। দু'টি feature individually weak কিন্তু together strong — drop হয়ে যেতে পারে।
৪ · Wrapper methods
Subset দিয়ে model train, performance দিয়ে compare। Computationally expensive কিন্তু accurate।
- Forward selection: empty থেকে শুরু — প্রতিটি step-এ best feature add।
- Backward elimination: all থেকে শুরু — প্রতিটি step-এ worst drop।
- Recursive Feature Elimination (RFE): model train, lowest importance drop, repeat।
Sklearn: RFE(estimator, n_features_to_select=k)। Cross-validated version: RFECV।
Cost: $n$ feature, $k$ select → $O(n \cdot k \cdot \text{train\_time})$।
৫ · Embedded methods
(ক) Lasso (L1 regularization):
$$\min_w \|y - Xw\|_2^2 + \lambda \|w\|_1$$
L1 penalty কিছু coefficient exactly ০-এ pull করে। সেই feature automatically dropped।
Sklearn: Lasso(alpha=0.1)। Ridge (L2) — coefficient shrink করে কিন্তু ০ করে না।
(খ) Tree feature_importance:
- প্রতিটি feature-এ training-এ split-এ contribution।
- RandomForest, XGBoost, LightGBM সবারই
.feature_importances_attribute। - Threshold-এ low-importance drop।
(গ) SHAP values:
- Game-theory based explanation।
- Each feature-এ row-level contribution।
- Global feature ranking।
সুবিধা: model-aware; interactions ধরে; production-ready।
Caveat: tree importance correlated feature-এ unstable; SHAP slow।
৬ · Multicollinearity ও VIF
দু'টি feature highly correlated হলে — redundant। Linear model coefficient unstable। VIF (Variance Inflation Factor):
$$VIF_i = \frac{1}{1 - R^2_i}$$
যেখানে $R^2_i$ — feature $i$-কে অন্য সব দিয়ে predict-এর R²। VIF > ১০ — drop consider।
৭ · Leakage-এ সাবধানতা
- Selection train-set-এ: full data-তে selection → test info leak।
- Cross-validation pipeline: প্রতিটি fold-এ selection নতুন।
- Time-series: walk-forward selection।
- Hyperparameter (e.g., k in SelectKBest): nested CV-এ tune।
৮ · Filter — sklearn কোড
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, mutual_info_classif, f_classif
X, y = load_breast_cancer(return_X_y=True)
feature_names = load_breast_cancer().feature_names
# Top 10 features by mutual information
selector = SelectKBest(mutual_info_classif, k=10)
X_new = selector.fit_transform(X, y)
selected = [feature_names[i] for i in selector.get_support(indices=True)]
print(f"Total features: {X.shape[1]}")
print(f"Selected: {X_new.shape[1]}")
print(f"\nTop 10:")
for f, score in sorted(zip(feature_names, selector.scores_), key=lambda t: -t[1])[:10]:
print(f" {f}: {score:.3f}")
৯ · Lasso embedded selection
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler
X, y = load_diabetes(return_X_y=True)
names = load_diabetes().feature_names
X_scaled = StandardScaler().fit_transform(X)
lasso = LassoCV(cv=5, random_state=0).fit(X_scaled, y)
print(f"Best alpha: {lasso.alpha_:.4f}")
print(f"\nCoefficients:")
for n, c in zip(names, lasso.coef_):
flag = "✓" if abs(c) > 0.001 else "✗"
print(f" {flag} {n:10s}: {c:+.2f}")
n_selected = (np.abs(lasso.coef_) > 0.001).sum()
print(f"\nSelected: {n_selected}/{len(names)}")
১০ · Tree feature importance
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
names = load_breast_cancer().feature_names
rf = RandomForestClassifier(n_estimators=100, random_state=0).fit(X, y)
# Sort by importance
importance = sorted(zip(names, rf.feature_importances_), key=lambda t: -t[1])
print("Top 10 features by RF importance:")
for n, imp in importance[:10]:
print(f" {n:30s}: {imp:.4f}")
১১ · Bangladesh ML pipeline-এ feature selection
- bKash fraud (5000 features): filter (chi², MI) → 200; RFE → 50; XGBoost feature importance → 30 production।
- Daraz CTR: mostly tree-importance based — embedded efficient।
- Pathao surge: Lasso-cv pre-selected; linear model production।
- Microfinance loan default: domain-driven (income, history, demography); selection refinement।
১২ · Common pitfalls
- Selection on full data: leakage; train-only।
- Univariate-only: interaction-rich data-এ poor।
- Importance trust: correlated feature-এ unstable।
- Domain ignoring: stat-only — business knowledge integrate করুন।
- Future leakage: "customer_lifetime_value" target predict-এ — definition-এ leak।
ভাবনার প্রশ্ন
প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।
প্র ০১ "Filter univariate" এবং "interaction" — দু'টো feature individually unimportant কিন্তু together important। Concrete example দিন। কীভাবে detect?
এটি ML-এর সবচেয়ে subtle phenomenon — synergy বা interaction effect।
Concrete example ১: XOR pattern
- $x_1 \in \{0, 1\}$, $x_2 \in \{0, 1\}$।
- $y = x_1 \oplus x_2$ (XOR)।
- Truth table:
- (0,0) → 0, (0,1) → 1, (1,0) → 1, (1,1) → 0।
- $\text{corr}(x_1, y) = 0$, $\text{corr}(x_2, y) = 0$।
- Univariate filter — both drop।
- কিন্তু combined — perfectly predict।
Concrete example ২: Fraud detection
- "transaction_hour" alone — fraud rate uniform।
- "is_first_time_recipient" alone — slight elevated।
- Combined: "first-time recipient at 3am" — fraud rate ১০x।
- Univariate miss; interaction crucial।
Concrete example ৩: Healthcare
- Drug A — overall benefit zero (mean across population)।
- Drug A + gene marker B — strong benefit।
- "Personalized medicine" rests on interactions।
Bangladesh example: Microfinance default
- "Income" alone — moderate predictor।
- "Loan size" alone — moderate predictor।
- "Income / Loan-size ratio" — strong predictor।
- Univariate miss; engineered interaction key।
Detection methods:
(১) Tree-based importance
- Random Forest, XGBoost interaction natively capture।
- Feature importance still reflect both।
- Tree split — interaction implicitly modeled।
(২) SHAP interaction values
- SHAP feature-pair contribution explicitly।
shap.TreeExplainer.shap_interaction_values()।- Top interaction pairs identify।
(৩) Friedman's H-statistic
- Specifically interaction strength measure।
- 0 = no interaction, 1 = pure interaction।
sklearn.inspection.partial_dependence2D।
(৪) Polynomial features (manual)
PolynomialFeatures(degree=2, interaction_only=True)।- Pairwise multiplications create।
- Explosion: $n$ feature → $n^2$ — careful।
(৫) Domain-driven feature engineering
- Best approach often।
- "income/expense ratio" engineered।
- "hour × is_weekend" interaction।
Recommendation pipeline:
- Domain expert review — likely interactions list।
- Engineer key interactions explicitly।
- Tree model train — auto-detect rest।
- SHAP analysis — surprising interactions reveal।
- Refine feature engineering iteratively।
Univariate filter — কোথায় usefulful:
- Initial screening — 5,000 → 500।
- Feature-wise garbage detection।
- Computational efficiency।
- But never sole criterion।
মূল উপলব্ধি: Univariate quick screen কাজে লাগে কিন্তু sufficient না। Tree model + SHAP interaction-aware। Best — domain knowledge দিয়ে interaction explicit বানানো।
প্র ০২ Lasso-এর L1 vs Ridge-এর L2 — coefficient-এর behavior কেন আলাদা? কেন L1 zero তৈরি করে কিন্তু L2 না?
এটি ML-এর সবচেয়ে elegant geometric insight গুলোর একটা। Optimization landscape বুঝলে — intuitive।
Loss formulation:
Lasso: $L = \|y - Xw\|_2^2 + \lambda \|w\|_1 = \sum (y_i - \hat{y}_i)^2 + \lambda \sum |w_j|$
Ridge: $L = \|y - Xw\|_2^2 + \lambda \|w\|_2^2 = \sum (y_i - \hat{y}_i)^2 + \lambda \sum w_j^2$
Constraint formulation (equivalent):
Lasso: minimize MSE subject to $\sum |w_j| \le t$ (L1 ball)।
Ridge: minimize MSE subject to $\sum w_j^2 \le t$ (L2 ball)।
Geometric picture:
- MSE contours — ellipses on (w₁, w₂) plane।
- Lasso constraint — diamond (rotated square)।
- Ridge constraint — circle।
- Optimal w — যেখানে contour first touches constraint।
Why diamond produces zeros:
- Diamond-এর "corner" axis-এ — যেমন (t, 0) corner।
- Ellipse এই corner-এ touch করার সম্ভাবনা বেশি (corner sharp)।
- Touch on axis = one coordinate exactly 0।
- 2D-এ এই argument; high-D-এ corner আরও বেশি, zero-feature বেশি।
Why circle does not produce zeros:
- Circle smooth — কোনো corner নেই।
- Ellipse touch generic point-এ — both coordinates non-zero।
- Coefficient shrink হয় কিন্তু ০ হয় না।
Mathematical proof — subdifferential:
- $|w|$-এর derivative ০-এ undefined; subdifferential $[-1, 1]$।
- Optimum-এ KKT condition: gradient ∈ subdifferential।
- Range $[-1, 1]$ allow $w^* = 0$ — exact sparsity।
- $w^2$-এর derivative ০-এ ০ — exact sparsity rare (only by chance)।
Practical implications:
Lasso:
- Automatic feature selection।
- Sparse model — production-friendly।
- High-D-এ ($n < p$): Lasso essential।
- Issue: correlated feature-এ random selection (ভাল feature drop, similar-correlated keep)।
Ridge:
- All feature retain — small weight।
- Multicollinearity handle ভাল (stable coefficient)।
- Smooth regularization।
- Issue: feature selection না।
Elastic Net — best of both:
$$L = \text{MSE} + \lambda_1 \|w\|_1 + \lambda_2 \|w\|_2^2$$
- L1 + L2 mixed।
- Sparsity + stability।
- Sklearn:
ElasticNet(alpha=0.1, l1_ratio=0.5)। - Correlated feature group together select।
Hyperparameter tuning:
- $\lambda$ (alpha): $0 \to \infty$।
- $\lambda = 0$: no regularization।
- $\lambda \to \infty$: all weights → 0।
- CV-তে select:
LassoCV।
Bayesian interpretation:
- Lasso = MAP estimate with Laplace prior (sharp peak at 0)।
- Ridge = MAP with Gaussian prior।
- Laplace's sharp peak → coefficients pulled to 0।
Common applications:
- Genomics: 20K gene → handful predictive। Lasso natural।
- NLP: 100K word features → top hundred। Lasso।
- Bangladesh fintech: 5K transaction features → 30 production। Lasso first cut।
Best practices:
- Always scale features before Lasso/Ridge (penalty fair)।
- CV for alpha।
- Lasso path visualize:
lasso_path()— alpha varies, see coefficients enter/exit। - Stability selection: bootstrap + Lasso → reproducible feature set।
মূল উপলব্ধি: L1 vs L2 — geometry-driven behavior। Lasso natural feature selector; Ridge stabilizer। Elastic Net balance। Production-এ — Lasso first cut, then refine।
প্র ০৩ "Tree feature_importance" সাবধানে use করতে বলা হয়। কোন কোন বিকৃতি? Permutation importance ও SHAP কেন better?
Random Forest-এর built-in feature importance — quick কিন্তু flawed। Production-grade interpretability-এ সাবধানতা।
Default tree importance — Mean Decrease Impurity (MDI):
- প্রতিটি split-এ Gini/entropy reduction।
- সব tree-এ sum এ feature-এর contribution।
- Sklearn-এ
.feature_importances_default।
সমস্যা ১: High-cardinality bias
- Many-unique-value feature-এ split-options বেশি।
- Inflated importance — even if no real signal।
- Numerical feature continuous → many split options।
- Categorical feature few values → low split options।
- Numeric feature artificially favored।
Concrete example:
- Add a column "random_int_1_to_1000" — pure noise।
- RF importance often top-10!
- Cardinality fakes signal।
সমস্যা ২: Correlated features
- X1 ও X2 highly correlated; both predictive।
- Tree randomly chooses one for split।
- Importance distributed arbitrarily।
- "X1 important" run-1, "X2 important" run-2 — same data, different result।
সমস্যা ৩: Training data only
- MDI training-set splits-এ based।
- Generalization-এর reflection না।
- Overfitting-prone feature inflated importance।
Permutation importance — alternative:
- Train model।
- Test set-এ prediction performance baseline measure।
- একটি feature-এর values randomly permute।
- Performance drop measure।
- Drop = importance।
Permutation importance-এর সুবিধা:
- Test-set based — generalization reflect।
- Cardinality-bias কম।
- Model-agnostic — যেকোনো model-এ apply।
- Sklearn:
permutation_importance()।
Permutation-এর সমস্যা:
- Correlated feature-এ — permute X1, X2-ও deteriorate (still correlated to permuted)। Importance underestimate।
- Computationally expensive (per feature, per repeat)।
- Interpretation: "if this feature randomly garbage" — not "if removed"।
SHAP — Shapley values:
- Game-theory: cooperative game-এ player-এর fair contribution।
- প্রতিটি row-এ feature contribution prediction-এ।
- Summing across rows → global importance।
- Mathematically principled।
SHAP-এর সুবিধা:
- Local + global interpretation।
- Interaction effects capture।
- Direction (positive/negative) shown।
- Audit-friendly — regulatory standard।
- Visualizations (waterfall, beeswarm, dependence plot)।
SHAP-এর সমস্যা:
- Computationally expensive (TreeSHAP fast for trees)।
- Correlated feature still tricky (different background distribution issue)।
- Library:
shap।
Comparison summary:
- MDI (default): fast, biased, training-set।
- Permutation: slower, less biased, test-set।
- SHAP: slowest, most principled, local+global।
Production recommendation:
- Quick screening: MDI।
- Decision-making: Permutation (test set)।
- Audit/compliance: SHAP।
- Combine: cross-check three methods।
Bangladesh examples:
- Bangladesh Bank-এর regulatory loan-default model — SHAP audit।
- BFIU-এ AML model — explainability mandate।
- Daraz CTR — MDI quick, SHAP for product team explanation।
Common mistake:
- "Top-3 RF importance — these features matter" — without permutation check।
- Cardinality-biased decision।
- Stakeholder explanation misleading।
মূল উপলব্ধি: Default tree importance fast কিন্তু biased। Permutation + SHAP combination — robust feature insight। Interpretability ML ethics-এর foundation।
প্র ০৪ "Future leakage feature" বলতে কী বোঝায়? bKash/Daraz-এর context-এ ৩-৪ উদাহরণ দিন।
Feature leakage — production ML-এর সবচেয়ে devastating bug। Model দেখায় ৯৫% accuracy, production-এ ৬০% — কারণ training-এ "future" দেখা।
Definition:
- Feature যা target-এর সাথে সম্পর্কিত প্রক্রিয়ার পরে compute হয়।
- Inference time-এ unavailable।
- Training-এ available — leakage সৃষ্টি।
Categories:
(১) Time leakage
- Feature created after target event।
- Production-এ "এখন থেকে" আমরা future জানি না।
(২) Target leakage
- Feature target-কে directly বা proxy-তে contain।
- "is_blocked_due_to_fraud" feature → "is_fraud" target predict — circular।
(৩) Train-test leakage
- Test data train-এ peek (encoding, scaling)।
- আগের পাঠে discussed।
bKash example ১: "n_failed_login_attempts"
- Fraud prediction-এ — ৯৯% accuracy।
- কিন্তু — failed login fraud detection trigger করে; security team account flag করে।
- "n_failed_login" historical = pre-flag, modern = post-flag mixed।
- Training data-তে (post-fraud entries-এ ৫০ failed login) but production prediction time-এ (০ failed login) — vastly different distribution।
- Solution: only use pre-fraud-flag failed logins।
bKash example ২: "customer_complaint_count"
- Fraud detection-এ "had complaint" feature।
- Customer complaint = fraud detect করে ফেলে fast track।
- Complaint = consequence, not cause।
- Model "complaint exists → fraud" — but at scoring time, no complaint yet।
bKash example ৩: "lifetime_transaction_count"
- Churn prediction-এ feature।
- Lifetime = entire history, including AFTER target prediction window।
- Churned customer-এর "lifetime" small (closed early)।
- Active customer-এর "lifetime" growing।
- Trivial classifier।
- Solution: time-aware aggregation (last 30 days, fixed window)।
Daraz example ১: "review_count"
- Conversion prediction-এ feature।
- Review post-purchase লেখা হয়।
- "আমি কি কিনব?" predict-এর সময় — review count ০ (not yet purchased)।
- Training-এ: purchased → reviewed → review_count high → conversion target = 1।
- Trivial pattern।
Daraz example ২: "delivery_time"
- Customer satisfaction prediction।
- Order placement-এ scoring; delivery_time at-time unknown।
- Use: estimated_delivery_time at order-time, not actual।
Daraz example ৩: "return_status"
- Refund prediction-এ "return_status = returned"।
- Tautological — returned items are returned।
- Training accuracy 100%, no business value।
Daraz example ৪: "next_login_time"
- Engagement prediction-এ — future login time encode।
- Inference-এ unavailable।
- Subtle if not careful with timestamps।
Detection methods:
(১) Suspiciously high accuracy
- Baseline AUC 0.7, model 0.99 — investigate।
- Real-world tasks rarely achieve perfection।
(২) Feature importance review
- Top feature definition examine।
- "How is this computed? When?"
- Domain expert validate।
(৩) Time-based train-test split
- Random split — leakage hides।
- Time-based: train pre-2024, test 2024+।
- Production-mimic — leakage exposed।
(৪) Production scoring simulation
- "Score this hypothetical row" — feature available কি?
- Each feature timestamp manually verify।
(৫) Forward-only feature engineering
- Each feature-এর creation-time documented।
- Target-time strictly before feature-time required।
- Feature store enforce।
Prevention:
- Snapshot data at scoring-time।
- Time-aware feature store।
- Cohort-based training: "users joined Jan-Jun 2024, target = active in Dec 2024"।
- Pre-window aggregations: "transactions in last 7 days BEFORE target window"।
Tools:
- Feast, Tecton — feature store with time-travel।
- MLflow — feature lineage track।
- Custom: timestamp-aware ETL।
মূল উপলব্ধি: Feature leakage = "tomorrow's news in today's headline"। Detection-এ চোখ-চক্ষু skeptical থাকতে হয়। প্রতিটি feature-এ "এটা কখন available?" প্রশ্ন। Senior ML engineer-এর mark — leakage paranoia।
অনুশীলন
-
SelectKBest: Sklearn-এর Iris-এ mutual information দিয়ে top-2 feature বাছুন। কোনগুলো? কেন?
from sklearn.datasets import load_iris from sklearn.feature_selection import SelectKBest, mutual_info_classif X, y = load_iris(return_X_y=True) sel = SelectKBest(mutual_info_classif, k=2).fit(X, y) print(load_iris().feature_names) print(sel.get_support())সাধারণত "petal length" ও "petal width" — species-classification-এ এই দু'টি সবচেয়ে discriminative। Sepal-এর variability species-এ overlap বেশি।
-
Lasso path: Diabetes dataset-এ Lasso fit করে কোন alpha-এ কত feature ০-এ — visualize।
from sklearn.datasets import load_diabetes from sklearn.linear_model import lasso_path import numpy as np X, y = load_diabetes(return_X_y=True) alphas, coefs, _ = lasso_path(X, y, alphas=np.logspace(-3, 1, 20)) print("Alpha → number of zero coefficients") for a, c in zip(alphas, coefs.T): print(f" {a:.4f}: {(np.abs(c) < 1e-5).sum()}")Alpha বাড়ানোর সাথে সাথে — zero coefficient সংখ্যা বাড়ে। CV-তে optimal alpha।
-
Permutation importance: RandomForest-এ default vs permutation importance compare করুন। কোনটি বেশি stable?
from sklearn.ensemble import RandomForestClassifier from sklearn.inspection import permutation_importance from sklearn.datasets import load_breast_cancer X, y = load_breast_cancer(return_X_y=True) rf = RandomForestClassifier(random_state=0).fit(X, y) print("MDI top 5:", sorted(zip(rf.feature_importances_, range(30)), reverse=True)[:5]) perm = permutation_importance(rf, X, y, n_repeats=10, random_state=0) print("Perm top 5:", sorted(zip(perm.importances_mean, range(30)), reverse=True)[:5])সাধারণত top-features দু'টিতে similar; কিন্তু rank ও magnitude আলাদা। Cardinality-affected feature MDI-তে inflated।
আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ
- পাঠ ২২ · Time series basics পরবর্তী পাঠ EDA-র শেষ stop — temporal data।
- পাঠ ২০ · Feature scaling আগের পাঠ Scale-এর পর selection।
- পাঠ ১৬ · EDA এই পাঠের সাথে সম্পর্কিত EDA-তে candidate identify।
- সব AI Courses দেখুন ABCL TECH Python, ML, DL, NLP, CV, GenAI, RL, MLOps।