Feature engineering ভিত্তি
এই পাঠে যা শিখবেন
- Numerical features — scaling ও transformation
- Categorical features — encoding strategies
- Date/time features — cyclical encoding
- Interaction features — power of combination
- Pipeline — leakage-free preprocessing
১ · কেন feature engineering এত গুরুত্বপূর্ণ
Andrew Ng-এর famous quote: "Coming up with features is difficult, time-consuming, requires expert knowledge. Applied machine learning is basically feature engineering."
মডেল raw data থেকে শেখে — কিন্তু "raw" features সবসময় ভাল signal না। একই ডেটায় — চমৎকার features-এ সরল মডেল > weak features-এ XGBoost।
১) Algorithmic requirement: SVM, KNN scaling demand করে। Tree-এর কম matter।
২) Domain knowledge inject: "BMI = weight/height²" — derived feature direct insight।
৩) Linear ↔ Non-linear bridge: $x \to (x, x^2, \log x)$ — linear model-কে non-linear ক্ষমতা।
২ · Numerical scaling
Feature-এ ব্যাপক range mismatch — "age" ০-১০০, "income" ০-১০M। SVM/KNN/NN — ব্যাপক feature dominate। তিন strategy:
-
StandardScaler (z-score): $x' = (x - \mu) / \sigma$. Mean ০, std ১।
সাধারণত default — Gaussian-like data-এ optimal। -
MinMaxScaler: $x' = (x - x_{min}) / (x_{max} - x_{min})$. Range [০, ১]।
Bounded values-এ ভাল (যেমন pixel values)। -
RobustScaler: median ও IQR ব্যবহার। Outlier-robust।
Heavy-tailed distribution-এ preferred।
৩ · Skewed feature — log transform
Income, transaction amount, file size — heavy right-skew। মডেল উচ্চ values-এ dominated।
$$x' = \log(1 + x)$$
np.log1p(x) — ০ values handle করে। Distribution near-Gaussian হয়, model behavior improve।
৪ · Categorical encoding
"city = ঢাকা, চট্টগ্রাম, সিলেট" — মডেল string বুঝে না। Encoding দরকার:
-
One-hot encoding: প্রতি category-এর জন্য একটি binary column। Low cardinality (<১০) preferred।
Cons: high cardinality-এ dimension explosion। -
Label encoding: 0, 1, 2, ... order ধরায় — যা arbitrary হলে problematic।
Tree-based model OK; linear model-এ avoid। - Target encoding: Category-এর target mean। High-cardinality (e.g., zip code, user_id)-এ powerful — কিন্তু leakage-prone।
- Frequency encoding: Category frequency replace। Simple, effective।
- Embedding: Deep learning-এ — learnable dense vector। Massive cardinality-এ scalable।
৫ · Date/Time features
"timestamp = 2026-05-07 14:30:00" — single column থেকে অনেক features:
- Components: year, month, day, hour, minute।
- Day of week: 0-6। Pattern — weekend behavior।
- Is weekend, is holiday — boolean flags।
- Time since: "last purchase থেকে কতদিন"। Recency বিশাল signal।
- Cyclical encoding: hour as $(\sin(2\pi h/24), \cos(2\pi h/24))$। ২৩ ও ০ কাছাকাছি — direct integer-এ এটা দেখা যায় না।
৬ · Interaction features
দু'টি feature combine — মডেলের জন্য সরাসরি signal:
- Multiplication: $x_1 \times x_2$ — "high earner + high spender" combination।
- Ratio: debt/income, expense/revenue।
- Difference: price - market_avg।
- Polynomial: $x, x^2, x^3$ — sklearn-এ
PolynomialFeatures।
Tree-based model — interaction automatic শেখে। Linear model — manually create করতে হবে।
৭ · scikit-learn-এ practical pipeline
import pandas as pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
# Synthetic — Daraz customer churn
np.random.seed(42)
n = 1000
df = pd.DataFrame({
'age': np.random.randint(18, 70, n),
'income': np.random.exponential(30000, n),
'days_since_last_buy': np.random.exponential(30, n),
'city': np.random.choice(['ঢাকা', 'চট্টগ্রাম', 'সিলেট'], n),
'device': np.random.choice(['mobile', 'desktop'], n)
})
df['churn'] = (df['days_since_last_buy'] > 60).astype(int)
X = df.drop('churn', axis=1)
y = df['churn']
# Numerical ও categorical features আলাদা handle
num_cols = ['age', 'income', 'days_since_last_buy']
cat_cols = ['city', 'device']
num_pipe = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler())
])
cat_pipe = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('onehot', OneHotEncoder(handle_unknown='ignore'))
])
preprocessor = ColumnTransformer([
('num', num_pipe, num_cols),
('cat', cat_pipe, cat_cols)
])
clf = Pipeline([
('prep', preprocessor),
('model', LogisticRegression())
])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
clf.fit(X_tr, y_tr)
print(f"Test accuracy: {clf.score(X_te, y_te):.4f}")৮ · Missing values handling
- Drop rows: Few missing — okay। বেশি — data loss।
- Mean/median imputation: Simple, default।
- Mode categorical-এ।
- Missing indicator: "is_missing" boolean column যোগ — sometimes powerful signal।
- Model-based: KNN imputer, IterativeImputer।
৯ · Feature selection
সব feature ভাল না — কিছু noise/redundant:
- Filter methods: correlation, mutual information, χ² test।
- Wrapper methods: Recursive Feature Elimination (RFE)।
- Embedded methods: Lasso (L15) — automatic selection।
- Importance-based: Tree feature importance (L26)।
১০ · Common mistakes
- Test set-এ scaling fit: Train statistics use করতে হবে।
- Target leakage feature: "next month sales" predict করতে "this month total sales"।
- Over-engineered features: ৫০০ features — ১০০০ samples-এ overfit।
- Domain-blind transformation: "age" log-transform — meaningful না।
- Encoding leakage: Target encoding-এ training set-এর target mean test-এ আসা।
ভাবনার প্রশ্ন
প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।
প্র ০১ "Deep learning has killed feature engineering" — এই দাবি কতটা সত্য, কতটা ভুল? কোন domain-এ feature engineering এখনো dominant?
এটি ML-এর সবচেয়ে hotly debated প্রশ্নের একটি। Reality nuanced — context-dependent।
Deep learning কোথায় feature engineering-কে replace করেছে:
(১) Image:
- ২০১২ AlexNet-এর আগে — SIFT, HOG, color histograms manually engineered।
- CNN — convolution layers automatic features শেখে — edge → shape → object।
- Hand-crafted features-এর accuracy কয়েক gen-এ surpassed।
(২) Text:
- BoW, TF-IDF, n-grams — classical features।
- Word2Vec → BERT → GPT — embedding learn করে।
- Manual feature engineering largely obsolete in NLP।
(৩) Speech:
- MFCC, spectrogram হাত-করা features।
- End-to-end deep model raw waveform-এ।
কোথায় Feature engineering Still dominate:
(১) Tabular data:
- Kaggle dominance এখনও XGBoost/LightGBM with engineered features।
- TabNet, NODE — DL alternatives — competitive কিন্তু not dominant।
- Banking, healthcare, e-commerce — tabular feature engineering critical।
(২) Time series:
- Lag features, rolling statistics, seasonal decomposition।
- Transformers helpful কিন্তু engineered features competitive।
(৩) Mixed-modal small data:
- Medical records — ১০K patient।
- DL data-hungry — feature engineering more sample-efficient।
(৪) Domain-knowledge-rich:
- Physics simulation, drug discovery — domain priors-এর benefit।
- "Feature × physics knowledge" superior to "raw + DL"।
(৫) Constrained inference:
- Edge devices, real-time — small model + good features।
- DL model size constraint।
Hybrid approaches:
- "Feature engineering + DL" — neural network input-এ engineered features।
- Embedding learning + manual features concat।
- Production-এ common।
Modern reality:
- Foundation models (GPT-4, CLIP) — generic representation।
- Fine-tuning + minimal engineering — good results often।
- "Prompt engineering" — feature engineering's modern cousin।
Skill perspective:
- Junior — feature engineering taught।
- Senior — knows when to use, when to skip।
- Domain knowledge always valuable।
Future trends:
- AutoML — automatic feature engineering।
- FT-Transformer, TabNet — DL on tabular advancing।
- Foundation models for tabular research active।
মূল উপলব্ধি: "Killed" overstatement। DL replaced feature engineering in unstructured domains। Tabular, mixed-modal, domain-rich — feature engineering live and well।
প্র ০২ "Target encoding" powerful কিন্তু leakage-prone — কেন? Cross-validated target encoding কীভাবে কাজ করে?
Target encoding — Kaggle-এর secret weapon, কিন্তু careless use করলে disaster। Mechanism বুঝা vital।
Target encoding কী:
- Categorical column → category-এর target mean।
- "city = ঢাকা" → ঢাকা residents-এর গড় churn rate।
- উদাহরণ: ঢাকা=০.১২, চট্টগ্রাম=০.১৮, সিলেট=০.০৮।
- One-hot vs target encoding: one column instead of K।
- High-cardinality (১০০০ zip codes)-এ powerful।
কেন powerful:
- Direct correlation with target।
- Smooth representation।
- Less sparse than one-hot।
- Tree-based model — easy split।
Naive approach — leakage problem:
- Whole training set-এ target mean compute।
- সেই encoding train sample-এ apply।
- সমস্যা: Sample-এর own target encoding-এ leak করে।
- Train accuracy artificially high; test-এ collapse।
উদাহরণ:
- "city = সিলেট" — ১০ samples, ৩ churn।
- Target encoding = ০.৩।
- একটি specific সিলেট sample (churn=১) — সেই sample-এর encoding-এ self-included।
- Model overfits this leakage।
Cross-validated solution:
- Train data-কে K folds-এ ভাগ।
- Fold $i$-এর encoding compute হয় বাকি $K-1$ folds থেকে।
- প্রতি sample-এর encoding — its own fold-এর information ছাড়াই।
- Test set encoding — full training set থেকে।
Smoothing — small categories:
- Category ৫ samples — mean unreliable।
- Smoothed: $\frac{n \cdot \bar{y}_{\text{cat}} + m \cdot \bar{y}_{\text{global}}}{n + m}$
- $m$ = smoothing parameter; small $n$ → global mean closer।
- "Bayesian" interpretation।
Production considerations:
- Test/production-এ unseen category — global mean fallback।
- Statistics save করতে হয় — at deployment।
- Periodic update — drift detection।
Time-aware target encoding:
- Time-series-এ — encoding past data-এর basis-এ।
- Future leakage avoid।
scikit-learn-এ:
- 1.3+ —
TargetEncoderbuilt-in (cross-validated)। - category_encoders library — alternatives (CatBoost encoder, James-Stein)।
Alternative — CatBoost approach:
- "Ordered target statistics" — temporal order-এ encoding।
- Each sample-এর encoding শুধু earlier samples থেকে।
- Implicit cross-validation।
When NOT to use:
- Low cardinality (<১০) — one-hot simpler, equal performance।
- Few samples per category — unreliable means।
- High target variance within category — encoding loses info।
মূল উপলব্ধি: Target encoding "free lunch" না। Properly implemented (CV + smoothing) — powerful। Naive — silent disaster।
প্র ০৩ "Cyclical encoding" এ "hour" কে $(\sin, \cos)$ pair-এ encode কেন? "Day of week" — কি একই treatment?
Cyclical encoding — subtle technique। Many practitioners miss; ফলস্বরূপ wrong feature representations।
সমস্যা — naive integer encoding:
- Hour 0-23 — integer।
- Linear model বলে — "hour 23 ≈ hour 24 ≈ ০"।
- কিন্তু integer 23 ও 0 — ২৩ unit apart!
- মডেল বলে — "midnight ও 11pm completely different"।
- আসলে — adjacent। Cyclic feature-এ wraparound miss।
Cyclical encoding solution:
$$x_{\sin} = \sin\left(\frac{2\pi h}{24}\right), \quad x_{\cos} = \cos\left(\frac{2\pi h}{24}\right)$$
- প্রতি hour — circle-এ একটি point।
- Hour 0 ও 23 — close in $(\sin, \cos)$ space।
- Hour 0 ও 12 — opposite (max distance)।
- Distance — actual cyclic distance reflect।
কেন দু'টো values (sin AND cos):
- Single sin value — multiple hours-এ same (sin(π/4) = sin(3π/4))।
- Cos যোগ করলে — unique identification।
- Together — 2D embedding of cyclic feature।
Day of week — হ্যাঁ cyclic, কিন্তু:
- Sunday → Saturday — strict cyclic? — argument both ways।
- Some cultures Sunday start, others Monday।
- Weekend ↔ weekday — দু'টি distinct group।
Better alternatives for day-of-week:
- One-hot — ৭ binary columns। Tree-based-এ optimal।
- Cyclical — linear model-এ ভাল।
- is_weekend binary feature — সাধারণত প্রধান signal।
- Combination — all three।
অন্য cyclic features:
- Month (১-১২): January ও December close — cyclical encoding।
- Day of month (১-৩১): Less obvious cycle। Sometimes useful।
- Wind direction (০-৩৬০°): Classic cyclical।
- Latitude/longitude: Spherical — different encoding।
Tree-based vs Linear:
- Tree: Splits learn cycle implicit। Cyclical encoding rarely helps।
- Linear/SVM/NN: Cyclical encoding important।
- NN: Embedding learn করতে পারে — cyclical encoding pre-coded helpful।
Common mistakes:
- Sin/cos দু'টোর মাত্র একটি use।
- Wrong period (24 instead of 12 for am/pm)।
- Tree model-এ apply (unnecessary)।
- Forgetting boundary effects (Dec 31 → Jan 1)।
Beyond simple cycles:
- Multiple periods: Hour-of-day + day-of-week + month — interaction।
- Fourier features — higher-frequency components।
- "hour modulo 6" — work shift patterns।
Real-world examples:
- Restaurant dinner orders peak 7-9pm — wraparound capture critical।
- Shift workers — 11pm-7am pattern।
- Bike-sharing demand — clear cyclical।
Beyond ML:
- Statistics — circular statistics field।
- Physics — angular variables।
- Astronomy — celestial coordinates।
মূল উপলব্ধি: Cyclical features need cyclical encoding — particularly linear/distance-based models। Tree models often robust without। Domain knowledge → right transformation।
প্র ০৪ "Garbage in, garbage out" — feature engineering-এর philosophical principle। কিন্তু "good features" identify-এ scientific approach কী? আপনার diagnostic checklist?
Feature evaluation — art + science। Production ML practitioner-দের differentiator।
Good feature-এর ৪ properties:
- Predictive: Target-এর সাথে correlation।
- Stable: Time-এর সাথে stable distribution।
- Available: Production-এ inference time-এ available।
- Interpretable: Domain expert-এর কাছে meaningful।
Diagnostic checklist:
(১) Univariate analysis:
- Distribution plot — histogram, boxplot।
- Missing values — count ও pattern।
- Outliers — visual inspection।
- Skewness, kurtosis।
(২) Target relationship:
- Numerical-numerical: Pearson, Spearman correlation। Scatter plot।
- Numerical-categorical: ANOVA, Box plot per class।
- Categorical-categorical: Chi-square, contingency table।
- Mutual information: Non-linear dependencies catch।
(৩) Feature-feature relationship:
- Correlation matrix — multicollinearity।
- VIF (Variance Inflation Factor)।
- Redundant features — drop one।
(৪) Stability tests:
- Train vs test distribution — KS test।
- Time-based split — feature stable over time?
- Population shift indicator (PSI)।
(৫) Information value (IV):
- Banking/credit standard।
- $IV = \sum (\text{Good\%} - \text{Bad\%}) \times \ln(\text{Good\%}/\text{Bad\%})$
- IV < ০.০২ — useless।
- IV ০.০২-০.১ — weak।
- IV ০.১-০.৩ — medium।
- IV ০.৩+ — strong।
- IV > ০.৫ — suspicious leakage।
(৬) Model-based importance:
- Tree feature importance (L26)।
- SHAP values — model-agnostic।
- Permutation importance — robust।
(৭) Ablation study:
- Remove feature, retrain, measure performance drop।
- "True contribution" measure।
- Computationally expensive but gold standard।
(৮) Production checks:
- Latency — feature compute time।
- Availability — every prediction time।
- Cost — data acquisition / API calls।
- Compliance — privacy, regulation।
Feature lifecycle:
- Brainstorm domain expert সাথে।
- Quick validation — basic correlation।
- Implementation — production-grade computation।
- Model integration — performance impact।
- Production deployment — monitoring।
- Periodic re-evaluation — drift।
Common pitfalls:
- Suspicious high correlation: Probably leakage।
- Time leakage: Feature collected after target known।
- Survivorship bias: Training set biased — feature appears predictive।
- Confounding: Feature correlates with target through hidden variable।
Practical workflow:
- Start: domain experts + EDA।
- Generate: 50-100 candidate features।
- Filter: univariate IV, correlation।
- Combine: top features-এ interaction।
- Model: LightGBM with all।
- Reduce: importance + ablation।
- Validate: held-out, time-shift।
- Deploy: monitor drift।
Qualitative wisdom:
- "Feature ratios stable across populations" — generalize।
- "Aggregations over time-windows" — robust।
- "Recent + historical" — both included।
- "Domain knowledge > algorithmic discovery"।
Tools:
- EDA: pandas-profiling, sweetviz।
- Feature engineering: featuretools, tsfresh।
- Importance: SHAP, eli5।
- Monitoring: Evidently, WhyLabs।
মূল উপলব্ধি: "Good feature" definition multi-dimensional। Single test যথেষ্ট না। Systematic evaluation + domain expertise + production reality।
অনুশীলন
-
চিনুন: নিচের প্রতিটি raw feature — কী transformation দরকার?
- (ক) "monthly_income" — range ০ থেকে ১০M
- (খ) "city" — ৬৪ districts of Bangladesh
- (গ) "transaction_time" — timestamp
- (ঘ) "user_id" — ১০M unique values
- (ক) log1p transform + StandardScaler — heavy skew।
- (খ) High cardinality — target encoding বা frequency encoding। One-hot ৬৪ columns acceptable।
- (গ) Components extract: hour (cyclical sin/cos), day-of-week (one-hot), is_weekend, time_since_previous।
- (ঘ) Embedding (DL) বা feature aggregation per user (mean, count, recency)। Direct one-hot impossible।
-
scikit-learn: উপরের pipeline চালান। ColumnTransformer-এ একটি new feature add করুন — "log_income" + interaction "age × days_since_last_buy"।
FunctionTransformer use করে log। Custom transformer তৈরি interaction-এর জন্য। Pipeline-এ insert:
('log', FunctionTransformer(np.log1p), ['income'])। Performance compare। -
চিন্তা: Bkash transaction fraud detection — ৫টি engineered features design করুন (raw transaction data থেকে)।
- (১) Amount-z-score (per user-এর mean থেকে standard deviations)।
- (২) Time-since-last-transaction।
- (৩) Frequency-last-1-hour (velocity check)।
- (৪) New-recipient flag (first time pay this person)।
- (৫) Time-of-day cyclical encoding।
- আরো — geographic distance from usual location, device change, day-of-week pattern।
আরও পড়ুন
- পাঠ ০৯ · Linear Regression — শূন্য থেকে পরবর্তী মডিউল M2 Engineered features-এ প্রথম মডেল।
- পাঠ ০৭ · Confusion matrix আগের পাঠ Diagnose → কোন features add করব।
- পাঠ ৪২ · Pipeline ও Deployment পরবর্তী মডিউল Production-grade feature pipeline।
- সব AI Courses ABCL TECH Python, ML, DL, NLP, CV, GenAI, RL, MLOps।