Feature scaling — standardize, minmax

এটি ML pipeline-এর সবচেয়ে subtle bug। Beginner-রা প্রায় সবাই এই ভুল করেন।

সঠিক pattern:

• scaler.fit(X_train) — শুধু train data দেখে mean, std compute।
• scaler.transform(X_train) — train data scale।
• scaler.transform(X_test) — same fitted parameter দিয়ে test scale।

ভুল pattern:

• scaler.fit_transform(X_test) — test-এর own statistics দেখে scale।
• বা: scaler.fit(X_full) তারপর train/test split — entire data থেকে statistics।

Leakage কেন:

• Test data future-এর প্রতিনিধি — production-এ সেই data আমরা জানি না।
• Test-এর mean/std দেখা = future থেকে info "leak" current model-এ।
• Real production-এ — incoming data scale-এর জন্য historical (train) statistics ব্যবহার।

Concrete example:

• Train income mean = ৩০,০০০। Test mean = ৫০,০০০ (different distribution)।
• Wrong: test-এ fit_transform — test-এর mean ০, std ১।
• Correct: train-এর scaler apply — test-এ mean ~০.৪, std ~১.২।
• Production-এ মডেল train scale expect করে; test-এ আলাদা fit মানে — model-এর input distribution train-এর সাথে align না।

Accuracy কীভাবে inflated:

(১) Subtle case — same scale

• Test ও train similar distribution হলে — leakage minor।
• Accuracy ০.১-১% inflated।
• Detection difficult।

(২) Severe case — distribution shift

• Train historical, test recent।
• Test-এ fit করে "look correct" — কিন্তু model production-এ fail।
• Apparent accuracy ৯৫%, production reality ৭০%।

(৩) Cross-validation leakage

• fit on full data, then split — fold-এর holdout-এর info-ও fit-এ entered।
• K-fold CV inflated।

Sklearn Pipeline-এর role:

• Pipeline([('scaler', ...), ('model', ...)]) — automatic correct।
• CV-এ প্রতিটি fold-এ scaler নতুন fit।
• Manual scaling করার দরকার নেই; Pipeline-এ wrap।

Time-series-এ আরও critical:

• Train: ২০২০-২০২৩, test: ২০২৪।
• ২০২৪-এর data দিয়ে fit = future leakage absolute।
• Walk-forward validation: প্রতিটি step-এ train পর্যন্ত fit।

Detection signs:

• Train accuracy ও test accuracy gap বড়।
• Cross-validation score > production score।
• Distribution drift evident।

Real-world consequences:

• Bangladesh fintech model: leakage-due to "fraud rate ৯৯%" claim — production ৭০%।
• Healthcare model: train-test confusion → FDA review reject।
• Reputation damage — ML team trust।

Best practices:

• Pipeline সর্বদা।
• Train/test split first, preprocess second।
• Save fitted scaler — production-এ reuse।
• Test-এ separate validation set।

মূল উপলব্ধি: Test = future। Future থেকে info current decision-এ leak করা = unethical + technically wrong। Pipeline + discipline = leakage-free ML।

এটি real production decision। Log + Standard প্রায় সব time better — কেন?

Skewed data-তে শুধু StandardScaler-এর সমস্যা:

• Distribution shape unchanged — শুধু shift ও rescale।
• Skewness preserved।
• Long tail এখনো long (just shifted)।
• Outlier still outlier।

Income-এর realistic distribution (Bangladesh):

• Median ~২৫,০০০ BDT।
• ৯০th percentile ~৭৫,০০০।
• ৯৯th percentile ~৩,০০,০০০।
• Top earner ~৫০,০০,০০০+।
• Long right tail — extreme skewness।

StandardScaler-এ:

• Mean ~৪০,০০০, std ~৩,০০,০০০ (extreme outlier inflate)।
• Most user z-score = (২৫০০০ - ৪০০০০)/৩০০০০০ = -০.০৫।
• সবাই মাঝামাঝি, কিন্তু extreme এ +১০, +২০।
• Linear/NN model-এ — extreme dominate gradient।

Log + StandardScaler:

• $\log(25000) = 10.13$, $\log(50000) = 10.82$, $\log(5000000) = 15.42$।
• Log-income-এর mean ~১১, std ~১.৫।
• Distribution near-Gaussian।
• Z-score বেশিরভাগ [-2, +2]; extreme +৩।
• Linear model — well-behaved।

Why log:

• Income-এ multiplicative structure: ১০x increase = comparable jump।
• Log additive-এ convert।
• "৫০K vs ৫০০K" — logically same step (10x) as "৫০০K vs ৫০০০K"।
• Linear coefficient interpretation: "log-income unit increase → y change" = "income 2.71x → y change"।

Mathematical justification — Lognormal distribution:

• Income often lognormal (multiplicative process)।
• Log-income normal — Gaussian assumptions hold।
• Linear regression OLS ভালো work।

When log-StandardScaler ভাল:

• Right-skewed data: income, transaction, population, prices।
• Multiplicative relationship।
• Log-normal distribution suspected।

When just StandardScaler enough:

• Already near-normal (e.g., heights, BMI)।
• Bounded data (e.g., percentages, ratings)।
• Already log-transformed at source।

When neither — RobustScaler:

• Outlier extreme but not necessarily skewed।
• Median + IQR robust।

Box-Cox alternative:

• PowerTransformer(method='box-cox') — log-এর generalization।
• $\lambda$ data-driven choice।
• Need positive values; Yeo-Johnson handles negative।

Practical recipe:

from sklearn.preprocessing import StandardScaler
import numpy as np

X_log = np.log1p(X_income)
scaler = StandardScaler().fit(X_log_train)
X_scaled = scaler.transform(X_log)

Or scikit-learn pipeline:

from sklearn.preprocessing import FunctionTransformer
pipe = Pipeline([
    ('log', FunctionTransformer(np.log1p)),
    ('scale', StandardScaler()),
    ('model', LogisticRegression())
])

Empirical check:

• Plot histogram before/after।
• Skewness before: 5+; after log: ~0।
• Q-Q plot — log-transformed data straight line near-Normal।

Bangladesh use cases:

• Microfinance loan amount: log মাস্ট।
• bKash transaction: log।
• Real estate price: log।
• Grameenphone usage MB: log।

মূল উপলব্ধি: StandardScaler "skew" সরায় না — শুধু center ও rescale। Skewed data → log first, scale second। দু'ধাপ pipeline financial/economic feature-এ standard।

মোটামুটি true, কিন্তু nuance আছে। Tree-based model-এর mathematical invariance কেন এবং কখন break হয়।

কেন tree scaling-অজ্ঞ:

• Decision tree-এর split: "feature ≤ threshold"।
• Threshold value-এ optimize হয়।
• $x \to a \cdot x + b$ — threshold automatically adjust।
• Information gain (or Gini) split-এ same।
• মডেল identical performance, identical structure।

Mathematical statement:

Tree model monotonic transformation-এর প্রতি invariant। $f(x)$ monotonic হলে — tree on $x$ ও tree on $f(x)$ same predictions।

Validation:

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)

rf1 = RandomForestClassifier(random_state=0).fit(X, y)
rf2 = RandomForestClassifier(random_state=0).fit(X_scaled, y)

print(f"Original: {rf1.score(X, y):.4f}")
print(f"Scaled: {rf2.score(X_scaled, y):.4f}")

Result: identical accuracy। Same random_state সব split same threshold-value (scaled adjusted)।

Exception ১: Categorical interaction

• One-hot-এ ০/১ values।
• Numeric "income" 100,000-এ — split গভীরে ঢুকে।
• One-hot column কম informative নয়, কিন্তু tree algorithm subtly biased toward continuous feature।
• Workaround: gradient boosting parameters (min_samples_split, min_child_weight) tune।

Exception ২: Regularization

• XGBoost reg_alpha (L1), reg_lambda (L2)।
• Penalty leaf weight-এ — leaf weight feature scale-এর সাথে correlated।
• Scaled feature-এ penalty more uniform।
• Marginal effect; usually negligible।

Exception ৩: Distance-based loss

• Standard XGBoost loss (binary cross-entropy, RMSE) scaling-অজ্ঞ।
• Custom distance-based loss (rare) সাজাতে পারে।

Exception ৪: Numerical precision

• Feature value 10^10 vs 10^-10 — float precision issue।
• Practical না; production data এত extreme rare।

Exception ৫: Skewness

• Tree-এ scaling-অজ্ঞ, কিন্তু log-transform-এ improvement আসতে পারে।
• Long tail-এ split point কম optimal — log compress করে।
• Tested: usually 0.5-1% AUC improvement।

Practical guidance for tree models:

• Don't waste time scaling — same accuracy।
• Skewness extreme হলে — log transform try।
• Categorical feature handling — native (LightGBM) বা target encoding।
• Missing — XGBoost native handle।

When NOT scaling helps tree pipeline:

• Faster training: no preprocessing step।
• Interpretability: original feature value visible in feature_importance।
• Debugging: raw value-এ understand splits।
• Production: simpler pipeline, less code।

Hybrid pipeline scenario:

• Stacking: tree (no scale) + linear model (scale)।
• Different preprocessing per branch।
• Feature pipeline complexity।

Production Bangladesh examples:

• Daraz CTR (XGBoost): no scaling।
• bKash fraud (LightGBM): no scaling।
• Pathao surge (linear+RF): scaling for linear, not RF।

Counterintuitive finding (research):

• Some papers report 0.1-0.3% improvement with scaling for XGBoost।
• Statistical noise + random_state variation।
• No reliable systematic gain।

মূল উপলব্ধি: Tree model scaling-এ time waste। Skewness থাকলে — log transform। Pipeline simplicity gain। অন্য model-এর সাথে compare-এ — same/better baseline।

তিনটি scaler-এর use case ভিন্ন। Decision criteria বুঝলে — সঠিক choice automatic।

StandardScaler — z-score

$$x' = \frac{x - \mu}{\sigma}$$

• Mean = 0, std = 1।
• Range unbounded।
• Normal-distributed-এ optimal।

Use case:

• Linear regression — Gaussian assumption।
• Logistic regression — gradient descent।
• SVM (linear, RBF) — margin computation।
• PCA — variance-based।
• Most ML default।

Limitation:

• Outlier-এ mean ও std affected।
• Skewed data-এ misleading।

Bangladesh example:

• Customer age (18-65, near-normal): StandardScaler perfect।
• Test score (0-100, near-normal): StandardScaler।
• BMI: StandardScaler।

MinMaxScaler — [0, 1]

$$x' = \frac{x - x_{min}}{x_{max} - x_{min}}$$

• Bounded।
• All values [0, 1]।
• Distribution shape unchanged।

Use case:

• Image pixel data (0-255 → 0-1)।
• Neural network input (sigmoid output natural)।
• Features must be bounded (e.g., embedding similarity)।
• Visualization (axes 0-1)।

Limitation:

• Outlier extreme: একটি extreme value সব others-কে [0, 0.001]-এ squash।
• New extreme value test-এ > 1 (out of [0,1] range)।

Bangladesh example:

• Image classification (Daraz product image): pixel/255।
• Daraz rating (1-5): MinMax → (rating-1)/4।
• Probability score (0-1): already normalized, MinMax identity।

RobustScaler — median ও IQR

$$x' = \frac{x - \text{median}}{IQR}$$

• Median ও IQR — outlier-এ stable।
• Range unbounded।
• Normal-এ standard-এর চেয়ে কম সাপ্ত (factor ~1.35×)।

Use case:

• Outlier-heavy data: income, transaction amount।
• Sensor data with malfunction।
• Financial returns (occasional crash)।

Limitation:

• Normal data-এ slightly less efficient than StandardScaler।
• Outlier preserve — drop করতে পারে না।

Bangladesh example:

• bKash transaction amount (extreme outlier): RobustScaler।
• BSEC stock returns: RobustScaler।
• Microfinance loan recovery: RobustScaler (default extreme)।

Side-by-side comparison example:

• Data: $[1, 2, 3, 4, 5, 100]$ (one outlier)।
• Mean = 19.2, std = 39.5।
• Median = 3.5, IQR = 2.75।

StandardScaler:

• $[1, 2, 3, 4, 5, 100] \to [-0.46, -0.44, -0.41, -0.39, -0.36, 2.04]$
• প্রায় সবাই -0.4 কাছাকাছি; একটি 2.0।

MinMaxScaler:

• $\to [0, 0.01, 0.02, 0.03, 0.04, 1.0]$
• বেশিরভাগ near-০; একটি ১।
• Extreme outlier domination।

RobustScaler:

• $\to [-0.91, -0.55, -0.18, 0.18, 0.55, 35.1]$
• Inlier-গুলো well-spread; outlier extreme।
• Best for inlier discrimination।

Decision flowchart:

1. Outlier আছে? → RobustScaler।
2. Bounded range দরকার (image, NN)? → MinMaxScaler।
3. Normal-distributed? → StandardScaler।
4. Skewed? → log + StandardScaler।

Empirical recommendation:

• Default: StandardScaler।
• Outlier > 5%: RobustScaler।
• Image/NN: MinMaxScaler।
• Highly skewed: PowerTransformer (Yeo-Johnson)।

Multi-scaler ensemble:

• প্রতিটি feature-এ different scaler — ColumnTransformer।
• income: log + StandardScaler।
• age: StandardScaler।
• rating: MinMaxScaler।
• Production code complexity বাড়ে কিন্তু performance optimal।

মূল উপলব্ধি: "একটি scaler সবকিছুতে" — naive approach। Feature-by-feature scaling decision — production ML excellence-এর pattern।