Ridge ও Lasso — regularization
এই পাঠে যা শিখবেন
- Overfitting recap, regularization-এর intuition
- Ridge vs Lasso — গণিত ও geometry
- Closed-form Ridge, Lasso-এর coordinate descent
- scikit-learn দিয়ে compare
১ · Overfitting recap
Linear regression-এ feature বেশি বা data কম হলে — model train data মুখস্থ করে। Test-এ pathological। Coefficients massive হয়ে যায় — $w_j = 10^4$ যেখানে natural scale ১০০।
Solution — explicit penalty on large $\mathbf{w}$। Loss-এ extra term:
$$L_{\text{regularized}} = L_{\text{data}} + \lambda \cdot R(\mathbf{w})$$
$\lambda$ — hyperparameter (regularization strength)। $R(\mathbf{w})$ — penalty function।
২ · Ridge regression (L2)
Penalty: $\|\mathbf{w}\|_2^2 = \sum_j w_j^2$।
$$L_{\text{Ridge}} = \frac{1}{N} \|\mathbf{y} - X\mathbf{w}\|^2 + \lambda \sum_j w_j^2$$
Closed-form:
$$\mathbf{w}_{\text{Ridge}} = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y}$$
$\lambda I$ — diagonal-এ যোগ — singular matrix সমস্যা solve, weights shrink।
৩ · Lasso regression (L1)
Penalty: $\|\mathbf{w}\|_1 = \sum_j |w_j|$।
$$L_{\text{Lasso}} = \frac{1}{N} \|\mathbf{y} - X\mathbf{w}\|^2 + \lambda \sum_j |w_j|$$
$|w_j|$ — at $w_j = 0$ non-differentiable। Closed-form নেই। Coordinate descent বা proximal gradient ব্যবহার হয়।
Magic: Lasso অনেক $w_j$-কে exactly $0$ করে। Automatic feature selection।
৪ · Geometric difference
Constrained optimization view: minimize MSE subject to $R(\mathbf{w}) \leq t$।
- Ridge: $\sum w_j^2 \leq t$ — circle (sphere)।
- Lasso: $\sum |w_j| \leq t$ — diamond (rotated square)।
MSE-র contour line এই shape-কে যেখানে স্পর্শ করে — সেটাই optimal $\mathbf{w}^*$। Diamond-এর corners — axis-এ। তাই Lasso solution often axis-এ — কিছু $w_j = 0$।
৫ · NumPy দিয়ে Ridge — closed-form
import numpy as np
np.random.seed(0)
N, n = 50, 20 # 50 samples, 20 features (overfit-prone)
X = np.random.randn(N, n)
true_w = np.zeros(n)
true_w[:5] = [1, -1, 2, -2, 0.5] # only 5 useful
y = X @ true_w + np.random.randn(N) * 0.3
# OLS — overfit
w_ols = np.linalg.lstsq(X, y, rcond=None)[0]
# Ridge — closed form
def ridge(X, y, lam):
n = X.shape[1]
return np.linalg.solve(X.T @ X + lam * np.eye(n), X.T @ y)
for lam in [0.01, 1.0, 10.0]:
w = ridge(X, y, lam)
err = np.sum((w - true_w) ** 2)
print(f"lambda={lam:5.2f}: ||w||²={np.sum(w**2):.3f}, "
f"recovery err={err:.3f}")
# Compare with OLS
print(f"\nOLS: ||w||²={np.sum(w_ols**2):.3f}, "
f"recovery err={np.sum((w_ols - true_w)**2):.3f}")৬ · sklearn — Ridge, Lasso, comparison
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
np.random.seed(0)
N, n = 100, 30
X = np.random.randn(N, n)
true_w = np.zeros(n)
true_w[:5] = [2, -1.5, 1, -0.5, 0.3]
y = X @ true_w + np.random.randn(N) * 0.5
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
# Scale
sc = StandardScaler()
X_tr = sc.fit_transform(X_tr)
X_te = sc.transform(X_te)
models = {
"OLS": LinearRegression(),
"Ridge": Ridge(alpha=1.0),
"Lasso": Lasso(alpha=0.1),
}
for name, m in models.items():
m.fit(X_tr, y_tr)
test_score = m.score(X_te, y_te)
nonzero = np.sum(np.abs(m.coef_) > 1e-4)
print(f"{name:6s}: R²={test_score:.4f}, non-zero coefs={nonzero}/{n}")৭ · $\lambda$ choice — cross-validation
from sklearn.linear_model import RidgeCV, LassoCV
import numpy as np
np.random.seed(0)
X = np.random.randn(100, 20)
y = X[:, 0] * 2 - X[:, 1] * 1.5 + np.random.randn(100) * 0.5
# Auto-tune lambda
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 50)).fit(X, y)
lasso_cv = LassoCV(alphas=np.logspace(-3, 3, 50), cv=5).fit(X, y)
print(f"Ridge best alpha: {ridge_cv.alpha_:.4f}")
print(f"Lasso best alpha: {lasso_cv.alpha_:.4f}")
# Lasso-এর non-zero coefficients
print(f"\nLasso non-zero features: {np.where(np.abs(lasso_cv.coef_) > 1e-4)[0]}")RidgeCV, LassoCV — automatic $\lambda$ selection via cross-validation। Production-এ এটাই default।
৮ · কখন কোনটি
- Ridge: সব feature contribute করে; multicollinearity exists; smooth coefficients চাই।
- Lasso: অনেক feature irrelevant ধারণা; feature selection automate; interpretable model।
- Elastic Net (L16): Ridge + Lasso combined — best of both।
৯ · Bayesian interpretation
Ridge — Gaussian prior on weights ($w_j \sim \mathcal{N}(0, \sigma^2)$) → MAP estimation।
Lasso — Laplace prior ($w_j \sim \text{Laplace}(0, b)$) → MAP estimation।
$\lambda$ — prior strength।
ভাবনার প্রশ্ন
প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।
প্র ০১ Lasso "exactly zero" coefficients দেয়, Ridge দেয় না — geometric proof কী? "Soft thresholding" কী?
Sparsity emergence — convex optimization-এর elegant result।
Subgradient analysis:
- $|w_j|$ at $w_j = 0$ — subgradient $[-1, 1]$।
- Optimality condition — gradient ০-এ হিট।
- $|\partial L_{\text{data}} / \partial w_j| < \lambda$ → $w_j = 0$ optimal।
Soft thresholding:
$$\text{soft}(z, \lambda) = \begin{cases} z - \lambda & z > \lambda \\ 0 & |z| \leq \lambda \\ z + \lambda & z < -\lambda \end{cases}$$
- Lasso coordinate descent এই operator use করে।
- $|z| \leq \lambda$ — exact ০।
- $|z| > \lambda$ — shifted।
Ridge analogous:
- $w_j^* = z / (1 + \lambda)$।
- "Hard thresholding" নেই।
- Just smooth shrinkage।
- ৭ → ৩.৫ — never exactly ০।
Geometric corner argument:
- Diamond corner — sparse $\mathbf{w}$।
- Circle smooth — interior optimal।
- Higher dims — more "axes" → more sparsity।
Empirical observation:
- Lasso — হাজার features-এ ১০-১০০ select করে।
- Ridge — সব features non-zero, কিন্তু many small।
- Different "philosophy" of regularization।
Implementation:
- Coordinate descent: Per-coord optimize, others fixed।
- LARS: Least Angle Regression — entire path।
- Proximal gradient: ISTA, FISTA — modern।
Compressed sensing connection:
- Donoho/Candes/Tao — sparse signal recovery।
- L1 — convex relaxation of L0।
- Theoretical guarantees (RIP)।
Limitations:
- Highly correlated features — Lasso unstable choice।
- $N < p$ — at most $N$ features selected।
- Group selection issues।
Variants:
- Adaptive Lasso: Variable-weighted L1।
- Group Lasso: Group-level sparsity।
- SCAD/MCP: Non-convex alternatives।
মূল উপলব্ধি: L1 sparsity — convex optimization gift। Geometry, gradient, statistics — multiple lenses একই conclusion। Lasso-র simple appearance — deep theoretical machinery।
প্র ০২ Bangladesh-এ একটি credit scoring মডেল — Ridge, Lasso, ও Elastic Net কোনটি? Regulatory-friendly কোনটি?
Banking ML — regulation এবং performance balance crucial।
Bangladesh Bank context:
- Capital adequacy guidelines।
- Model risk management framework।
- Audit trail required।
- Disparate impact testing।
Feature landscape:
- ~৫০-১০০ features typical।
- Demographic, financial, behavioral।
- Domain expert input critical।
- Feature dependencies common।
Model selection criteria:
(১) Predictive accuracy:
- AUC — primary metric।
- KS statistic — banking standard।
- Default rate prediction।
(২) Interpretability:
- Each feature's role explainable।
- Sign consistency (intuition)।
- Magnitude meaningful।
(৩) Stability:
- Coefficients stable across samples।
- Reproducibility।
- Drift monitoring possible।
Ridge analysis:
- + Smooth shrinkage।
- + All features used — no info loss।
- + Stable for correlated features।
- − Many small coefficients — interpretation noise।
- Suitable: established business, all features pre-validated।
Lasso analysis:
- + Automatic feature selection।
- + Sparse model — clear interpretation।
- + Reduced operational complexity।
- − Unstable with correlated features।
- − Random selection between equivalent features।
- Suitable: feature exploration, model debugging।
Elastic Net analysis:
- + L1 + L2 combined।
- + Group selection for correlated features।
- + Tunable balance।
- + Practically often best।
- − More hyperparameters।
- Suitable: production credit scoring।
Recommendation:
- First production: Logistic regression with L2 (Ridge equivalent)।
- Mature system: Elastic Net for stability + selection।
- Feature exploration: Lasso for hypothesis generation।
Regulatory considerations:
(১) Disparate impact:
- Sensitive features (gender, religion) avoid।
- Proxy detection — postal code → district demographics।
- L1 might select proxies — careful audit।
(২) Adverse action notice:
- Customers — top features for denial।
- Lasso — clear non-zero list।
- Ridge — many small features — less clear।
(৩) Stability requirement:
- Model retraining quarterly।
- Stable coefficients = trustworthy।
- Ridge wins on stability।
Validation framework:
- K-fold CV with temporal awareness।
- Out-of-time validation।
- Stress testing — economic scenarios।
- Backtesting on historical data।
Production architecture:
- Champion model: Elastic Net।
- Challenger: XGBoost (interpretability via SHAP)।
- A/B testing framework।
- Monitor calibration drift।
Bangladesh-specific:
- Mobile banking adoption rapid।
- Mobile features (bKash usage) increasingly important।
- Geographic features (urban/rural)।
- Seasonal patterns (Eid, harvest)।
- SME-specific features।
মূল উপলব্ধি: Single best model নেই — context-dependent। Banking-এ Elastic Net often sweet spot। Regulation, interpretability, accuracy — triple constraint। L1/L2 tools সঠিক balance।
প্র ০৩ Deep learning-এর "weight decay" Ridge-এর সাথে সম্পর্ক কী? Adam-এর সাথে weight decay কেন subtly ভিন্ন?
Modern DL-এ regularization — Ridge-এর direct heir, কিন্তু subtleties উল্লেখযোগ্য।
Weight decay basic:
- Loss-এ $\lambda \|\mathbf{w}\|^2$ যোগ।
- Gradient: $\nabla L + 2\lambda \mathbf{w}$।
- Update: $\mathbf{w} \leftarrow (1 - 2\eta\lambda) \mathbf{w} - \eta \nabla L$।
- "Decay" — weights shrink toward zero each step।
SGD + L2 = SGD + weight decay:
- Mathematically equivalent।
- Loss-এ penalty বা separate decay term — same।
- Implementation choice।
Adam + L2 ≠ Adam + weight decay:
- Loshchilov & Hutter (2017) — AdamW।
- Subtle but important difference।
- Adam — adaptive per-parameter rates।
Adam-এর problem:
- Loss-এ L2 — gradient include penalty।
- Adam normalize gradient by running variance।
- Penalty effect distorted by adaptive rate।
- Effective regularization — feature-specific (unintended)।
AdamW solution:
- Weight decay separately apply।
- $\mathbf{w} \leftarrow \mathbf{w} - \eta(\hat{m}/\sqrt{\hat{v}} + \lambda \mathbf{w})$।
- Decay independent of gradient adaptation।
- Cleaner regularization।
Empirical impact:
- AdamW — ImageNet, NLP — often better।
- BERT, GPT — AdamW standard।
- Default optimizer in modern code।
BatchNorm-এর সাথে interaction:
- BN scale-invariant।
- Weight decay-এর effective regularization indirect।
- "Effective learning rate" tunable via decay।
- Active research area।
$\lambda$ choice in DL:
- Image classification: $\lambda \approx 5 \times 10^{-4}$ standard।
- NLP/Transformer: $\lambda \approx 0.01-0.1$।
- Smaller models — more decay typically।
- Larger models — less decay (capacity-data balance)।
Other DL regularization:
(১) Dropout (২০১৪):
- Random neuron deactivation during training।
- Implicit ensemble।
- Pre-Transformer dominant।
(২) Data augmentation:
- Implicit input regularization।
- Image: rotation, crop, color।
- Text: synonym replacement, masking।
(৩) Early stopping:
- Validation loss monitoring।
- "Effective L2 regularization" theoretical link।
- Cheap and effective।
(৪) Stochastic depth:
- Random layer dropping (very deep networks)।
- ResNet variant।
(৫) Mixup/CutMix:
- Sample interpolation।
- Strong implicit regularization।
- Modern image training standard।
L1 in DL:
- Rare in main loss।
- Specific applications: pruning, sparsity।
- "Lottery ticket hypothesis" — sparse subnetworks।
- Group sparsity in structured pruning।
Implicit regularization:
- SGD itself regularizes।
- Over-parameterization helps (counter-intuitive)।
- "Double descent" phenomenon।
- Belkin et al. (2019)।
মূল উপলব্ধি: Ridge concept DL-এ alive কিন্তু transformed। AdamW — modern default। Regularization toolkit — multiple complementary techniques। Single regularizer rarely sufficient।
প্র ০৪ "Bias-variance" lens-এ regularization কীভাবে? $\lambda$ বাড়ালে bias বাড়ে, variance কমে — কেন?
Bias-variance tradeoff — regularization বোঝার সবচেয়ে cleaner lens।
Bias-variance decomposition:
- $E[(\hat{f}(x) - f(x))^2] = \text{Bias}^2 + \text{Variance} + \text{Noise}$।
- Bias — average prediction vs true।
- Variance — predictions across samples।
- Noise — irreducible।
OLS — high variance, low bias:
- Unbiased estimator (asymptotically)।
- High variance with $N \approx p$।
- Different samples — wildly different coefficients।
- Overfitting symptom।
Ridge — bias trade for variance reduction:
- $\lambda > 0$ → biased estimator।
- Bias toward ০।
- Variance dramatically reduced।
- Net MSE often lower।
$\lambda$ effect:
- $\lambda = 0$ — OLS, low bias, high variance।
- $\lambda \to \infty$ — $\mathbf{w} \to 0$, high bias, zero variance।
- Optimal $\lambda$ — minimize total error।
Why bias increases:
- Penalty pulls $\mathbf{w}$ toward ০।
- True $\mathbf{w}$ ≠ ০ generally।
- Predictions systematically biased।
- "Pessimistic" estimator।
Why variance decreases:
- Penalty bounds $\mathbf{w}$ values।
- Sample noise — limited impact।
- Coefficients stable across samples।
- Smaller "radius" of estimates।
Mathematical proof sketch (Ridge):
- $\mathbf{w}_{\text{Ridge}} = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y}$।
- $E[\mathbf{w}_{\text{Ridge}}] = (X^\top X + \lambda I)^{-1} X^\top X \mathbf{w}^*$।
- $\neq \mathbf{w}^*$ unless $\lambda = 0$ → biased।
- $\text{Var}[\mathbf{w}_{\text{Ridge}}] = \sigma^2 (X^\top X + \lambda I)^{-1} X^\top X (X^\top X + \lambda I)^{-1}$।
- $< \text{Var}[\mathbf{w}_{\text{OLS}}]$ in matrix sense।
Practical implications:
- Few samples (high variance regime): Strong regularization helps।
- Many samples (low variance): Weak regularization sufficient।
- Many features: Stronger regularization।
- Strong signal: Less regularization needed।
Cross-validation discovery:
- CV error curve — U-shape in $\lambda$।
- Left side (small $\lambda$): variance high, error high।
- Right side (large $\lambda$): bias high, error high।
- Bottom of U: optimal $\lambda$।
Beyond linear regression:
- Decision tree depth — complexity tradeoff।
- K in K-NN — bias-variance।
- Network depth/width — DL capacity।
- Universal principle।
Modern DL twist:
- Over-parameterization — counter-intuitive low variance।
- "Double descent" — curve shape complex।
- Belkin et al. — modern interpolation regime।
- Classical bias-variance — partial picture।
Bayesian perspective:
- Prior — bias toward simple models।
- Posterior — combines prior + data।
- $\lambda$ — prior strength।
- MAP estimation natural framework।
Practical workflow:
- Multiple $\lambda$ try।
- CV estimate each।
- "১-SE rule": Simplest model within ১ SE of best।
- Production deployment।
Diagnostic:
- Train acc high, val acc low — high variance — increase $\lambda$।
- Both low — high bias — decrease $\lambda$ or add features।
- Both high — adequate model।
মূল উপলব্ধি: Regularization — bias-variance trade তবে net win possible। $\lambda$ — explicit trade dial। Modern ML — implicit trade everywhere। Understanding this — separates ML practitioners from button-pressers।
অনুশীলন
-
হিসাব করুন: Ridge $\lambda = 1$, $X = [[1], [1]]$, $y = [2, 4]$।
- $X^\top X$ কত? $X^\top X + \lambda I$ কত?
- $\mathbf{w}_{\text{Ridge}}$ কত?
- $X^\top X = 2$।
- $+\lambda I = 3$।
- $X^\top y = 6$।
- $w = 6/3 = 2$।
- OLS-এর $w = 6/2 = 3$ — Ridge shrunk।
-
sklearn: Diabetes dataset-এ Ridge, Lasso, OLS compare করুন। Cross-validation দিয়ে best $\alpha$ বের করুন।
from sklearn.datasets import load_diabetes from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV from sklearn.model_selection import train_test_split X, y = load_diabetes(return_X_y=True) X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3) ols = LinearRegression().fit(X_tr, y_tr) ridge = RidgeCV().fit(X_tr, y_tr) lasso = LassoCV().fit(X_tr, y_tr) for name, m in [("OLS", ols), ("Ridge", ridge), ("Lasso", lasso)]: print(f"{name}: test R² = {m.score(X_te, y_te):.4f}") -
চিন্তা: ১০০০ features, ১০০ samples — কোন regularization? Sparsity expected হলে?
$N \ll p$ extreme regime। Lasso essential — অনেক feature ০ select। Ridge fail — সব feature shrink করে। Elastic Net (next lesson) — best balance।
আরও পড়ুন
- পাঠ ১৬ · Elastic Net পরবর্তী পাঠ Ridge + Lasso combined — best of both worlds।
- পাঠ ১৪ · Multiclass softmax আগের পাঠ Multi-class loss recap।
- পাঠ ০৪ · Bias-variance এই পাঠের সাথে সম্পর্কিত Regularization-এর foundation।
- সব AI Courses ABCL TECH Python, ML, DL, NLP, CV, GenAI, RL, MLOps।