ARIMA ও Prophet দিয়ে forecast

এটি forecasting practitioner-এর nuanced understanding। দু'টি tool-এর strength ও limitation বুঝলে — সঠিক choice automatic।

ARIMA-র strengths:

• Statistical theory — convergence, asymptotic properties proven।
• Parameter interpretable — $\phi$ persistence, $\theta$ shock।
• Confidence interval mathematically rigorous।
• Stationary white-noise process-এ optimal।
• Academic publication-এ standard।
• Short-term forecast precise।

ARIMA-র preferred situations:

(১) Stationary or near-stationary data

• Stock returns (after differencing)।
• Inflation rate।
• Stable manufacturing process।

(২) Short data history

• 1-2 years of data।
• Prophet seasonality estimation-এ 2+ years চাই।
• ARIMA flexible-এ modest data।

(৩) Statistical inference needed

• Hypothesis test on coefficients।
• Significance assessment।
• Academic/regulatory context।

(৪) Pure auto-correlation pattern

• "Yesterday's value strongly predicts today's"।
• Mean-reverting process।
• AR(1) pure।

(৫) Volatility modeling

• GARCH (ARIMA-family extension) — financial volatility।
• Prophet volatility natively model করে না।

(৬) High-frequency data

• Tick-by-tick stock data।
• Sensor reading-এ second-by-second।
• Prophet-এর daily-or-coarser focus।

Prophet ভুল হবে যেখানে:

(১) Sub-daily granularity

• Hourly minute-level data।
• Prophet-এ "daily seasonality" support আছে কিন্তু minute-level optimized না।
• SARIMA বা DL preferred।

(২) Pure stochastic process

• No discernible trend/seasonality।
• White noise-এর close।
• Prophet over-fit components আবিষ্কার করে।
• Random walk-এ ARIMA simpler।

(৩) Volatility focus

• Mean predict সহজ, variance predict-এ Prophet weak।
• Risk management, options pricing — GARCH।

(৪) Strict accuracy bound

• Aerospace, medical — exact CI চাই।
• Prophet Bayesian কিন্তু regularization heuristic।
• ARIMA proper interval।

(৫) Tiny data

• 50 observation।
• Prophet seasonality fit করতে পারে না।
• Simple AR baseline better।

(৬) Multivariate causality

• VAR (Vector AR) — multiple series interaction।
• Prophet single series + exogenous regressor (limited)।
• Granger causality test ARIMA-context-এ।

Concrete Bangladesh examples:

ARIMA preferred:

• BSEC daily returns: log-differenced + ARIMA + GARCH।
• USD-BDT exchange rate: random walk-এর close, ARIMA(0,1,0)।
• Tea export quarterly data (short history): ARIMA।

Prophet preferred:

• Daraz daily revenue: trend + yearly + weekly + Eid।
• bKash transaction count: multi-seasonality + holidays।
• Pathao monthly user growth: trend + holiday।

Both tested:

• BTCL bandwidth: SARIMA + Prophet — ensemble।
• Hospital admission: Prophet baseline + SARIMA refinement।

Hybrid approach:

• Prophet trend + seasonality fit।
• Residual-এ ARIMA।
• Final = Prophet + ARIMA(residual)।
• Best of both worlds।

Modern alternatives (২০২৪):

• NeuralProphet (Prophet + NN)।
• StatsForecast (Nixtla) — fast classical methods।
• Foundation models — TimeGPT, Lag-Llama।
• MLForecast — gradient boosting on time features।

Decision framework:

1. Plot data — pattern visible?
2. Data length — > 2 years?
3. Holidays affect outcome?
4. Multiple seasonalities?
5. Need explainability?
6. Statistical inference required?

মূল উপলব্ধি: "Prophet সবসময় better" myth। Each tool's domain। Senior practitioner দু'টোই comfortable, situation-এ choose। Modern approach — ensemble।

Information criteria — model comparison-এর foundation। Frequentist ও Bayesian দুই দিক থেকেই derived।

AIC formula:

$$AIC = 2k - 2 \ln(\hat{L})$$

• $k$ — parameter সংখ্যা।
• $\hat{L}$ — maximum likelihood।
• Lower better।

BIC formula:

$$BIC = k \ln(n) - 2 \ln(\hat{L})$$

• $n$ — sample size।
• $k \ln(n)$ — parameter penalty।

মূল পার্থক্য — penalty:

• AIC: penalty per parameter = 2।
• BIC: penalty per parameter = $\ln(n)$।
• $n > e^2 = 7.4$ — BIC penalty > AIC।
• BIC simpler model prefer।

Intuition behind AIC (information theory):

• Kullback-Leibler divergence approximation।
• "True model ও fitted model-এর information loss minimize"।
• Prediction-focus।
• Akaike (১৯৭৩) derive।

Intuition behind BIC (Bayesian):

• Marginal likelihood (Bayes factor) approximation।
• "Posterior probability of model"।
• True model selection focus।
• Schwarz (১৯৭৮) derive।

কখন AIC preferred:

• Forecast accuracy primary goal।
• "Best predictive" model।
• Complex underlying process suspected — more parameters captures it।
• Sample size moderate।

কখন BIC preferred:

• Identifying "true" simpler model।
• Parsimony — Occam's razor।
• Hypothesis testing flavor।
• Large sample size।
• Genomics, model selection competitions।

Asymptotic behavior:

• BIC consistent: $n \to \infty$, true model selected (if in candidate set)।
• AIC efficient: prediction error minimum among candidates।
• Different optimality criteria।

Disagreement common:

• AIC pick ARIMA(2,1,2)।
• BIC pick ARIMA(1,1,1)।
• Sample size $n = 200$, AIC penalty 2, BIC penalty $\ln(200) \approx 5.3$।
• BIC simpler।

Practical recommendation:

• Both compute।
• Short data ($n < 100$): AIC usually fine।
• Large data ($n > 1000$): BIC reasonable।
• Disagreement → cross-validation tiebreak।

Other criteria:

(১) AICc — corrected AIC:

$$AICc = AIC + \frac{2k(k+1)}{n - k - 1}$$

• Small sample bias correction।
• Hyndman recommendation for time series।

(২) HQIC — Hannan-Quinn:

$$HQIC = 2k \ln(\ln(n)) - 2\ln(\hat{L})$$

• Penalty intermediate (AIC ও BIC এর মাঝে)।
• Less common।

(৩) Cross-validation:

• Walk-forward MAE/RMSE।
• Direct prediction performance।
• Most reliable but expensive।

auto_arima behavior:

• Default AIC — pmdarima।
• BIC option available।
• Search space tune।

Common mistakes:

• "AIC=42 better than AIC=45 by 3" — depends on scale; difference more important is delta > 2।
• Comparing across different data — invalid (sample size affects)।
• Treating IC as ground truth — they're heuristic, not absolute।

Information criteria limitations:

• "In-sample" — out-of-sample performance separate।
• Likelihood-based — non-likelihood model can't compare।
• Penalty heuristic — true Bayesian marginal likelihood theoretically better।
• "Best AIC model still might be poor" — relative not absolute।

Bangladesh ML practice:

• auto_arima Bangladesh data-team standard।
• BIC for parsimony culture (regulatory)।
• AICc small data (quarterly economic series)।
• Cross-validation final arbiter (Daraz CTR forecast)।

মূল উপলব্ধি: AIC vs BIC — different optimality goals (prediction vs true model)। Both compute, sense-check। Information criteria heuristic — final validation cross-validation।

Real production scenario। Daraz প্রতি SKU প্রতি warehouse-এ এই ধরনের forecasting করে।

Step ১: Data preparation

• Daily quantity sold per SKU।
• 2+ years history (2 Eid cycles)।
• Outlier handling (stockout days exclude)।
• Format: ds (date), y (quantity)।

Step ২: Holiday calendar

import pandas as pd

holidays = pd.DataFrame([
    {'holiday':'eid_fitr', 'ds':'2023-04-22', 'lower_window':-7, 'upper_window':3},
    {'holiday':'eid_fitr', 'ds':'2024-04-10', 'lower_window':-7, 'upper_window':3},
    {'holiday':'eid_fitr', 'ds':'2025-03-30', 'lower_window':-7, 'upper_window':3},
    {'holiday':'eid_adha', 'ds':'2023-06-29', 'lower_window':-5, 'upper_window':2},
    {'holiday':'eid_adha', 'ds':'2024-06-17', 'lower_window':-5, 'upper_window':2},
    {'holiday':'eid_adha', 'ds':'2025-06-07', 'lower_window':-5, 'upper_window':2},
    {'holiday':'pohela_boishakh', 'ds':'2024-04-14', 'lower_window':-3, 'upper_window':1},
    {'holiday':'pohela_boishakh', 'ds':'2025-04-14', 'lower_window':-3, 'upper_window':1},
    {'holiday':'victory_day', 'ds':'2024-12-16', 'lower_window':-1, 'upper_window':1},
])
holidays['ds'] = pd.to_datetime(holidays['ds'])

Note: lower_window=-7 means 7 days before, capturing pre-Eid shopping surge।

Step ৩: Prophet model

from prophet import Prophet

model = Prophet(
    growth='linear',
    yearly_seasonality=True,
    weekly_seasonality=True,
    daily_seasonality=False,
    holidays=holidays,
    seasonality_mode='multiplicative',  # Eid surge proportional to base
    changepoint_prior_scale=0.05,
    holidays_prior_scale=10.0,
    seasonality_prior_scale=10.0
)

# Add custom regressor
df['is_ramadan'] = ramadan_indicator(df['ds'])
model.add_regressor('is_ramadan')

# Fit
model.fit(df)

Step ৪: Future dataframe

# 6 months ahead = 180 days
future = model.make_future_dataframe(periods=180)
future['is_ramadan'] = ramadan_indicator(future['ds'])
forecast = model.predict(future)

Step ৫: Forecast components

# Visualize components
fig = model.plot_components(forecast)
# 4 subplots: trend, holidays, weekly, yearly

Step ৬: Validation

from prophet.diagnostics import cross_validation, performance_metrics
df_cv = cross_validation(model, initial='730 days', period='30 days', horizon='90 days')
df_p = performance_metrics(df_cv)
print(df_p[['horizon','mape','rmse']].head())

Step ৭: SKU-specific tuning

• Fashion (Eid surge huge): holidays_prior_scale 20।
• Electronics (Eid moderate): default 10।
• Books (Boi Mela): pohela_boishakh window expand।

Step ৮: Anomaly handling

• COVID period (Mar-Aug 2020) exclude।
• Stockout days exclude (zero ≠ no demand)।
• Promotional spikes annotate।

Step ৯: Capacity & uncertainty

• Forecast intervals (yhat_lower, yhat_upper) inventory plan।
• Safety stock: 95th percentile।
• Lead time consideration (supplier 30+ days)।

Step ১০: Pipeline & monitoring

• Weekly retrain cron।
• Forecast accuracy track per SKU।
• Drift detection — actual vs predicted gap।
• Alert on consistent under/over-forecast।

Bangladesh-specific complications:

(১) Lunar drift

• Eid এক বছর before-Ramadan, পরের বছর during।
• Heat factor (Eid in summer vs winter)।
• Manual holiday list maintenance।

(২) Government policy

• VAT changes — sudden price shift।
• Import restriction — supply-side shock।
• Manual change-point intervention।

(৩) Exchange rate

• USD-BDT volatile — imported product price impacted।
• Cross-product dependency।
• Multi-variate model consider।

(৪) Product lifecycle

• New SKU — no history।
• Cold-start: similar SKU embedding।
• Hierarchical forecasting (category → subcategory → SKU)।

(৫) Cross-channel

• Pickaboo, Evaly absorb leak।
• Daraz market share fluctuating।
• External marketshare data।

Alternative architectures:

• Hierarchical Prophet: total → category → SKU।
• NeuralProphet: AR + NN component।
• DeepAR (Amazon): RNN-based, cross-series learning।
• N-BEATS: deep neural baseline for time series।
• TFT (Temporal Fusion Transformer): attention-based।

Production considerations:

• Compute: 10K SKU × Prophet train = parallel।
• Storage: forecast results in feature store।
• API latency: cached forecast, on-demand explain।
• Model versioning: A/B tests on champion model।

Success metrics:

• SKU-level MAPE।
• Stockout reduction।
• Inventory carrying cost।
• Forecast confidence interval coverage।

মূল উপলব্ধি: Real Daraz forecasting hundreds of SKU, Bangladesh-specific holiday-aware। Prophet starting point, customization heavy। Pipeline + monitoring + business alignment — production excellence।

Forecasting-এর fundamental constraint। Mathematical reason ও practical mitigation।

কেন accuracy decreases:

(১) Compounding uncertainty

• প্রতিটি step-এ noise added।
• $h$-step-ahead variance: $\sigma^2 \cdot h$ (random walk)।
• Linear in $h$ for ARIMA।
• Confidence interval expands।

(২) Iterative prediction

• Multi-step forecast: previous prediction next-step input।
• Error compound — like compound interest।
• "Telephone game" effect।

(৩) Distribution shift

• Future ≠ past।
• Economy, technology, behavior change।
• Long-horizon-এ change probability বেশি।

(৪) Trend uncertainty

• Linear trend long-term unrealistic।
• Saturation, decline, regime change।
• Prophet logistic growth — saturation handle।

(৫) Black swan

• COVID-19 — no model predicted।
• Russian invasion — global supply chain shock।
• Long-horizon-এ এমন event probability higher।

Mathematical illustration (random walk):

• $Y_t = Y_{t-1} + \epsilon_t$, $\epsilon \sim N(0, \sigma^2)$।
• $h$-step forecast variance = $h \sigma^2$।
• Standard error scales as $\sqrt{h}$।
• 1-day SE = $\sigma$, 100-day SE = $10\sigma$।
• Confidence interval 10x wider।

Strategies for long-horizon:

(১) Direct forecasting

• $h$-step-specific model। প্রতিটি horizon-এ separate model।
• 1-day model, 7-day model, 30-day model — separate train।
• Avoid iterative compound।
• Computationally heavier।

(২) Hierarchical forecasting

• Aggregate level (country) — long-horizon stable।
• Disaggregate (city) — short-horizon।
• Reconcile bottom-up + top-down।
• library: hts, scikit-hts।

(৩) Ensemble

• ARIMA + Prophet + ETS + ML — average।
• Bias cancellation।
• Variance reduction।
• Generally robust।

(৪) Bayesian/probabilistic

• Distribution forecast (not point)।
• Pinball loss, quantile loss।
• Expected business cost minimize।
• Prophet provides intervals natively।

(৫) Saturation modeling

• Logistic growth (Prophet)।
• Carrying capacity specify।
• Realistic long-term।

(৬) Scenario analysis

• Optimistic, baseline, pessimistic scenarios।
• Different assumption sets।
• Decision robust to scenario।

(৭) Frequent retraining

• "6-month forecast" refresh weekly।
• New data incorporate।
• Recent error correct।
• Production pipeline।

(৮) Exogenous information

• Macroeconomic forecasts incorporate।
• Industry projections।
• Competitive intelligence।
• Multi-variate model।

(৯) Cross-learning

• Many similar series — transfer learning।
• DeepAR, TFT — global model।
• Single series limited; many series rich pattern।

(১০) Domain knowledge integration

• Expert prior — Bayesian prior or constraints।
• Business rules embedded।
• "Sales can't grow more than 30% YoY" — constraint।

Bangladesh long-horizon scenarios:

• Daraz 6-month inventory: hierarchical + Prophet + frequent retrain।
• Bangladesh GDP 5-year: scenario analysis (high/low growth)।
• Population census 10-year: cohort-component method (demographic)।
• Climate (BMD): physics-based + statistical।

Reporting long-horizon:

• Wide confidence intervals — honest।
• Decision support, not exact prediction।
• Scenario probabilities।
• Caveats explicit।

Common mistake:

• "Linear trend extrapolate 5 years" — naive।
• "Single point forecast 1 year" — false confidence।
• "Past performance = future" — black swan ignore।

Real-world humility:

• Hyndman, Athanasopoulos textbook: "Forecast about ৩০ days ahead reasonable for most series"।
• Beyond — wide uncertainty acknowledgment।
• Model + judgment + scenario।

Modern foundation models (২০২৪+):

• TimeGPT — Nixtla foundation model।
• Lag-Llama — open-source।
• Pre-trained on millions of series — better long-horizon transfer।
• Zero-shot forecast।
• Bangladesh-context fine-tuning।

মূল উপলব্ধি: Long-horizon forecast = limited science + structured guess। Probabilistic + scenario + frequent update — production approach। Forecasting humble craft, not omniscient prophecy।