TRPO — Trust Region Policy Optimization
এই পাঠে যা শিখবেন
- TRPO-র motivation — large-step instability
- Surrogate objective ও KL constraint
- Conjugate gradient + line search
- কেন PPO TRPO-কে replace করেছে
১ · Vanilla policy gradient-এর নাজুকতা
$\theta \leftarrow \theta + \alpha \nabla J$।
Step size $\alpha$ — যদি বড় হয় — policy "অনুপস্থিত" হতে পারে। অর্থাৎ — extreme distribution shift। তখন rolled-out trajectory আগের policy-র data থেকে ভিন্ন — gradient estimate ভুল।
এই collapse — policy rarely recover করে। RL training-এর সবচেয়ে frustrating issue।
২ · Trust region idea
Schulman et al. (2015) — Pieter Abbeel-এর Berkeley group। ধারণা — old policy থেকে too far না সরে এমন step নাও।
"Trust region" — current policy-এর চারপাশে এক ছোট region যেখানে surrogate objective accurate। সেই region-এর বাইরে যাও না।
৩ · Surrogate objective
Vanilla PG-এর exact objective $J(\theta)$ — non-trivial to compute। Approximation:
$$L_{\theta_{old}}(\theta) = \mathbb{E}_{s, a \sim \pi_{old}} \left[ \frac{\pi_\theta(a|s)}{\pi_{old}(a|s)} A^{\pi_{old}}(s, a) \right]$$
"Importance sampling" ratio দিয়ে — old data থেকে new policy-র estimate। valid যদি new policy old-এর কাছাকাছি।
৪ · KL constraint
Constrained optimization:
$$\max_\theta L_{\theta_{old}}(\theta) \quad \text{s.t.} \quad \mathbb{E}_s [D_{KL}(\pi_{old}(\cdot|s) \| \pi_\theta(\cdot|s))] \le \delta$$
$\delta$ small (~0.01) — conservative step। large — aggressive।
Properly chosen $\delta$ এবং exact optimization-এ — guaranteed $J(\theta_{new}) \ge J(\theta_{old})$। অর্থাৎ — non-decreasing performance prove।
৫ · Solution — natural gradient + line search
Constrained problem-এর Lagrangian — quadratic approximate KL constraint। closed-form direction:
$$\Delta \theta = \sqrt{\frac{2 \delta}{g^T F^{-1} g}} F^{-1} g$$
$g = \nabla L$, $F$ = Fisher information matrix।
Practical: $F^{-1}$ compute infeasible (large NN)। তাই — conjugate gradient method (HessianFree) iterative-ভাবে $F^{-1} g$ approximate।
৬ · Backtracking line search
Quadratic approximation imperfect। তাই step-এর পর check:
- Surrogate $L$ improved?
- KL constraint satisfied?
Both yes — accept। নাহলে — step half করে retry। সাধারণত ২-৩ retry-এ পেয়ে যায়।
৭ · TRPO algorithm — pseudocode
- Rollout — collect trajectories using $\pi_{\theta_{old}}$।
- Compute advantages $A_t$ (GAE)।
- Compute policy gradient $g$।
- Conjugate gradient: $x = F^{-1} g$ approximate।
- Step size: $\beta = \sqrt{2 \delta / (x^T F x)}$।
- Proposed step: $\theta \leftarrow \theta_{old} + \beta x$।
- Backtracking line search to ensure KL ≤ δ ও improvement।
- Update value network (separate)।
৮ · TRPO — empirical results
- MuJoCo continuous control — দারুণ (Schulman 2015)।
- Atari — comparable to A3C/DQN।
- Robotics-এ standard before PPO।
কিন্তু — implementation hard। conjugate gradient, line search, Hessian-vector product — ১০০০+ lines code। typical PyTorch/TF implementation 800+ lines।
৯ · কেন PPO replace করল
OpenAI (Schulman, 2017) — same author। PPO simpler, comparable performance।
- PPO clipped objective — KL approximation, no second-order optimization।
- Adam optimizer যথেষ্ট — conjugate gradient লাগে না।
- Multiple SGD epoch on collected data — TRPO single step।
- ~১০০ lines code।
আজ — TRPO খুব কম use। কিন্তু এর theory (monotonic improvement, surrogate objective) — PPO ও modern RL-এর foundation।
ভাবনার প্রশ্ন
প্র ০১Importance ratio $\pi_\theta / \pi_{old}$ off-policy correction-এর মত — কিন্তু TRPO on-policy?
Subtle distinction।
Strict off-policy: behavior policy $b$ ≠ target $\pi$, fundamentally different। importance ratio multi-step product (variance explosion)।
TRPO: "near-on-policy"। $\pi_{old}$ ও $\pi_{new}$ very close (KL ≤ δ)। single-step ratio। multi-step product avoid।
Operationally on-policy কারণ:
- Data collected with $\pi_{old}$।
- Update with one step (no buffer reuse across iterations)।
- Each iteration starts fresh rollout।
"Multiple epochs" PPO innovation: same data বহুবার update — eventually $\pi$ shifts। PPO clipping এই risk control।
মূল উপলব্ধি: on-policy/off-policy spectrum। TRPO/PPO middle — "stale" data acceptable as long as constrained close।
প্র ০২Fisher information matrix কী, কেন natural gradient-এ?
$F = \mathbb{E}_{a \sim \pi_\theta}[\nabla \log \pi_\theta \cdot \nabla \log \pi_\theta^T]$।
Geometric meaning:
- $F$ — local "metric" on policy manifold।
- Distance not Euclidean — KL-defined।
- $F^{-1}$ scales gradient to KL-distance terms।
Why natural gradient:
- Vanilla gradient — Euclidean step in θ-space।
- Natural — step in distribution space।
- Two parameterization-এ same KL difference → same natural step। reparameterization-invariant।
Computation challenge:
- $F$ — $|\theta| \times |\theta|$ matrix। NN-এ millions x millions — infeasible।
- $F^{-1} g$ direct — even worse।
Solution: conjugate gradient।
- $F$-vector product cheap (autodiff)।
- CG iteratively solve $F x = g$ — $O(k)$ matrix-vector products।
- $k \approx 10$ enough।
Modern alternative: KFAC (Kronecker-factored approximation) — block-diagonal $F^{-1}$ approx।
প্র ০৩"Monotonic improvement guarantee" empirically সবসময় hold করে?
Theoretically: TRPO proves $J(\theta_{new}) \ge J(\theta_{old}) - $ small-error term।
Empirical reality:
- সাধারণত yes — return curve smooth upward।
- কিন্তু — finite-sample estimate-এর variance, line search threshold — occasional dips।
- Practice-এ — return periodic-ভাবে noisy কিন্তু trend monotonic।
Approximation breakdowns:
- Quadratic KL approximation only valid for small step।
- Surrogate $L$-এর accuracy beyond trust region degraded।
- Non-convex landscape — local optima।
PPO-তে guarantee নেই কিন্তু empirically robust:
- Clipping looser than KL constraint।
- Multiple epochs may violate trust region in worst case।
- কিন্তু — Adam + clipping + early stop যথেষ্ট stable।
প্র ০৪TRPO/PPO-এর intuition — কেন large step problem?
Several intertwined issues:
(১) Distribution shift:
- $\pi_\theta$ change → state distribution shift।
- Old data বিভিন্ন policy-র — gradient estimate biased।
- Large step → large shift → biased gradient → wrong direction।
(২) Surrogate objective inaccuracy:
- $L$ — Taylor expansion around $\theta_{old}$।
- Far from $\theta_{old}$ — Taylor approximation error explode।
- Maximize wrong landscape।
(৩) Catastrophic forgetting:
- NN-এর parameters shared। large step একটা region-এ — অন্য region-এ catastrophic shift।
(৪) RL non-recovery:
- Supervised — bad step-ও recoverable (data fixed)।
- RL — bad policy → bad data → bad gradient → বার বার bad। downward spiral।
সমাধান (TRPO/PPO): small, safe step। নিশ্চিত করা যে — surrogate accurate ও policy quality-এ regress না।
অনুশীলন
-
KL compute: two Gaussian — $\mathcal{N}(0, 1)$ ও $\mathcal{N}(1, 1)$। KL divergence?
$D_{KL}(\mathcal{N}(\mu_1, \sigma_1^2) \| \mathcal{N}(\mu_2, \sigma_2^2)) = \log(\sigma_2/\sigma_1) + \frac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - 0.5$।
Here: $0 + (1+1)/2 - 0.5 = 0.5$।
-
Importance ratio: $\pi_{old}(a|s) = 0.4$, $\pi_\theta(a|s) = 0.5$, advantage $A = 2$. surrogate contribution?
$L = (0.5/0.4) \cdot 2 = 1.25 \cdot 2 = 2.5$।
-
Why δ matters: δ = 0.001 vs 0.1 — কী difference?
Small δ — conservative steps, slow learning, very stable। Large δ — aggressive, faster but risky। Schulman δ=0.01 default — sweet spot most tasks।
আরও পড়ুন
- পাঠ ২১ · PPO পরবর্তী পাঠTRPO simpler — clipping enough।
- পাঠ ১৯ · A2C & A3C আগের পাঠParallel actor-critic।
- পাঠ ২২ · SAC এই পাঠের সাথে সম্পর্কিতOff-policy continuous SOTA।
- সব AI Courses ABCL TECHPython, ML, DL, NLP, CV, GenAI, RL — সব একসাথে।