পাঠ ২০ · ৩২-এর মধ্যে · মডিউল ৩
Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / TRPO

TRPO — Trust Region Policy Optimization

TRPO — guaranteed monotonic improvement via KL constraint
৭ মিনিট পড়া উচ্চ · Advanced Math-heavy

এই পাঠে যা শিখবেন

  • TRPO-র motivation — large-step instability
  • Surrogate objective ও KL constraint
  • Conjugate gradient + line search
  • কেন PPO TRPO-কে replace করেছে

১ · Vanilla policy gradient-এর নাজুকতা

$\theta \leftarrow \theta + \alpha \nabla J$।

Step size $\alpha$ — যদি বড় হয় — policy "অনুপস্থিত" হতে পারে। অর্থাৎ — extreme distribution shift. তখন rolled-out trajectory আগের policy-র data থেকে ভিন্ন — gradient estimate ভুল।

এই collapse — policy rarely recover করে। RL training-এর সবচেয়ে frustrating issue.

২ · Trust region idea

Schulman et al. (2015) — Pieter Abbeel-এর Berkeley group. ধারণা — old policy থেকে too far না সরে এমন step নাও।

"Trust region" — current policy-এর চারপাশে এক ছোট region যেখানে surrogate objective accurate. সেই region-এর বাইরে যাও না।

৩ · Surrogate objective

Vanilla PG-এর exact objective $J(\theta)$ — non-trivial to compute. Approximation:

$$L_{\theta_{old}}(\theta) = \mathbb{E}_{s, a \sim \pi_{old}} \left[ \frac{\pi_\theta(a|s)}{\pi_{old}(a|s)} A^{\pi_{old}}(s, a) \right]$$

"Importance sampling" ratio দিয়ে — old data থেকে new policy-র estimate. valid যদি new policy old-এর কাছাকাছি।

৪ · KL constraint

Constrained optimization:

$$\max_\theta L_{\theta_{old}}(\theta) \quad \text{s.t.} \quad \mathbb{E}_s [D_{KL}(\pi_{old}(\cdot|s) \| \pi_\theta(\cdot|s))] \le \delta$$

$\delta$ small (~0.01) — conservative step. large — aggressive.

Theorem (Schulman 2015)

Properly chosen $\delta$ এবং exact optimization-এ — guaranteed $J(\theta_{new}) \ge J(\theta_{old})$। অর্থাৎ — non-decreasing performance prove.

৫ · Solution — natural gradient + line search

Constrained problem-এর Lagrangian — quadratic approximate KL constraint. closed-form direction:

$$\Delta \theta = \sqrt{\frac{2 \delta}{g^T F^{-1} g}} F^{-1} g$$

$g = \nabla L$, $F$ = Fisher information matrix.

Practical: $F^{-1}$ compute infeasible (large NN). তাই — conjugate gradient method (HessianFree) iterative-ভাবে $F^{-1} g$ approximate.

৬ · Backtracking line search

Quadratic approximation imperfect. তাই step-এর পর check:

  • Surrogate $L$ improved?
  • KL constraint satisfied?

Both yes — accept. নাহলে — step half করে retry. সাধারণত ২-৩ retry-এ পেয়ে যায়।

TRPO — Trust Region in θ-space θ-space (parameters) θ_old KL ≤ δ (trust region) θ_new ✓ vanilla PG: collapse! L max TRPO: maximize surrogate L within KL ball. step too far = surrogate inaccurate. Without trust region: aggressive step → policy collapse. Recovery rare.
TRPO-এর geometric intuition — old policy-এর চারপাশে KL-defined trust region, এর ভেতরে surrogate maximize.

৭ · TRPO algorithm — pseudocode

  1. Rollout — collect trajectories using $\pi_{\theta_{old}}$।
  2. Compute advantages $A_t$ (GAE).
  3. Compute policy gradient $g$।
  4. Conjugate gradient: $x = F^{-1} g$ approximate.
  5. Step size: $\beta = \sqrt{2 \delta / (x^T F x)}$।
  6. Proposed step: $\theta \leftarrow \theta_{old} + \beta x$।
  7. Backtracking line search to ensure KL ≤ δ ও improvement.
  8. Update value network (separate).

৮ · TRPO — empirical results

  • MuJoCo continuous control — দারুণ (Schulman 2015).
  • Atari — comparable to A3C/DQN.
  • Robotics-এ standard before PPO.

কিন্তু — implementation hard. conjugate gradient, line search, Hessian-vector product — ১০০০+ lines code. typical PyTorch/TF implementation 800+ lines.

৯ · কেন PPO replace করল

OpenAI (Schulman, 2017) — same author. PPO simpler, comparable performance.

  • PPO clipped objective — KL approximation, no second-order optimization.
  • Adam optimizer যথেষ্ট — conjugate gradient লাগে না।
  • Multiple SGD epoch on collected data — TRPO single step.
  • ~১০০ lines code.

আজ — TRPO খুব কম use. কিন্তু এর theory (monotonic improvement, surrogate objective) — PPO ও modern RL-এর foundation.

TRPO implement-এর সরাসরি দরকার rare. কিন্তু এই পাঠ-এর ideas বুঝলে PPO clearly বুঝবেন।

ভাবনার প্রশ্ন

প্র ০১Importance ratio $\pi_\theta / \pi_{old}$ off-policy correction-এর মত — কিন্তু TRPO on-policy?

Subtle distinction.

Strict off-policy: behavior policy $b$ ≠ target $\pi$, fundamentally different. importance ratio multi-step product (variance explosion).

TRPO: "near-on-policy". $\pi_{old}$ ও $\pi_{new}$ very close (KL ≤ δ)। single-step ratio. multi-step product avoid.

Operationally on-policy কারণ:

  • Data collected with $\pi_{old}$।
  • Update with one step (no buffer reuse across iterations).
  • Each iteration starts fresh rollout.

"Multiple epochs" PPO innovation: same data বহুবার update — eventually $\pi$ shifts. PPO clipping এই risk control.

মূল উপলব্ধি: on-policy/off-policy spectrum. TRPO/PPO middle — "stale" data acceptable as long as constrained close.

প্র ০২Fisher information matrix কী, কেন natural gradient-এ?

$F = \mathbb{E}_{a \sim \pi_\theta}[\nabla \log \pi_\theta \cdot \nabla \log \pi_\theta^T]$।

Geometric meaning:

  • $F$ — local "metric" on policy manifold.
  • Distance not Euclidean — KL-defined.
  • $F^{-1}$ scales gradient to KL-distance terms.

Why natural gradient:

  • Vanilla gradient — Euclidean step in θ-space.
  • Natural — step in distribution space.
  • Two parameterization-এ same KL difference → same natural step. reparameterization-invariant.

Computation challenge:

  • $F$ — $|\theta| \times |\theta|$ matrix. NN-এ millions x millions — infeasible.
  • $F^{-1} g$ direct — even worse.

Solution: conjugate gradient.

  • $F$-vector product cheap (autodiff).
  • CG iteratively solve $F x = g$ — $O(k)$ matrix-vector products.
  • $k \approx 10$ enough.

Modern alternative: KFAC (Kronecker-factored approximation) — block-diagonal $F^{-1}$ approx.

প্র ০৩"Monotonic improvement guarantee" empirically সবসময় hold করে?

Theoretically: TRPO proves $J(\theta_{new}) \ge J(\theta_{old}) - $ small-error term.

Empirical reality:

  • সাধারণত yes — return curve smooth upward.
  • কিন্তু — finite-sample estimate-এর variance, line search threshold — occasional dips.
  • Practice-এ — return periodic-ভাবে noisy কিন্তু trend monotonic.

Approximation breakdowns:

  • Quadratic KL approximation only valid for small step.
  • Surrogate $L$-এর accuracy beyond trust region degraded.
  • Non-convex landscape — local optima.

PPO-তে guarantee নেই কিন্তু empirically robust:

  • Clipping looser than KL constraint.
  • Multiple epochs may violate trust region in worst case.
  • কিন্তু — Adam + clipping + early stop যথেষ্ট stable.
প্র ০৪TRPO/PPO-এর intuition — কেন large step problem?

Several intertwined issues:

(১) Distribution shift:

  • $\pi_\theta$ change → state distribution shift.
  • Old data বিভিন্ন policy-র — gradient estimate biased.
  • Large step → large shift → biased gradient → wrong direction.

(২) Surrogate objective inaccuracy:

  • $L$ — Taylor expansion around $\theta_{old}$।
  • Far from $\theta_{old}$ — Taylor approximation error explode.
  • Maximize wrong landscape.

(৩) Catastrophic forgetting:

  • NN-এর parameters shared. large step একটা region-এ — অন্য region-এ catastrophic shift.

(৪) RL non-recovery:

  • Supervised — bad step-ও recoverable (data fixed).
  • RL — bad policy → bad data → bad gradient → বার বার bad. downward spiral.

সমাধান (TRPO/PPO): small, safe step. নিশ্চিত করা যে — surrogate accurate ও policy quality-এ regress না।

অনুশীলন

  1. KL compute: two Gaussian — $\mathcal{N}(0, 1)$ ও $\mathcal{N}(1, 1)$। KL divergence?

    $D_{KL}(\mathcal{N}(\mu_1, \sigma_1^2) \| \mathcal{N}(\mu_2, \sigma_2^2)) = \log(\sigma_2/\sigma_1) + \frac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - 0.5$।

    Here: $0 + (1+1)/2 - 0.5 = 0.5$।

  2. Importance ratio: $\pi_{old}(a|s) = 0.4$, $\pi_\theta(a|s) = 0.5$, advantage $A = 2$. surrogate contribution?

    $L = (0.5/0.4) \cdot 2 = 1.25 \cdot 2 = 2.5$।

  3. Why δ matters: δ = 0.001 vs 0.1 — কী difference?

    Small δ — conservative steps, slow learning, very stable. Large δ — aggressive, faster but risky. Schulman δ=0.01 default — sweet spot most tasks.

আরও পড়ুন

এই পাঠে ভুল পেয়েছেন? জানান
পূর্ববর্তী পাঠ
পাঠ ১৯ · A2C & A3C