পাঠ ৩২ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / Pose estimation

Pose estimation

Human pose estimation — keypoint detection
৭ মিনিট পড়া মাঝারি · Intermediate MediaPipe কোড

এই পাঠে যা শিখবেন

  • Pose keypoint convention
  • Top-down vs bottom-up
  • Heatmap regression
  • MediaPipe practical use

১ · Body keypoint convention

  • COCO 17-keypoint: standard — eyes, ears, shoulders, elbows, wrists, hips, knees, ankles.
  • OpenPose 25-keypoint: + foot, neck.
  • MediaPipe 33-keypoint: face details added.
  • Hand: 21 keypoint per hand.
  • Face: 68 (dlib) বা 468 (MediaPipe).
কেন্দ্রীয় ধারণা

Pose estimation = "structured keypoint regression". Per-pixel heatmap predict — peak = keypoint location. Sub-pixel accuracy from heatmap distribution.

২ · Top-down approach

  1. Person detect (Faster R-CNN, YOLO).
  2. Each person crop → CNN → 17 heatmap (one per keypoint).
  3. Heatmap argmax → keypoint coordinate.
  • Models: SimpleBaseline, HRNet, ViTPose.
  • Pros: high accuracy, well-localized.
  • Cons: per-person inference, slower.

৩ · Bottom-up approach

  1. Single CNN forward — all keypoint heatmap.
  2. Part Affinity Field (OpenPose) — keypoint connect.
  3. Group keypoint to person.
  • Models: OpenPose, HigherHRNet.
  • Pros: single forward, scale to multi-person.
  • Cons: grouping ambiguity in crowd.

৪ · Heatmap regression

Each keypoint → 2D Gaussian heatmap centered at ground-truth location.

$$H(x, y) = \exp\left( -\frac{(x - x_k)^2 + (y - y_k)^2}{2\sigma^2} \right)$$

  • Network predict heatmap, MSE loss.
  • Inference: argmax → integer coordinate.
  • Sub-pixel: integrate near peak.

৫ · HRNet (২০১৯)

Sun et al. — High-Resolution Network. Pose-এ paradigm shift.

  • Multi-resolution parallel branches.
  • High-resolution preserved throughout.
  • Spatial precision keypoint detect-এ critical.
  • COCO pose 75% AP — SOTA.

৬ · MediaPipe Pose (Google, ২০২০)

  • 33-keypoint, mobile-optimized.
  • Two-stage — pose detector + landmark.
  • Tracking via temporal smoothing.
  • Realtime on smartphone.
  • Open source, easy use.

৭ · 3D pose estimation

  • 2D keypoint → 3D coordinate predict.
  • Methods: VideoPose3D (Pavllo et al.), MediaPipe BlazePose.
  • Application: motion capture, AR, animation.
  • SMPL model — full body mesh.
Pose estimation = "stick figure extract from photo". Each joint precise location. Animation, fitness, AR — all build on this stick figure.
Pose estimation — keypoint detection Image with pose CNN/HRNet multi-res extract 17 heatmap output 17 heatmaps per keypoint argmax → coord Heatmap-based — sub-pixel keypoint precise Top-down (per person crop) vs bottom-up (single forward)
Pose estimation — image → CNN → 17 heatmap → argmax keypoint. Stick figure extract.

৮ · MediaPipe practical

Python · MediaPipe 
# pip install mediapipe
import mediapipe as mp
import cv2

mp_pose = mp.solutions.pose
pose = mp_pose.Pose(
    static_image_mode=True,
    model_complexity=2,            # 0=lite, 1=full, 2=heavy
    enable_segmentation=False,
    min_detection_confidence=0.5
)

img = cv2.imread('person.jpg')
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
results = pose.process(img_rgb)

if results.pose_landmarks:
    h, w = img.shape[:2]
    for i, lm in enumerate(results.pose_landmarks.landmark):
        x, y = int(lm.x * w), int(lm.y * h)
        print(f"Landmark {i}: ({x}, {y}), confidence: {lm.visibility:.2f}")
        cv2.circle(img, (x, y), 5, (0, 255, 0), -1)

cv2.imwrite('output.jpg', img)
pose.close()

    


    

      MediaPipe — extremely user-friendly. Mobile realtime. Hand, face, pose, holistic — all included.
    


    
৯ · Use cases

    
    
      
  • Fitness app: push-up count, yoga form correct.
    • 
      
  • Sports analytics: player tracking, biomechanics.
    • 
      
  • Animation: motion capture for game/film.
    • 
      
  • AR/VR: body tracking, virtual try-on.
    • 
      
  • Healthcare: rehabilitation, gait analysis.
    • 
      
  • Sign language: pose + hand recognition.
    • 
      
  • Action recognition: walking, running, falling detect.
    • 
    


    
১০ · Modern models

    
    
      
  • HRNet: high-resolution preserve. Top-down accurate.
    • 
      
  • ViTPose (২০২২): ViT backbone. 81% AP.
    • 
      
  • RTMPose (২০২৩): realtime SOTA.
    • 
      
  • MMPose (OpenMMLab): comprehensive library.
    • 
      
  • SMPL-X: full body mesh + face + hands.
    • 
    


    
১১ · Bangladesh applications

    
    
      
  • Yoga/fitness app for local market.
    • 
      
  • Cricket batting/bowling analysis.
    • 
      
  • Bangla sign language recognition.
    • 
      
  • Rehabilitation hospital — physiotherapy assistant.
    • 
      
  • Virtual saree try-on.
    • 
      
  • Dance form analysis (classical Bangla dance).
    • 
    


    

      Pose estimation crowded scene-এ struggle. Occlusion, similar clothing — accuracy drop. Multiple person — explicit ID tracking দরকার (DeepSORT).
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        Heatmap regression vs direct coordinate regression — কেন heatmap প্রায়ই better?
      
      

        
Pose estimation-এর foundational design choice.

        
Direct regression:

        
    
          
  • Network output: 17 × 2 = 34 number (x, y per keypoint).
    • 
          
  • L2 loss.
    • 
          
  • Simple, fast.
    • 
        

        
Heatmap regression:

        
    
          
  • Network output: 17 × H × W heatmap.
    • 
          
  • Each heatmap — Gaussian peak at keypoint.
    • 
          
  • Inference: argmax.
    • 
        

        
Why heatmap better:

        
    
          
  • Spatial supervision: per-pixel signal — denser gradient.
    • 
          
  • Multi-modal output: uncertainty natural represent.
    • 
          
  • Sub-pixel accuracy: peak interpolation.
    • 
          
  • Translation equivariance: CNN natural fit.
    • 
        

        
Empirical:

        
    
          
  • Direct: 65% PCK.
    • 
          
  • Heatmap: 80%+ PCK.
    • 
          
  • Significant gap.
    • 
        

        
Direct still useful:

        
    
          
  • RTMPose — direct regression with tricks.
    • 
          
  • DETR-style — set prediction.
    • 
          
  • Mobile/realtime — direct faster.
    • 
        

        
Hybrid:

        
    
          
  • Heatmap + offset regression (coordinate refine).
    • 
          
  • Best of both.
    • 
        

        
মূল উপলব্ধি: Output structure matter. Heatmap CNN-এর spatial nature exploit. Modern direct regression catching up.

      

    


    

      
        প্র ০২
        OpenPose Part Affinity Field (PAF) কী? Bottom-up grouping কীভাবে?
      
      

        
OpenPose-এর key innovation.

        
Problem:

        
    
          
  • Single CNN — all person-এর keypoint detect.
    • 
          
  • Multiple wrist heatmap (multiple people).
    • 
          
  • "This wrist belongs to which person's elbow?" — ambiguity.
    • 
        

        
PAF idea:

        
    
          
  • Each limb (e.g., elbow→wrist) — vector field predict.
    • 
          
  • Vector points along limb direction.
    • 
          
  • Path integral between two keypoint = PAF score.
    • 
          
  • High score → likely connected.
    • 
        

        
Output:

        
    
          
  • 17 keypoint heatmap.
    • 
          
  • 2 × 19 limb PAF (x, y direction per limb).
    • 
        

        
Grouping algorithm:

        
    
          
  1. NMS on heatmap → candidate keypoint.
    2. 
          
  2. Pair-wise PAF score compute.
    3. 
          
  3. Greedy bipartite matching per limb type.
    4. 
          
  4. Connected component → person.
    5. 
        

        
Advantages:

        
    
          
  • Single forward pass — multi-person efficient.
    • 
          
  • Realtime even with 10+ people.
    • 
        

        
Limitations:

        
    
          
  • Crowded scene — grouping fail.
    • 
          
  • Occlusion — broken connection.
    • 
          
  • Top-down accuracy higher.
    • 
        

        
Modern variants:

        
    
          
  • HigherHRNet — better resolution.
    • 
          
  • Associative Embedding — learned grouping.
    • 
          
  • CenterNet pose — center + offset.
    • 
        

        
মূল উপলব্ধি: Bottom-up grouping — combinatorial structure CNN-এ encode. Vector field clever representation.

      

    


    

      
        প্র ০৩
        3D pose estimation — 2D image থেকে 3D infer. Ill-posed problem? কীভাবে কাজ করে?
      
      

        
3D pose — geometric ill-posed but ML solvable.

        
Why ill-posed:

        
    
          
  • 2D image — depth information lose.
    • 
          
  • Multiple 3D pose project to same 2D.
    • 
          
  • "Pose ambiguity".
    • 
        

        
Approaches:

        
    
          
  • Lifting: 2D keypoint → 3D coordinate (Martinez 2017).
    • 
          
  • Direct: image → 3D directly.
    • 
          
  • Multi-view: multiple camera triangulation.
    • 
          
  • Temporal: video — temporal consistency.
    • 
          
  • SMPL-based: parametric body model.
    • 
        

        
Lifting-based:

        
    
          
  • Train on motion capture data (e.g., Human3.6M).
    • 
          
  • Learn statistical prior — "natural pose".
    • 
          
  • Resolve ambiguity using prior.
    • 
        

        
SMPL (Skinned Multi-Person Linear, ২০১৫):

        
    
          
  • Parametric body model.
    • 
          
  • 10 shape + 72 pose parameter.
    • 
          
  • Photo-realistic 3D mesh.
    • 
          
  • VIBE, ROMP — predict SMPL parameter.
    • 
        

        
MediaPipe BlazePose 3D:

        
    
          
  • Per-keypoint x, y, z.
    • 
          
  • z relative to hip (not world coord).
    • 
          
  • Mobile realtime.
    • 
        

        
Application:

        
    
          
  • Motion capture — film, game.
    • 
          
  • VR avatar.
    • 
          
  • Sports — biomechanics analysis.
    • 
          
  • Healthcare — gait.
    • 
        

        
Bangladesh:

        
    
          
  • Game studio (Battery Low Interactive) — character animation.
    • 
          
  • Film industry — VFX assist.
    • 
          
  • Cricket coaching — biomechanics.
    • 
        

        
মূল উপলব্ধি: ill-posed problem — ML prior দিয়ে solvable. 3D pose modern CV-র mature subfield.

      

    


    

      
        প্র ০৪
        Bangladesh fitness app — pushup counter. MediaPipe Pose use করে কীভাবে count? Edge case কী?
      
      

        
Real Bangladesh use case. Local startups (e.g., Yoga House BD) explore.

        
Pushup detection logic:

        
    
          
  1. MediaPipe Pose extract keypoint.
    2. 
          
  2. Calculate elbow angle: shoulder-elbow-wrist.
    3. 
          
  3. Track angle across frames.
    4. 
          
  4. State machine:
            
      
              
    • UP: elbow ~170° (arms extended).
      • 
              
    • DOWN: elbow ~90° (chest near floor).
      • 
              
    • UP→DOWN→UP = one rep.
      • 
            
    
          
    5. 
          
  5. Count valid transitions.
    6. 
        

        
Form check:

        
    
          
  • Body straight: shoulder-hip-ankle alignment.
    • 
          
  • Hip drop detect — bad form.
    • 
          
  • Visual feedback to user.
    • 
        

        
Edge cases:

        
    
          
  • Lighting: dim — keypoint detection fail.
    • 
          
  • Camera angle: non-side view — angle calculation off.
    • 
          
  • Loose clothing: hide body landmark.
    • 
          
  • Background clutter: false detection.
    • 
          
  • Multiple people: pick "main" person (closest/largest).
    • 
          
  • Phone shake: noisy keypoint — temporal smooth.
    • 
          
  • Mat/clothes color: contrast issue.
    • 
        

        
Mitigation:

        
    
          
  • Onboarding: optimal camera position guide.
    • 
          
  • Confidence threshold — low conf reject.
    • 
          
  • Median filter on angle signal.
    • 
          
  • Min/max angle thresholds tuned.
    • 
        

        
App architecture:

        
    
          
  • Mobile camera → MediaPipe (on-device).
    • 
          
  • Realtime feedback overlay.
    • 
          
  • Voice cue ("good form", "lower").
    • 
          
  • Backend — workout history sync.
    • 
        

        
Bangla touch:

        
    
          
  • Bangla TTS feedback ("ভাল!", "নিচে")।
    • 
          
  • Local exercise (e.g., burpees, traditional).
    • 
          
  • Rural area — low-internet operation.
    • 
        

        
Business consideration:

        
    
          
  • Premium fitness app subscription.
    • 
          
  • Personal trainer + AI hybrid.
    • 
          
  • Corporate wellness program.
    • 
        

        
Privacy:

        
    
          
  • On-device processing — no cloud upload.
    • 
          
  • User data: count, not video.
    • 
          
  • Transparent privacy policy.
    • 
        

        
মূল উপলব্ধি: Domain-specific application — pose-based AI rich opportunity. Bangladesh — fitness, healthcare, sports vertical underserved.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        MediaPipe pose: Webcam realtime pose detect. 33 landmark visualize.
        
    
          
          
    
            
    cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    results = pose.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    if results.pose_landmarks:
        mp.solutions.drawing_utils.draw_landmarks(
            frame, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
    cv2.imshow('Pose', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'): break
    
          
    
        
    
      
    2. 

      
  2. 
        Squat counter: knee angle track, count squat repetition.
        
    
          
          
    
            
    Hip-knee-ankle angle compute. DOWN: ~90°, UP: ~170°। State machine count rep.
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangla cricket coach app — bowler arm action analyze. Pose estimation কীভাবে?
        
    
          
          
    
            
    Side-view video → 3D pose. Arm angle, release point, follow-through measure. Compare with pro reference. Real-time feedback.
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৩১ · Face recognition