পাঠ ৩২ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / Pose estimation

Pose estimation

Human pose estimation — keypoint detection
৭ মিনিট পড়া মাঝারি · Intermediate MediaPipe কোড

এই পাঠে যা শিখবেন

  • Pose keypoint convention
  • Top-down vs bottom-up
  • Heatmap regression
  • MediaPipe practical use

১ · Body keypoint convention

  • COCO 17-keypoint: standard — eyes, ears, shoulders, elbows, wrists, hips, knees, ankles।
  • OpenPose 25-keypoint: + foot, neck।
  • MediaPipe 33-keypoint: face details added।
  • Hand: 21 keypoint per hand।
  • Face: 68 (dlib) বা 468 (MediaPipe)।
কেন্দ্রীয় ধারণা

Pose estimation = "structured keypoint regression"। Per-pixel heatmap predict — peak = keypoint location। Sub-pixel accuracy from heatmap distribution।

২ · Top-down approach

  1. Person detect (Faster R-CNN, YOLO)।
  2. Each person crop → CNN → 17 heatmap (one per keypoint)।
  3. Heatmap argmax → keypoint coordinate।
  • Models: SimpleBaseline, HRNet, ViTPose।
  • Pros: high accuracy, well-localized।
  • Cons: per-person inference, slower।

৩ · Bottom-up approach

  1. Single CNN forward — all keypoint heatmap।
  2. Part Affinity Field (OpenPose) — keypoint connect।
  3. Group keypoint to person।
  • Models: OpenPose, HigherHRNet।
  • Pros: single forward, scale to multi-person।
  • Cons: grouping ambiguity in crowd।

৪ · Heatmap regression

Each keypoint → 2D Gaussian heatmap centered at ground-truth location।

$$H(x, y) = \exp\left( -\frac{(x - x_k)^2 + (y - y_k)^2}{2\sigma^2} \right)$$

  • Network predict heatmap, MSE loss।
  • Inference: argmax → integer coordinate।
  • Sub-pixel: integrate near peak।

৫ · HRNet (২০১৯)

Sun et al. — High-Resolution Network। Pose-এ paradigm shift।

  • Multi-resolution parallel branches।
  • High-resolution preserved throughout।
  • Spatial precision keypoint detect-এ critical।
  • COCO pose 75% AP — SOTA।

৬ · MediaPipe Pose (Google, ২০২০)

  • 33-keypoint, mobile-optimized।
  • Two-stage — pose detector + landmark।
  • Tracking via temporal smoothing।
  • Realtime on smartphone।
  • Open source, easy use।

৭ · 3D pose estimation

  • 2D keypoint → 3D coordinate predict।
  • Methods: VideoPose3D (Pavllo et al.), MediaPipe BlazePose।
  • Application: motion capture, AR, animation।
  • SMPL model — full body mesh।
Pose estimation = "stick figure extract from photo"। Each joint precise location। Animation, fitness, AR — all build on this stick figure।
Pose estimation — keypoint detection Image with pose CNN/HRNet multi-res extract 17 heatmap output 17 heatmaps per keypoint argmax → coord Heatmap-based — sub-pixel keypoint precise Top-down (per person crop) vs bottom-up (single forward)
Pose estimation — image → CNN → 17 heatmap → argmax keypoint। Stick figure extract।

৮ · MediaPipe practical

Python · MediaPipe 
# pip install mediapipe
import mediapipe as mp
import cv2

mp_pose = mp.solutions.pose
pose = mp_pose.Pose(
    static_image_mode=True,
    model_complexity=2,            # 0=lite, 1=full, 2=heavy
    enable_segmentation=False,
    min_detection_confidence=0.5
)

img = cv2.imread('person.jpg')
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
results = pose.process(img_rgb)

if results.pose_landmarks:
    h, w = img.shape[:2]
    for i, lm in enumerate(results.pose_landmarks.landmark):
        x, y = int(lm.x * w), int(lm.y * h)
        print(f"Landmark {i}: ({x}, {y}), confidence: {lm.visibility:.2f}")
        cv2.circle(img, (x, y), 5, (0, 255, 0), -1)

cv2.imwrite('output.jpg', img)
pose.close()

    


    

      MediaPipe — extremely user-friendly। Mobile realtime। Hand, face, pose, holistic — all included।
    


    
৯ · Use cases

    
    
      
  • Fitness app: push-up count, yoga form correct।
    • 
      
  • Sports analytics: player tracking, biomechanics।
    • 
      
  • Animation: motion capture for game/film।
    • 
      
  • AR/VR: body tracking, virtual try-on।
    • 
      
  • Healthcare: rehabilitation, gait analysis।
    • 
      
  • Sign language: pose + hand recognition।
    • 
      
  • Action recognition: walking, running, falling detect।
    • 
    


    
১০ · Modern models

    
    
      
  • HRNet: high-resolution preserve। Top-down accurate।
    • 
      
  • ViTPose (২০২২): ViT backbone। 81% AP।
    • 
      
  • RTMPose (২০২৩): realtime SOTA।
    • 
      
  • MMPose (OpenMMLab): comprehensive library।
    • 
      
  • SMPL-X: full body mesh + face + hands।
    • 
    


    
১১ · Bangladesh applications

    
    
      
  • Yoga/fitness app for local market।
    • 
      
  • Cricket batting/bowling analysis।
    • 
      
  • Bangla sign language recognition।
    • 
      
  • Rehabilitation hospital — physiotherapy assistant।
    • 
      
  • Virtual saree try-on।
    • 
      
  • Dance form analysis (classical Bangla dance)।
    • 
    


    

      Pose estimation crowded scene-এ struggle। Occlusion, similar clothing — accuracy drop। Multiple person — explicit ID tracking দরকার (DeepSORT)।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        Heatmap regression vs direct coordinate regression — কেন heatmap প্রায়ই better?
      
      

        
Pose estimation-এর foundational design choice।

        
Direct regression:

        
    
          
  • Network output: 17 × 2 = 34 number (x, y per keypoint)।
    • 
          
  • L2 loss।
    • 
          
  • Simple, fast।
    • 
        

        
Heatmap regression:

        
    
          
  • Network output: 17 × H × W heatmap।
    • 
          
  • Each heatmap — Gaussian peak at keypoint।
    • 
          
  • Inference: argmax।
    • 
        

        
Why heatmap better:

        
    
          
  • Spatial supervision: per-pixel signal — denser gradient।
    • 
          
  • Multi-modal output: uncertainty natural represent।
    • 
          
  • Sub-pixel accuracy: peak interpolation।
    • 
          
  • Translation equivariance: CNN natural fit।
    • 
        

        
Empirical:

        
    
          
  • Direct: 65% PCK।
    • 
          
  • Heatmap: 80%+ PCK।
    • 
          
  • Significant gap।
    • 
        

        
Direct still useful:

        
    
          
  • RTMPose — direct regression with tricks।
    • 
          
  • DETR-style — set prediction।
    • 
          
  • Mobile/realtime — direct faster।
    • 
        

        
Hybrid:

        
    
          
  • Heatmap + offset regression (coordinate refine)।
    • 
          
  • Best of both।
    • 
        

        
মূল উপলব্ধি: Output structure matter। Heatmap CNN-এর spatial nature exploit। Modern direct regression catching up।

      

    


    

      
        প্র ০২
        OpenPose Part Affinity Field (PAF) কী? Bottom-up grouping কীভাবে?
      
      

        
OpenPose-এর key innovation।

        
Problem:

        
    
          
  • Single CNN — all person-এর keypoint detect।
    • 
          
  • Multiple wrist heatmap (multiple people)।
    • 
          
  • "This wrist belongs to which person's elbow?" — ambiguity।
    • 
        

        
PAF idea:

        
    
          
  • Each limb (e.g., elbow→wrist) — vector field predict।
    • 
          
  • Vector points along limb direction।
    • 
          
  • Path integral between two keypoint = PAF score।
    • 
          
  • High score → likely connected।
    • 
        

        
Output:

        
    
          
  • 17 keypoint heatmap।
    • 
          
  • 2 × 19 limb PAF (x, y direction per limb)।
    • 
        

        
Grouping algorithm:

        
    
          
  1. NMS on heatmap → candidate keypoint।
    2. 
          
  2. Pair-wise PAF score compute।
    3. 
          
  3. Greedy bipartite matching per limb type।
    4. 
          
  4. Connected component → person।
    5. 
        

        
Advantages:

        
    
          
  • Single forward pass — multi-person efficient।
    • 
          
  • Realtime even with 10+ people।
    • 
        

        
Limitations:

        
    
          
  • Crowded scene — grouping fail।
    • 
          
  • Occlusion — broken connection।
    • 
          
  • Top-down accuracy higher।
    • 
        

        
Modern variants:

        
    
          
  • HigherHRNet — better resolution।
    • 
          
  • Associative Embedding — learned grouping।
    • 
          
  • CenterNet pose — center + offset।
    • 
        

        
মূল উপলব্ধি: Bottom-up grouping — combinatorial structure CNN-এ encode। Vector field clever representation।

      

    


    

      
        প্র ০৩
        3D pose estimation — 2D image থেকে 3D infer। Ill-posed problem? কীভাবে কাজ করে?
      
      

        
3D pose — geometric ill-posed but ML solvable।

        
Why ill-posed:

        
    
          
  • 2D image — depth information lose।
    • 
          
  • Multiple 3D pose project to same 2D।
    • 
          
  • "Pose ambiguity"।
    • 
        

        
Approaches:

        
    
          
  • Lifting: 2D keypoint → 3D coordinate (Martinez 2017)।
    • 
          
  • Direct: image → 3D directly।
    • 
          
  • Multi-view: multiple camera triangulation।
    • 
          
  • Temporal: video — temporal consistency।
    • 
          
  • SMPL-based: parametric body model।
    • 
        

        
Lifting-based:

        
    
          
  • Train on motion capture data (e.g., Human3.6M)।
    • 
          
  • Learn statistical prior — "natural pose"।
    • 
          
  • Resolve ambiguity using prior।
    • 
        

        
SMPL (Skinned Multi-Person Linear, ২০১৫):

        
    
          
  • Parametric body model।
    • 
          
  • 10 shape + 72 pose parameter।
    • 
          
  • Photo-realistic 3D mesh।
    • 
          
  • VIBE, ROMP — predict SMPL parameter।
    • 
        

        
MediaPipe BlazePose 3D:

        
    
          
  • Per-keypoint x, y, z।
    • 
          
  • z relative to hip (not world coord)।
    • 
          
  • Mobile realtime।
    • 
        

        
Application:

        
    
          
  • Motion capture — film, game।
    • 
          
  • VR avatar।
    • 
          
  • Sports — biomechanics analysis।
    • 
          
  • Healthcare — gait।
    • 
        

        
Bangladesh:

        
    
          
  • Game studio (Battery Low Interactive) — character animation।
    • 
          
  • Film industry — VFX assist।
    • 
          
  • Cricket coaching — biomechanics।
    • 
        

        
মূল উপলব্ধি: ill-posed problem — ML prior দিয়ে solvable। 3D pose modern CV-র mature subfield।

      

    


    

      
        প্র ০৪
        Bangladesh fitness app — pushup counter। MediaPipe Pose use করে কীভাবে count? Edge case কী?
      
      

        
Real Bangladesh use case। Local startups (e.g., Yoga House BD) explore।

        
Pushup detection logic:

        
    
          
  1. MediaPipe Pose extract keypoint।
    2. 
          
  2. Calculate elbow angle: shoulder-elbow-wrist।
    3. 
          
  3. Track angle across frames।
    4. 
          
  4. State machine:
            
      
              
    • UP: elbow ~170° (arms extended)।
      • 
              
    • DOWN: elbow ~90° (chest near floor)।
      • 
              
    • UP→DOWN→UP = one rep।
      • 
            
    
          
    5. 
          
  5. Count valid transitions।
    6. 
        

        
Form check:

        
    
          
  • Body straight: shoulder-hip-ankle alignment।
    • 
          
  • Hip drop detect — bad form।
    • 
          
  • Visual feedback to user।
    • 
        

        
Edge cases:

        
    
          
  • Lighting: dim — keypoint detection fail।
    • 
          
  • Camera angle: non-side view — angle calculation off।
    • 
          
  • Loose clothing: hide body landmark।
    • 
          
  • Background clutter: false detection।
    • 
          
  • Multiple people: pick "main" person (closest/largest)।
    • 
          
  • Phone shake: noisy keypoint — temporal smooth।
    • 
          
  • Mat/clothes color: contrast issue।
    • 
        

        
Mitigation:

        
    
          
  • Onboarding: optimal camera position guide।
    • 
          
  • Confidence threshold — low conf reject।
    • 
          
  • Median filter on angle signal।
    • 
          
  • Min/max angle thresholds tuned।
    • 
        

        
App architecture:

        
    
          
  • Mobile camera → MediaPipe (on-device)।
    • 
          
  • Realtime feedback overlay।
    • 
          
  • Voice cue ("good form", "lower")।
    • 
          
  • Backend — workout history sync।
    • 
        

        
Bangla touch:

        
    
          
  • Bangla TTS feedback ("ভাল!", "নিচে")।
    • 
          
  • Local exercise (e.g., burpees, traditional)।
    • 
          
  • Rural area — low-internet operation।
    • 
        

        
Business consideration:

        
    
          
  • Premium fitness app subscription।
    • 
          
  • Personal trainer + AI hybrid।
    • 
          
  • Corporate wellness program।
    • 
        

        
Privacy:

        
    
          
  • On-device processing — no cloud upload।
    • 
          
  • User data: count, not video।
    • 
          
  • Transparent privacy policy।
    • 
        

        
মূল উপলব্ধি: Domain-specific application — pose-based AI rich opportunity। Bangladesh — fitness, healthcare, sports vertical underserved।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        MediaPipe pose: Webcam realtime pose detect। 33 landmark visualize।
        
    
          
          
    
            
    cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    results = pose.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    if results.pose_landmarks:
        mp.solutions.drawing_utils.draw_landmarks(
            frame, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
    cv2.imshow('Pose', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'): break
    
          
    
        
    
      
    2. 

      
  2. 
        Squat counter: knee angle track, count squat repetition।
        
    
          
          
    
            
    Hip-knee-ankle angle compute। DOWN: ~90°, UP: ~170°। State machine count rep।
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangla cricket coach app — bowler arm action analyze। Pose estimation কীভাবে?
        
    
          
          
    
            
    Side-view video → 3D pose। Arm angle, release point, follow-through measure। Compare with pro reference. Real-time feedback।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৩১ · Face recognition