পাঠ ১০ · ৩৩-এর মধ্যে · মডিউল ২
Home / AI Courses / MLOps / DVC

Data versioning — DVC

DVC — Data Version Control
৮ মিনিট পড়া মধ্য · Intermediate CLI + YAML

এই পাঠে যা শিখবেন

  • DVC কীভাবে কাজ করে — .dvc file ও content-addressable storage
  • Basic workflow — add, push, pull, checkout
  • DVC pipelines — stage definition + caching
  • DVC vs Git LFS vs lakeFS — কখন কোনটা

১ · কেন data version control

Code git-এ — সমস্যা নেই। কিন্তু ৫ GB-এর dataset, ১০ GB-এর model — git-এ commit করলে repo phul, GitHub block।

প্রশ্ন — "আজকের prod model কোন data-এ trained?" — এই answer-এর reproducibility-এর জন্য data version লাগে। DVCDVC (Data Version Control)Iterative.ai-এর open-source tool। Git এর সাথে integrate — large file/dataset-এর জন্য content-addressable hash-based versioning। এই সমস্যা solve করে।

২ · কীভাবে কাজ করে

  • Big file → DVC সেটার MD5/SHA hash compute করে।
  • File-কে remote storage-এ upload করে (S3 path = hash-based)।
  • Repo-তে একটি ছোট .dvc file create — hash + size store।
  • Git track করে শুধু .dvc file (small text)।
  • অন্য কেউ dvc pull দিলে — hash থেকে remote file resolve।
কী track হয়, কোথায়

Git-এ: code + .dvc file (pointers) + dvc.yaml (pipeline)।
DVC remote-এ: actual data files (hash-named)।
একটি repo clone-এ — code আসে; dvc pull-এ data আসে।

৩ · Basic workflow

Bash · DVC quickstart 
# initialize
$ git init
$ dvc init
$ git commit -m "init dvc"

# remote setup (S3 example)
$ dvc remote add -d storage s3://my-ml-bucket/dvc-store
$ git commit .dvc/config -m "add s3 remote"

# track a dataset
$ dvc add data/bangla-reviews.csv
# creates: data/bangla-reviews.csv.dvc  (small pointer file)
# updates: .gitignore (data file ignored)
$ git add data/bangla-reviews.csv.dvc .gitignore
$ git commit -m "add bangla reviews v1"

# push data to remote
$ dvc push

# someone else clones + pulls
$ git clone 
$ dvc pull             # downloads actual data from S3

# update data
$ # edit data/bangla-reviews.csv
$ dvc add data/bangla-reviews.csv
$ git commit data/bangla-reviews.csv.dvc -m "v2: more samples"
$ dvc push

# rollback to v1
$ git checkout HEAD~1 -- data/bangla-reviews.csv.dvc
$ dvc checkout

    


    

      Notice: data file কখনো git commit-এ যাচ্ছে না — শুধু .dvc pointer। Git history-তে data version-গুলো traceable, কিন্তু repo size ছোট।
    


    
৪ · DVC pipelines


    

      Reproducible workflow — dvc.yaml-এ stage define। DVC dependency graph build করে; কী change হয়েছে শুধু সেটাই rerun হয় (caching)।
    


    

      

        
        YAML · dvc.yaml
      

      
stages:
  prepare:
    cmd: python src/prepare.py data/raw data/prepared
    deps:
      - data/raw
      - src/prepare.py
    outs:
      - data/prepared

  featurize:
    cmd: python src/featurize.py data/prepared data/features
    deps:
      - data/prepared
      - src/featurize.py
    outs:
      - data/features

  train:
    cmd: python src/train.py data/features models/model.joblib
    deps:
      - data/features
      - src/train.py
    outs:
      - models/model.joblib
    metrics:
      - metrics.json:
          cache: false

  evaluate:
    cmd: python src/evaluate.py models/model.joblib data/features
    deps:
      - models/model.joblib
      - data/features
      - src/evaluate.py
    metrics:
      - eval.json:
          cache: false
    plots:
      - confusion_matrix.png

      

        
        
      

      

    


    

      dvc repro সব stage chain-এ run করে। যদি শুধু train.py change হয় — prepare + featurize cached থাকে, শুধু train + evaluate rerun। ML pipeline-এর "make" tool।
    


    
৫ · DVC vs alternatives


    
    
      
  • DVC: hash-based, git-integrated, free OSS, pipeline support। Bangladesh team-এর জন্য common choice।
    • 
      
  • Git LFS: GitHub-integrated, simpler, but $-storage cost, no pipelines। Small binary files (~MB) ভাল।
    • 
      
  • lakeFS: "Git for data lake" — branch/merge data। S3-compatible API। Bigger team, complex data lake।
    • 
      
  • S3 versioning: simplest — bucket version-on। No pipelines, no diff support।
    • 
      
  • HuggingFace Hub: dataset/model versioning + community sharing। Public-friendly।
    • 
    


    
৬ · Hash performance & big data


    

      DVC default MD5 use করে। Large file (GB+)-এ hashing slow হতে পারে। Optimizations:
    

    
    
      
  • dvc config core.hash_jobs N — parallel hashing।
    • 
      
  • Directories — DVC tree-hash; nested files efficient।
    • 
      
  • Symlink / reflink mode — file copy এড়ায় (cache.type)।
    • 
      
  • For >100 GB datasets — DVC works but think lakeFS for branching needs।
    • 
    


    
৭ · CI integration


    
    
      
  • CI runner-এ dvc pull — data fetch।
    • 
      
  • S3/GCS credentials — secrets manager।
    • 
      
  • dvc repro — pipeline rerun reproducibility check।
    • 
      
  • Cache reuse — fast re-runs।
    • 
    


    

      
        
          
            
          
        

        DVC — git pointers, remote stores actual files
        Code in git, data in S3

        
        
        Local workspace

        
        📝 src/train.py (git)

        
        📋 data.csv.dvc (git)

        
        📊 data.csv (gitignored)

        
        ⚙️ dvc.yaml (git)

        
        
        GitHub (small)
        .py + .dvc + .yaml
        ~MB total
        history fast

        
        
        S3 / GCS (big)
        data files (hashed)
        ~GB to TB
        deduplicated

        
        
        Teammate
        git clone +
        dvc pull

        
        
        git push
        
        dvc push

        
        
      
      
DVC = git split: code goes to GitHub (small, fast); data goes to S3-compatible (big, hashed)। .dvc file repo-তে — hash-pointer।

    


    

      Bangladesh team-এর জন্য DVC + S3 (AWS Mumbai) — most common stack। On-prem MinIO-ও DVC remote হিসেবে use। Setup ১ ঘণ্টায়, daily benefit বছরভর।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "DVC vs Git LFS vs lakeFS — Bangladesh ML team-এর জন্য কোনটা?"
      
      

        

          এই তিনটি tool different problem space — choose carefully।
        

        
Git LFS:

        
    
          
  • GitHub-এ deeply integrated।
    • 
          
  • Small binary (image, PDF, model file ~MB)।
    • 
          
  • Free tier ১ GB, paid bandwidth।
    • 
          
  • No pipeline support।
    • 
          
  • Best for: simple ML repo with few binary artifacts।
    • 
        

        
DVC:

        
    
          
  • Open source, free OSS।
    • 
          
  • Any S3/GCS/Azure/SSH backend।
    • 
          
  • Pipeline support (dvc.yaml)।
    • 
          
  • Best for: ML team with structured pipeline + cost-sensitive।
    • 
        

        
lakeFS:

        
    
          
  • Git-like branch/merge for data lakes।
    • 
          
  • S3-compatible API — applications transparent।
    • 
          
  • Heavier infrastructure (manifest store, k8s deployment)।
    • 
          
  • Best for: data engineering team, large-scale data lake (TB+)।
    • 
        

        
Bangladesh decision matrix:

        
    
          
  • Single ML developer, GitHub-only → LFS।
    • 
          
  • Small ML team (2-5), cost-sensitive, structured pipeline → DVC।
    • 
          
  • Larger team (10+) with data engineers, complex data lake → lakeFS।
    • 
          
  • Enterprise + budget — Databricks/Snowflake-এর built-in versioning, both DVC + lakeFS replace।
    • 
        

        
Common Bangladesh setup:

        
    
          
  • ৮০% — DVC + S3 (Mumbai)।
    • 
          
  • ~১০% — Git LFS for simple repos।
    • 
          
  • ~৫% — lakeFS at larger orgs।
    • 
          
  • ~৫% — proprietary (Databricks)।
    • 
        

        
Migration possibility:

        
    
          
  • LFS → DVC: relatively easy। Pull files, dvc add।
    • 
          
  • DVC → lakeFS: data move + path update; harder।
    • 
          
  • Most teams stick with first choice years।
    • 
        

        
Practical advice:

        
    
          
  • Start with DVC unless specific reason।
    • 
          
  • S3 Mumbai bucket (low-latency from BD)।
    • 
          
  • Bandwidth cost: egress care; team in same region cheap।
    • 
        

        

          মূল উপলব্ধি: Bangladesh ML team-এর majority — DVC sweet spot। Free, flexible, pipeline-aware। LFS too simple at scale, lakeFS too heavy for small team। DVC + MinIO (on-prem) compliance setup-এও extends।
        

      

    


    

      
        প্র ০২
        "Large dataset (50 GB+) hashing slow। Workflow practical?"
      
      

        

          Hashing performance — DVC-এর scaling concern।
        

        
Default behavior:

        
    
          
  • MD5 hash per file। 50 GB single file → ~৩-৫ মিনিট hashing।
    • 
          
  • Many small files vs one big file: many small slower (per-file overhead)।
    • 
          
  • Network attached storage (NFS) further slow।
    • 
        

        
Optimization techniques:

        
(১) Parallel hashing:

        
    
          
  • dvc config core.hash_jobs 8 — multi-thread।
    • 
          
  • I/O bound, so doesn't scale linearly with cores। ৪x typical।
    • 
        

        
(২) Different hash algorithm:

        
    
          
  • DVC 3.0+: md5 default; sha256 available।
    • 
          
  • xxhash faster (custom build)।
    • 
        

        
(৩) Cache type:

        
    
          
  • Default: copy file to cache — slow, large disk।
    • 
          
  • Symlink: dvc config cache.type symlink — instant।
    • 
          
  • Reflink (CoW): Btrfs/APFS — best of both। On-prem-এ reflink common nowadays।
    • 
        

        
(৪) Partial dataset versioning:

        
    
          
  • Don't dvc-track raw 50 GB landmass。 Filter to working subset। Track that।
    • 
          
  • Raw data: external versioning (S3 native)। DVC tracks the prepared subset।
    • 
        

        
(৫) Directory-level vs file-level:

        
    
          
  • dvc add on directory — DVC computes a tree hash + per-file hashes।
    • 
          
  • Faster than per-file invocation।
    • 
        

        
Practical large dataset workflow:

        
    
          
  • Initial dvc add: one-time slow (overnight if needed)।
    • 
          
  • Subsequent updates: only changed files re-hash।
    • 
          
  • "Append-only" dataset patterns: monthly chunk add — faster।
    • 
        

        
Bandwidth concerns:

        
    
          
  • dvc pull 50 GB on 10 Mbps link — ১২ ঘণ্টা। Painful।
    • 
          
  • Solution: regional S3, faster local cache, partial pull।
    • 
          
  • dvc pull data/subset/ — only that subdirectory।
    • 
        

        
"Don't version huge raw, version derived":

        
    
          
  • Raw 500 GB photos — keep S3 versioned, not DVC।
    • 
          
  • Preprocessed 5 GB embeddings — DVC track।
    • 
          
  • Most ML iteration on derived; raw rarely changes।
    • 
        

        
Real BD example:

        
    
          
  • One e-commerce team: 200 GB raw images (S3 versioned), 8 GB feature vectors (DVC), models in registry।
    • 
          
  • Each layer right tool।
    • 
        

        

          মূল উপলব্ধি: DVC theoretical limits beyond practical limits — but discipline matters। Don't dvc-add everything। Tier strategy: raw S3-native, derived DVC, models registry। Hashing slow only on first add; subsequent fast।
        

      

    


    

      
        প্র ০৩
        "DVC pipelines — কখন worth setting up vs simple Python script?"
      
      

        

          DVC pipelines (dvc.yaml) — like Make for ML। Setup overhead vs benefit।
        

        
Simple Python script enough when:

        
    
          
  • One-shot training, no recurrent data prep।
    • 
          
  • Solo developer, small project।
    • 
          
  • No expensive intermediate steps to cache।
    • 
        

        
DVC pipeline worth when:

        
    
          
  • Multi-stage workflow (download → preprocess → feature → train → eval)।
    • 
          
  • Some stages expensive (preprocessing 2 hours)।
    • 
          
  • Iterative experimentation — change one step, others should not rerun।
    • 
          
  • Team — onboarding through pipeline definition।
    • 
          
  • Reproducibility audit need।
    • 
        

        
Caching benefit example:

        
    
          
  • Stage A (download): 30 min। Stage B (preprocess): 2 hours। Stage C (train): 30 min।
    • 
          
  • Without DVC: change train.py → 3 hours total rerun।
    • 
          
  • With DVC: A & B cached → 30 min only।
    • 
          
  • 10 iterations: 30 hour saved।
    • 
        

        
DVC vs Airflow vs Make:

        
    
          
  • Make: file-based dependency, simple। But no data versioning integration।
    • 
          
  • DVC: file + data version aware, ML-specific। Local & CI।
    • 
          
  • Airflow: production orchestration, schedule, distributed। Heavier।
    • 
          
  • Common: DVC for local dev, Airflow for production scheduled runs।
    • 
        

        
Bangladesh context — ROI tipping point:

        
    
          
  • Solo + simple → script।
    • 
          
  • 3+ engineer or pipeline > 1 hour → DVC যথেষ্ট value।
    • 
          
  • Production daily schedule → Airflow + DVC together।
    • 
        

        
Common pitfalls:

        
    
          
  • Stages not idempotent → retry breaks।
    • 
          
  • External dependencies not declared → cache invalidation wrong।
    • 
          
  • Output overlap between stages → confusing।
    • 
          
  • "Mega-stage" doing everything → no caching benefit।
    • 
        

        
Tip — start small:

        
    
          
  • First just dvc add data, no pipeline।
    • 
          
  • Later add 2-3 stage pipeline।
    • 
          
  • Don't over-engineer initial setup।
    • 
        

        

          মূল উপলব্ধি: DVC pipelines — multi-stage iterative ML-এ excellent। Solo simple script — overkill। Tipping point usually at: ৩+ stages, expensive intermediate, team আকার ২+।
        

      

    


    

      
        প্র ০৪
        "PII / sensitive data — DVC-এর সাথে কীভাবে handle?"
      
      

        

          Sensitive data versioning special care দাবি করে।
        

        
DVC-এর data flow risks:

        
    
          
  • Local cache (default .dvc/cache) — file copy। Disk encrypt না হলে exposed।
    • 
          
  • Remote storage — encryption depends on backend (S3 SSE, GCS default)।
    • 
          
  • Hash-as-key — hash itself doesn't leak data, but file contents in storage do।
    • 
          
  • Git history — .dvc file with hash; if leak, attacker knows what to try if data accessible।
    • 
        

        
Anonymization-first approach:

        
    
          
  • NID, mobile, email — hash before DVC track।
    • 
          
  • "Pseudonymization at source" — original PII never enters versioned data।
    • 
          
  • Bangladesh Bank guideline এর সাথে aligned।
    • 
        

        
Encryption layers:

        
    
          
  • S3 SSE-KMS: server-side encryption with managed keys।
    • 
          
  • S3 SSE-C: customer-provided keys।
    • 
          
  • Client-side encryption: data encrypt before upload (sodium, AES)।
    • 
          
  • DVC currently no built-in encryption layer; rely on backend।
    • 
        

        
Access control:

        
    
          
  • S3 bucket policy — least privilege। Specific IAM roles, no wildcard।
    • 
          
  • VPC endpoint — bucket access only from internal network।
    • 
          
  • MFA delete — accidental deletion prevention।
    • 
        

        
GDPR / Bangladesh-future regulation:

        
    
          
  • "Right to be forgotten" — versioned data complicate। Customer delete request — all versions purge।
    • 
          
  • Solution: customer ID-keyed storage, surgical delete possible।
    • 
          
  • Or: anonymize before versioning — avoid the problem।
    • 
        

        
Alternative for highly sensitive:

        
    
          
  • Don't DVC-version raw PII at all।
    • 
          
  • Version statistics + hashes; access raw data on-demand from primary store।
    • 
          
  • Audit log of data access।
    • 
        

        
Cache encryption (local):

        
    
          
  • FileVault (Mac), LUKS (Linux), BitLocker (Windows) — full disk।
    • 
          
  • OR DVC cache external encrypted volume।
    • 
          
  • Team policy: laptop encryption mandatory।
    • 
        

        
Common Bangladesh setup for sensitive data:

        
    
          
  • Anonymize (replace PII with hash)।
    • 
          
  • DVC + on-prem MinIO (data leaves not country)।
    • 
          
  • Encryption at rest (MinIO server-side)।
    • 
          
  • RBAC — bucket per project/team।
    • 
        

        

          মূল উপলব্ধি: Sensitive data + DVC = anonymize first। Raw PII version-control এ rarely needed for ML। Hash-pseudonymized version DVC-friendly + privacy-friendly। On-prem MinIO + encrypted backend BD compliance নির্ভরযোগ্য।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Setup: একটি local DVC repo তৈরি করুন (local filesystem-এর remote)। একটি CSV file dvc add ও commit করুন।
        
    
          
          
    
            
    dvc init && dvc remote add -d local /tmp/dvc-store && dvc add data.csv && git commit data.csv.dvc -m "v1"। দেখুন .dvc file কী contain করে — hash + size।
    
          
    
        
    
      
    2. 
      
  2. 
        Pipeline: উপরের dvc.yaml একটি simplified version (২ stage) তৈরি করুন। dvc repro চালান।
        
    
          
          
    
            
    প্রথম time-এ both stage execute। Second time same input — "Cached" message। শুধু code change করলে দেখবেন প্রভাবিত stage rerun।
    
          
    
        
    
      
    3. 
      
  3. 
        চিন্তা: আপনার একটি ৫ GB raw data আছে। Daily ১০০ MB add হয়। DVC strategy কী?
        
    
          
          
    
            
    "Append-only" pattern। Raw chunks daily-stamped folder-এ। Tree-hash দিয়ে directory track। Pipeline-এ raw filter → working subset। Working subset frequently version। Disk usage controlled।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৯ · Weights & Biases