Data versioning — DVC

এই তিনটি tool different problem space — choose carefully.

Git LFS:

• GitHub-এ deeply integrated.
• Small binary (image, PDF, model file ~MB).
• Free tier ১ GB, paid bandwidth.
• No pipeline support.
• Best for: simple ML repo with few binary artifacts.

DVC:

• Open source, free OSS.
• Any S3/GCS/Azure/SSH backend.
• Pipeline support (dvc.yaml).
• Best for: ML team with structured pipeline + cost-sensitive.

lakeFS:

• Git-like branch/merge for data lakes.
• S3-compatible API — applications transparent.
• Heavier infrastructure (manifest store, k8s deployment).
• Best for: data engineering team, large-scale data lake (TB+).

Bangladesh decision matrix:

• Single ML developer, GitHub-only → LFS.
• Small ML team (2-5), cost-sensitive, structured pipeline → DVC.
• Larger team (10+) with data engineers, complex data lake → lakeFS.
• Enterprise + budget — Databricks/Snowflake-এর built-in versioning, both DVC + lakeFS replace.

Common Bangladesh setup:

• ৮০% — DVC + S3 (Mumbai).
• ~১০% — Git LFS for simple repos.
• ~৫% — lakeFS at larger orgs.
• ~৫% — proprietary (Databricks).

Migration possibility:

• LFS → DVC: relatively easy. Pull files, dvc add.
• DVC → lakeFS: data move + path update; harder.
• Most teams stick with first choice years.

Practical advice:

• Start with DVC unless specific reason.
• S3 Mumbai bucket (low-latency from BD).
• Bandwidth cost: egress care; team in same region cheap.

মূল উপলব্ধি: Bangladesh ML team-এর majority — DVC sweet spot. Free, flexible, pipeline-aware. LFS too simple at scale, lakeFS too heavy for small team. DVC + MinIO (on-prem) compliance setup-এও extends.

Hashing performance — DVC-এর scaling concern.

Default behavior:

• MD5 hash per file. 50 GB single file → ~৩-৫ মিনিট hashing.
• Many small files vs one big file: many small slower (per-file overhead).
• Network attached storage (NFS) further slow.

Optimization techniques:

(১) Parallel hashing:

• dvc config core.hash_jobs 8 — multi-thread.
• I/O bound, so doesn't scale linearly with cores. ৪x typical.

(২) Different hash algorithm:

• DVC 3.0+: md5 default; sha256 available.
• xxhash faster (custom build).

(৩) Cache type:

• Default: copy file to cache — slow, large disk.
• Symlink: dvc config cache.type symlink — instant.
• Reflink (CoW): Btrfs/APFS — best of both. On-prem-এ reflink common nowadays.

(৪) Partial dataset versioning:

• Don't dvc-track raw 50 GB landmass。 Filter to working subset. Track that.
• Raw data: external versioning (S3 native). DVC tracks the prepared subset.

(৫) Directory-level vs file-level:

• dvc add on directory — DVC computes a tree hash + per-file hashes.
• Faster than per-file invocation.

Practical large dataset workflow:

• Initial dvc add: one-time slow (overnight if needed).
• Subsequent updates: only changed files re-hash.
• "Append-only" dataset patterns: monthly chunk add — faster.

Bandwidth concerns:

• dvc pull 50 GB on 10 Mbps link — ১২ ঘণ্টা। Painful.
• Solution: regional S3, faster local cache, partial pull.
• dvc pull data/subset/ — only that subdirectory.

"Don't version huge raw, version derived":

• Raw 500 GB photos — keep S3 versioned, not DVC.
• Preprocessed 5 GB embeddings — DVC track.
• Most ML iteration on derived; raw rarely changes.

Real BD example:

• One e-commerce team: 200 GB raw images (S3 versioned), 8 GB feature vectors (DVC), models in registry.
• Each layer right tool.

মূল উপলব্ধি: DVC theoretical limits beyond practical limits — but discipline matters. Don't dvc-add everything. Tier strategy: raw S3-native, derived DVC, models registry. Hashing slow only on first add; subsequent fast.

DVC pipelines (dvc.yaml) — like Make for ML. Setup overhead vs benefit.

Simple Python script enough when:

• One-shot training, no recurrent data prep.
• Solo developer, small project.
• No expensive intermediate steps to cache.

DVC pipeline worth when:

• Multi-stage workflow (download → preprocess → feature → train → eval).
• Some stages expensive (preprocessing 2 hours).
• Iterative experimentation — change one step, others should not rerun.
• Team — onboarding through pipeline definition.
• Reproducibility audit need.

Caching benefit example:

• Stage A (download): 30 min. Stage B (preprocess): 2 hours. Stage C (train): 30 min.
• Without DVC: change train.py → 3 hours total rerun.
• With DVC: A & B cached → 30 min only.
• 10 iterations: 30 hour saved.

DVC vs Airflow vs Make:

• Make: file-based dependency, simple. But no data versioning integration.
• DVC: file + data version aware, ML-specific. Local & CI.
• Airflow: production orchestration, schedule, distributed. Heavier.
• Common: DVC for local dev, Airflow for production scheduled runs.

Bangladesh context — ROI tipping point:

• Solo + simple → script.
• 3+ engineer or pipeline > 1 hour → DVC যথেষ্ট value.
• Production daily schedule → Airflow + DVC together.

Common pitfalls:

• Stages not idempotent → retry breaks.
• External dependencies not declared → cache invalidation wrong.
• Output overlap between stages → confusing.
• "Mega-stage" doing everything → no caching benefit.

Tip — start small:

• First just dvc add data, no pipeline.
• Later add 2-3 stage pipeline.
• Don't over-engineer initial setup.

মূল উপলব্ধি: DVC pipelines — multi-stage iterative ML-এ excellent. Solo simple script — overkill. Tipping point usually at: ৩+ stages, expensive intermediate, team আকার ২+.

Sensitive data versioning special care দাবি করে।

DVC-এর data flow risks:

• Local cache (default .dvc/cache) — file copy. Disk encrypt না হলে exposed.
• Remote storage — encryption depends on backend (S3 SSE, GCS default).
• Hash-as-key — hash itself doesn't leak data, but file contents in storage do.
• Git history — .dvc file with hash; if leak, attacker knows what to try if data accessible.

Anonymization-first approach:

• NID, mobile, email — hash before DVC track.
• "Pseudonymization at source" — original PII never enters versioned data.
• Bangladesh Bank guideline এর সাথে aligned.

Encryption layers:

• S3 SSE-KMS: server-side encryption with managed keys.
• S3 SSE-C: customer-provided keys.
• Client-side encryption: data encrypt before upload (sodium, AES).
• DVC currently no built-in encryption layer; rely on backend.

Access control:

• S3 bucket policy — least privilege. Specific IAM roles, no wildcard.
• VPC endpoint — bucket access only from internal network.
• MFA delete — accidental deletion prevention.

GDPR / Bangladesh-future regulation:

• "Right to be forgotten" — versioned data complicate. Customer delete request — all versions purge.
• Solution: customer ID-keyed storage, surgical delete possible.
• Or: anonymize before versioning — avoid the problem.

Alternative for highly sensitive:

• Don't DVC-version raw PII at all.
• Version statistics + hashes; access raw data on-demand from primary store.
• Audit log of data access.

Cache encryption (local):

• FileVault (Mac), LUKS (Linux), BitLocker (Windows) — full disk.
• OR DVC cache external encrypted volume.
• Team policy: laptop encryption mandatory.

Common Bangladesh setup for sensitive data:

• Anonymize (replace PII with hash).
• DVC + on-prem MinIO (data leaves not country).
• Encryption at rest (MinIO server-side).
• RBAC — bucket per project/team.

মূল উপলব্ধি: Sensitive data + DVC = anonymize first. Raw PII version-control এ rarely needed for ML. Hash-pseudonymized version DVC-friendly + privacy-friendly. On-prem MinIO + encrypted backend BD compliance নির্ভরযোগ্য।