dbt — analytics engineering

dbt স্পষ্টভাবেই ELT-এর tool. বুঝতে হলে আগে তিনটি step-এর order বুঝে নিন।

ETL (পুরাতন paradigm, ১৯৯০-২০১০):

• Extract: source থেকে data বের করুন।
• Transform: আলাদা compute server (Informatica, Talend)-এ transform.
• Load: warehouse-এ load.
• Warehouse expensive ছিল — minimal data রাখা।

ELT (modern, ২০১৫+):

• Extract: source থেকে।
• Load: raw data সরাসরি warehouse-এ।
• Transform: warehouse-এর ভেতরেই SQL দিয়ে।
• Cloud warehouse (Snowflake, BigQuery) cheap ও fast — raw রাখা সম্ভব।

কেন ELT জিতছে:

• Cloud economics: Snowflake/BigQuery storage cents/GB. compute decoupled.
• Schema-on-read: raw রাখলে — পরে নতুন transformation possible original থেকে।
• SQL ubiquity: আরও বেশি লোক SQL জানে — Python/Spark কম।
• Vendor offload: Fivetran, Airbyte EL handle; dbt T.

dbt-এর ভূমিকা:

• "T" layer-এর tool.
• Raw warehouse-এ থাকা মানে — dbt-এর কাজের জায়গা warehouse.
• Spark/EMR-এর জটিল compute layer extra.

Bangladesh-এর reality:

• Cost concern: Snowflake credit-এ ১০ জন অ্যানালিস্টের warehouse — মাসে $২,০০০-৫,০০০। ছোট startup-এর জন্য বেশি।
• Alternative: ClickHouse self-hosted, BigQuery free tier ($৩০০ credit), Postgres warehouse pattern.
• EL tool: Fivetran expensive. Airbyte open-source — popular bd-তে। Custom Python script-ও common.
• Bandwidth: বড় data extract slow connection-এ — incremental EL critical.
• Skill gap: dbt শিখতে শুধু SQL + Git. এটি বড় advantage.

Practical scenarios:

• BRAC Bank reporting: ELT excellent — সব transaction Snowflake/Synapse-এ load, dbt-এ analytics.
• bKash fraud detection: ELT কাজে আসে না — sub-second latency দরকার, streaming ETL.
• Daraz BI dashboard: ELT perfect — overnight refresh যথেষ্ট।
• BTRC compliance reports: ELT — historical data যেমন ছিল accurately preserve.

Hybrid pattern:

• "ETLT" — light transform (PII masking, format) load-এর সময়; deep transform warehouse-এ।
• Operational systems-এ heavy compute না করতে।

মূল কথা: Cloud warehouse + dbt = analytics future. Bangladesh-এর enterprise এই path-এ আসছে। কিন্তু streaming ও ML inference আলাদা stack — ELT সবকিছুর সমাধান নয়।

Materialization decision = cost vs latency vs complexity trade-off. প্রতিটি model-এ আলাদা।

View — কখন:

• Source table ছোট (১ GB-র নিচে)।
• Query infrequent (প্রতিদিন ১-১০ বার)।
• Underlying data fresh চাই।
• Storage cost-sensitive.

View — সমস্যা ১০০ GB-তে:

• প্রতি query-তে ১০০ GB scan — Snowflake-এ ~$৫।
• BI tool ২০ user × ৫ refresh × $৫ = $৫০০/দিন!
• Slow — query-time-এ aggregation চলে।

Table — কখন:

• Read-heavy (BI dashboard, API).
• Source ছোট-মাঝারি (১০ GB-র নিচে rebuild affordable).
• Result-এ heavy aggregation/join — pre-compute.

Table — সমস্যা ১০০ GB-তে:

• প্রতি dbt run-এ ১০০ GB rebuild — ৩০ মিনিট, $৫০।
• প্রতিদিন rebuild = $১,৫০০/মাস শুধু এই model-এ।
• Run window দীর্ঘ — downstream wait.

Incremental — ১০০ GB-তে আদর্শ:

• প্রতি run-এ শুধু গতকালের ~৫০ MB process.
• Run time ৩০ সেকেন্ড, cost cents.
• Read fast (table-এর মতো)।
• Trade-off: complexity বেশি, late-arriving data risk.

Decision tree (১০০ GB transaction table):

• "Query কতবার?" — দিনে ১০+: not view.
• "Source কত বড়?" — ১০ GB-র বেশি: not pure table.
• "New row append-only?" — হ্যাঁ: incremental.
• "Update history?" — হ্যাঁ: snapshot বা full refresh weekly.

Recommended pattern (bKash transactions):

# staging: view (cheap, transparent)
stg_transactions: {{ config(materialized='view') }}

# fact: incremental (huge, append-mostly)
fct_transactions: {{ config(
  materialized='incremental',
  unique_key='transaction_id',
  partition_by={'field': 'date', 'data_type': 'date'},
  cluster_by=['user_id']
) }}

# summary: table (small, queried often)
daily_revenue_summary: {{ config(materialized='table') }}

ভুল choice consequences:

• View on huge: warehouse bill explosion. Query timeout. Dashboard load slow.
• Table on huge: long run time, schedule miss. Late refresh = stale dashboards.
• Incremental wrong key: duplicate rows, missing data.
• Table on small read-once: wasted storage, complexity.

Late-arriving data trap:

• Incremental-এ WHERE date >= max(date) = older arriving row miss.
• Solution: lookback window (max - 3 days) + delete-insert.
• Or weekly full_refresh: dbt run --full-refresh -s fct_transactions.

মূল কথা: Materialization = engineering decision, business decision নয়। Volume measure, query pattern monitor, ব্যয় track — তারপর choose.

এটি real production scenario. dbt-এর lineage capability এখানে ভাল কাজে।

Affected detection:

# downstream impact
dbt list --select +my_model+
# (+ before = upstream, + after = downstream)

# শুধু changed model + downstream
dbt build --select state:modified+ \
  --state ./prod-manifest

# Slim CI (PR-এ শুধু changed পরীক্ষা)
dbt build --select state:modified+ \
  --defer --state ./prod-manifest

State comparison:

• state:modified — current vs main branch-এ যা ভিন্ন।
• state:new — main-এ নেই।
• "manifest.json" আগের build-এর — compare-এর basis.

CI/CD pipeline (GitHub Actions / GitLab):

name: dbt CI
on: pull_request

jobs:
  ci:
    steps:
      - uses: actions/checkout@v3
      - run: dbt deps
      - run: dbt seed --target ci
      - run: dbt build --select state:modified+ \
                --defer --state ./prod-manifest \
                --target ci
      - run: dbt source freshness

Safeguard layers:

• Layer 1 — Static: SQL linter (sqlfluff), naming convention check.
• Layer 2 — Compile: dbt parse — Jinja syntax error catch.
• Layer 3 — Build: changed model + downstream actually build in CI schema.
• Layer 4 — Test: not_null, unique, custom assertion.
• Layer 5 — Compare: dbt-audit-helper দিয়ে old vs new diff.
• Layer 6 — Approval: senior reviewer approve.
• Layer 7 — Deploy: merge → production run.

Defer pattern:

• CI-তে শুধু changed model build.
• Upstream reference — production manifest থেকে।
• ৫০০ model-এর pretty কম resource.
• 10x+ faster CI.

Documentation enforcement:

• dbt-checkpoint hook: undocumented column reject.
• required_docs: true — model description ছাড়া fail.

Test coverage:

• Primary key column-এ unique + not_null mandatory.
• Foreign key column-এ relationships test.
• Business rule (revenue > 0, status in valid list) custom test.

Production deployment:

• Blue-green: parallel schema build, atomic rename.
• Or zero-copy clone (Snowflake): CREATE TABLE x_new CLONE x.
• Rollback plan: previous run-এর schema retain (১ দিন)।

Junior onboarding patterns:

• Sandbox schema প্রতি dev-এর: dbt_alice, dbt_bob.
• Pull-request template — checklist (test added? doc updated? lineage checked?)।
• Pair programming — first ৩-৫ PR senior-এর সাথে।

Bangladesh enterprise context:

• Audit log requirement (BB compliance) — সব schema change tracked.
• Data lineage — dbt docs generated diagram regulator-কে দেখানো।
• PII handling — dbt-snowflake-এ row access policy + dbt macro.

মূল কথা: dbt + Git + CI/CD = software engineering rigor for analytics. ৫০০ model managing-এর জন্য এই disciplines essential — না হলে production pipeline বছরের পর বছর fragile.

Macro = Jinja template যা compile-time-এ SQL generate করে। C-এর preprocessor-এর মতো।

কখন macro proper:

• DRY principle: একই SQL snippet ৩+ model-এ repeat.
• Database-specific abstraction: Snowflake vs BigQuery syntax difference hide.
• Dynamic generation: runtime input (variable) থেকে SQL.
• Custom test: built-in test যথেষ্ট না।
• Project conventions: schema naming, table aliasing standard.

কখন macro নয়:

• একবার ব্যবহৃত SQL — over-engineering.
• Business logic — model-এ থাকা ভাল (visible).
• Data-dependent logic — macro static, runtime data নয়।

Bangladesh fintech macro 1: BDT formatting

{% macro format_bdt(amount_col) %}
  case
    when {{ amount_col }} >= 10000000 then
      concat(round({{ amount_col }} / 10000000.0, 2), ' কোটি')
    when {{ amount_col }} >= 100000 then
      concat(round({{ amount_col }} / 100000.0, 2), ' লাখ')
    when {{ amount_col }} >= 1000 then
      concat(round({{ amount_col }} / 1000.0, 2), ' হাজার')
    else cast({{ amount_col }} as varchar)
  end
{% endmacro %}

-- Usage:
select {{ format_bdt('total_revenue') }} as revenue_display
from fct_revenue

Bangladesh fintech macro 2: NID validation

{% macro is_valid_nid(nid_col) %}
  -- Bangladesh NID: 10, 13, or 17 digits
  ({{ nid_col }} ~ '^[0-9]{10}$' or
   {{ nid_col }} ~ '^[0-9]{13}$' or
   {{ nid_col }} ~ '^[0-9]{17}$')
{% endmacro %}

-- Used in test:
{% test valid_nid(model, column_name) %}
  select * from {{ model }}
  where not {{ is_valid_nid(column_name) }}
    and {{ column_name }} is not null
{% endtest %}

Bangladesh fintech macro 3: PII masking

{% macro mask_phone(phone_col) %}
  case
    when current_role() in ('analyst', 'admin') then {{ phone_col }}
    else concat(left({{ phone_col }}, 5), 'XXXXXX')
  end
{% endmacro %}

-- Bangladesh phone: 01XXXXXXXXX (11 digits)
-- Show: 01700XXXXXX

Bangladesh fintech macro 4: Fiscal year

{% macro fiscal_year(date_col) %}
  case
    when extract(month from {{ date_col }}) >= 7
    then concat(extract(year from {{ date_col }}), '-',
                extract(year from {{ date_col }}) + 1)
    else concat(extract(year from {{ date_col }}) - 1, '-',
                extract(year from {{ date_col }}))
  end
{% endmacro %}

-- Bangladesh fiscal year: July-June
-- 2026-05 → "2025-2026"
-- 2026-08 → "2026-2027"

Bangladesh fintech macro 5: Mobile operator detection

{% macro mobile_operator(phone_col) %}
  case
    when {{ phone_col }} like '0170%' or {{ phone_col }} like '0171%' or
         {{ phone_col }} like '0172%' or {{ phone_col }} like '0173%' or
         {{ phone_col }} like '0174%' or {{ phone_col }} like '0175%' or
         {{ phone_col }} like '0176%' or {{ phone_col }} like '0177%' or
         {{ phone_col }} like '0178%' or {{ phone_col }} like '0179%' then 'Grameenphone'
    when {{ phone_col }} like '0181%' or {{ phone_col }} like '0182%' or
         {{ phone_col }} like '0183%' or {{ phone_col }} like '0184%' or
         {{ phone_col }} like '0185%' or {{ phone_col }} like '0186%' or
         {{ phone_col }} like '0187%' or {{ phone_col }} like '0188%' or
         {{ phone_col }} like '0189%' then 'Robi'
    when {{ phone_col }} like '019%' then 'Banglalink'
    when {{ phone_col }} like '015%' then 'Teletalk'
    when {{ phone_col }} like '013%' then 'Grameenphone'  -- newer
    else 'Unknown'
  end
{% endmacro %}

Best practices:

• Macro file macros/ ফোল্ডারে।
• Naming: verb-based (format_bdt, is_valid_nid).
• Document with docstring.
• Test macro itself (yes, possible).
• Package candidate: reusable macros → dbt package publish.

Public packages worth knowing:

• dbt-utils: generate_series, surrogate_key, pivot — daily use.
• dbt-expectations: Great Expectations port — advanced data tests.
• dbt-codegen: auto-generate model SQL from source schema.

মূল কথা: Macro = power tool — sparingly use. Business logic visible-এ থাকুক, plumbing macro-এ। Bangladesh-এর domain-specific macro অসাধারণ contribution হতে পারে।