Denormalization & Real-World Trade-offs

Denormalization — তাত্ত্বিক বনাম বাস্তব

Read: ~36 min Medium 12 practice problems Live SQLite runner

1. The Sin That Saves Production

Two modules ago we worked hard to reach 3NF. Now we are going to deliberately break it. Why? Because in production, the cost of a query is real money — slow pages lose customers, expensive queries melt servers, and the elegance of a perfectly normalized schema does not pay the AWS bill.

পূর্ববর্তী module-এ আমরা 3NF-এ পৌঁছাতে অনেক পরিশ্রম করেছি। এই module-এ আমরা ইচ্ছাকৃতভাবে সেই নিয়ম ভাঙব। কারণ বাস্তবে query-এর খরচ মানে আসল টাকা — ধীর পেজ গ্রাহক হারায়, ভারী query সার্ভার গরম করে, আর একটি নিখুঁত normalized schema-এর "সৌন্দর্য" দিয়ে cloud bill মেটানো যায় না।

Denormalization is the controlled, intentional addition of redundancy back into a normalized schema, in exchange for faster reads. The keyword is controlled. Random duplication is a bug. Measured duplication, with code that keeps it in sync, is engineering.

2. Read-Heavy vs Write-Heavy — Know Your System

Every workload sits somewhere on a spectrum between two extremes. Where you sit decides how aggressively you should denormalize.

প্রতিটি system দুইটি চরমের মাঝে কোথাও থাকে — read-heavy বা write-heavy। আপনার system যেদিকে বেশি, সেদিক অনুযায়ী denormalization-এর পরিমাণ ঠিক হয়।
WorkloadRead : write ratioExamplesDenormalize?
OLTP write-heavyroughly 1 : 1bKash transactions, ride logsAlmost never. Normalize.
OLTP read-heavy50 : 1 or higherDaraz product page, news feedSelectively, on hot paths.
OLAP / analytics1000 : 1 or higherSales dashboards, BI reportsYes — star/snowflake schemas.
Cache layer10000 : 1Profile cards, leaderboardsYes — fully denormalized rows.
Iron law. Measure first, denormalize second. Every denormalization adds a consistency obligation; you only earn that cost back if a real, measured query was slow.

আগে measure করুন, তারপর denormalize করুন। প্রতিটি denormalization একটি consistency obligation যোগ করে; বাস্তব slow query না থাকলে সেই খরচ লাভে পরিণত হবে না।

3. Cached Aggregate Columns — A 100× Win

Consider a Daraz-style products page that shows the average rating of every product. The pure-3NF way computes it on demand:

slow-aggregate.sql
-- Pure 3NF query: scans every review, every time.
SELECT p.name,
       COUNT(r.review_id) AS reviews,
       ROUND(AVG(r.stars), 2) AS avg_stars
FROM products p
LEFT JOIN reviews r ON r.product_id = p.product_id
GROUP BY p.product_id, p.name;

With 200 reviews this is fast; with 200 million it is not. The denormalized solution: keep review_count and avg_stars right on products, and use triggers to keep them in sync.

cached-aggregate.sql
CREATE TABLE products (
    product_id   INTEGER PRIMARY KEY,
    name         TEXT,
    review_count INTEGER DEFAULT 0,
    stars_sum    INTEGER DEFAULT 0
);
CREATE TABLE reviews (
    review_id  INTEGER PRIMARY KEY,
    product_id INTEGER REFERENCES products(product_id),
    stars      INTEGER
);

CREATE TRIGGER reviews_after_insert
AFTER INSERT ON reviews
BEGIN
    UPDATE products
       SET review_count = review_count + 1,
           stars_sum    = stars_sum    + NEW.stars
     WHERE product_id = NEW.product_id;
END;

INSERT INTO products(product_id, name) VALUES (1, 'Phone');
INSERT INTO reviews(product_id, stars) VALUES (1, 5), (1, 4), (1, 5);

-- The product-page query is now a single-row read, no JOIN, no aggregate:
SELECT name,
       review_count,
       ROUND(CAST(stars_sum AS REAL) / review_count, 2) AS avg_stars
FROM products WHERE product_id = 1;
যখন প্রতি product page load-এ লক্ষ লক্ষ review scan হবে, তখন cached column রাখাই বুদ্ধিমানের কাজ। trigger ব্যবহার করে এই column-গুলো সর্বদা সিঙ্ক রাখা যায়। ফলে product page query ১০০ গুণেরও বেশি দ্রুত হতে পারে।
Don't forget DELETE and UPDATE. A cached aggregate must have triggers (or equivalent application logic) for every operation that can change the underlying data — INSERT, DELETE and UPDATE. Forgetting one is how stale aggregates appear.

cached aggregate-এর জন্য INSERT, DELETE এবং UPDATE — তিনটি ক্ষেত্রেই trigger লাগবে। একটি বাদ গেলে stale data শুরু হয়।

4. Materialized Views — Pre-Computed Reports

A materialized view is a query whose result is stored as a table. PostgreSQL, Oracle and SQL Server support them natively (CREATE MATERIALIZED VIEW … REFRESH). SQLite does not have the keyword, but the same effect is trivially achievable with a regular table plus triggers — exactly as we just did.

Materialized view হলো এমন একটি query যার result একটি real table-এ সংরক্ষিত থাকে। PostgreSQL-এ এটি built-in (CREATE MATERIALIZED VIEW), কিন্তু SQLite-এ regular table এবং trigger ব্যবহার করেই একই কাজ করা যায়।

Picture a daily revenue dashboard for a Pathao courier business. The truth lives in millions of orders rows; the dashboard wants 365 numbers, one per day.

materialized-view.sql
CREATE TABLE orders (
    order_id     INTEGER PRIMARY KEY,
    order_date   TEXT,
    total_amount INTEGER
);
INSERT INTO orders(order_date, total_amount) VALUES
 ('2026-05-08', 1200), ('2026-05-08', 700),
 ('2026-05-09', 450),  ('2026-05-09', 2100),
 ('2026-05-10', 800);

-- The "view": a regular table that stores the daily totals.
CREATE TABLE daily_revenue (
    order_date TEXT PRIMARY KEY,
    orders     INTEGER,
    revenue    INTEGER
);

-- Refresh — for a single-writer system this is fine to run nightly.
DELETE FROM daily_revenue;
INSERT INTO daily_revenue(order_date, orders, revenue)
SELECT order_date, COUNT(*), SUM(total_amount)
FROM orders
GROUP BY order_date;

SELECT * FROM daily_revenue ORDER BY order_date;
Refresh strategyWhen to useCost
On-demand (manual)Nightly batch reports.Stale during the day.
Scheduled (cron / pg_cron)Hourly dashboards.Up to one hour stale.
Trigger-based (incremental)Hot product cards, leaderboards.Higher write cost.
Streaming pipelineReal-time analytics, fraud detection.External infrastructure.

5. Star and Snowflake Schemas — OLAP Design

Online Analytical Processing (OLAP) systems live on the fully denormalized side. A star schema has one big fact table (one row per event, e.g. one row per sale) surrounded by small dimension tables (product, customer, store, time). Joins are at most one hop deep, and queries scan rows by date almost exclusively.

OLAP system সম্পূর্ণ denormalized দিকে থাকে। Star schema-তে একটি বড় fact টেবিল (যেমন প্রতি বিক্রয়ে একটি row) চারপাশে কয়েকটি ছোট dimension টেবিল (product, customer, store, time)। প্রতিটি query এক-ধাপ join-এ শেষ হয়।
fact_sales date_key, product_key, customer_key, qty, amount dim_date y/m/d, weekday dim_product name, category, brand dim_customer dim_store Figure 28.1 — star schema: একটি কেন্দ্রীয় fact টেবিল, চারদিকে ছোট ছোট dimension টেবিল।

A snowflake schema further normalizes the dimensions (e.g., product → category → department). It saves space at the cost of extra joins. Most modern OLAP engines prefer the flat star — disks are cheap, CPU joins are not.

star-schema.sql
-- A toy star schema for a Bangladeshi retail dashboard.
CREATE TABLE dim_date     (date_key TEXT PRIMARY KEY, year INTEGER, month INTEGER);
CREATE TABLE dim_product  (product_key INTEGER PRIMARY KEY, name TEXT, category TEXT);
CREATE TABLE dim_store    (store_key INTEGER PRIMARY KEY, city TEXT);
CREATE TABLE fact_sales (
    date_key    TEXT REFERENCES dim_date(date_key),
    product_key INTEGER REFERENCES dim_product(product_key),
    store_key   INTEGER REFERENCES dim_store(store_key),
    qty         INTEGER,
    amount      INTEGER
);

INSERT INTO dim_date    VALUES ('2026-05-10', 2026, 5);
INSERT INTO dim_product VALUES (1, 'Phone', 'Electronics'), (2, 'Book', 'Books');
INSERT INTO dim_store   VALUES (1, 'Dhaka'), (2, 'Chittagong');
INSERT INTO fact_sales VALUES
 ('2026-05-10', 1, 1, 2, 50000),
 ('2026-05-10', 2, 1, 5, 2250),
 ('2026-05-10', 1, 2, 1, 25000);

-- Single-hop analytic query: revenue per category per city.
SELECT p.category, s.city, SUM(f.amount) AS revenue
FROM fact_sales f
JOIN dim_product p ON p.product_key = f.product_key
JOIN dim_store   s ON s.store_key   = f.store_key
GROUP BY p.category, s.city;

6. OLTP vs OLAP — Two Worlds, One Company

🟢 OLTP (operational)

  • Many small writes per second.
  • Row-oriented storage.
  • Highly normalized (3NF / BCNF).
  • Examples: bKash, Daraz checkout, login.
  • Tools: PostgreSQL, MySQL, SQL Server.

🟡 OLAP (analytical)

  • Few queries, each scanning huge ranges.
  • Column-oriented storage.
  • Heavily denormalized (star schema).
  • Examples: monthly revenue, churn analysis.
  • Tools: BigQuery, Snowflake, ClickHouse, DuckDB.
OLTP = প্রতিদিনের ছোট ছোট transaction; এখানে normalize করা চাই। OLAP = বিশাল ডেটার ওপর বিশ্লেষণ; এখানে denormalize করা চাই। বাস্তব কোম্পানিতে দুটোই দরকার — সাধারণত দুটি আলাদা database, ETL pipeline দিয়ে যুক্ত।
The pattern most teams adopt. OLTP database (PostgreSQL) for live traffic, an ETL job that nightly copies and reshapes data into an OLAP warehouse (BigQuery / Snowflake). One source of truth, two storage shapes for two access patterns.

অধিকাংশ কোম্পানি OLTP-এর জন্য PostgreSQL ব্যবহার করে এবং nightly ETL দিয়ে data কপি করে BigQuery বা Snowflake-এ দেয়। এক truth, দুই shape।

7. Rules of Safe Denormalization

  1. Always start normalized. Denormalize only when a measured query is too slow.
  2. Document the duplication. Every cached column needs a comment naming the source-of-truth table and the maintenance trigger.
  3. Cover INSERT, UPDATE, DELETE. If any of the three is missing a maintenance hook, your aggregate will drift.
  4. Add a recompute job. Even with perfect triggers, run a nightly full recompute to detect drift early.
  5. Treat the source as truth. When the cache disagrees with the base tables, the base tables win — never the other way around.
  6. Re-test after every schema change. Adding a new column to reviews must trigger a review of the cached aggregate.
নিরাপদ denormalization-এর ৬টি নিয়ম — প্রথমে normalize, তারপর measure-ভিত্তিক denormalize, প্রতিটি duplicate column-এর source এবং maintenance trigger কমেন্টে লিখুন, INSERT/UPDATE/DELETE — তিনটির জন্যই hook রাখুন, nightly recompute চালান, cache vs source-এ source-ই সর্বদা সঠিক, এবং schema পরিবর্তনের পর cache-এর বৈধতা যাচাই করুন।

8. Glossary (শব্দকোষ)

TermMeaningবাংলায়
DenormalizationIntentional redundancy for read performance.read দ্রুত করার জন্য ইচ্ছাকৃত পুনরাবৃত্তি।
Materialized viewA query result stored as a table.একটি query-এর ফলাফল table-এ সংরক্ষিত।
Cached aggregateA column holding a running SUM/COUNT/AVG.SUM/COUNT/AVG-এর pre-computed কলাম।
OLTPOnline transaction processing — daily writes.দৈনন্দিন write-নির্ভর system।
OLAPOnline analytical processing — large reads.বড় read-নির্ভর analytics system।
Star schemaCentral fact table joined to flat dimension tables.একটি কেন্দ্রীয় fact + dimension টেবিল।
Snowflake schemaStar schema with normalized dimensions.dimension গুলো নিজেরাও normalized।

9. Practice Problems

Twelve problems on real denormalization decisions. Most have a runnable SQLite answer.

বাস্তব denormalization-এর ১২টি প্রশ্ন। বেশিরভাগেরই চালু-করার-যোগ্য উত্তর আছে।
  1. Add a trigger that decrements review_count and stars_sum when a review is deleted.
    review delete হলে review_count ও stars_sum কমানোর জন্য trigger লিখুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans1.sql
    CREATE TRIGGER reviews_after_delete
    AFTER DELETE ON reviews
    BEGIN
        UPDATE products
           SET review_count = review_count - 1,
               stars_sum    = stars_sum    - OLD.stars
         WHERE product_id = OLD.product_id;
    END;
    INSERT INTO reviews(review_id, product_id, stars) VALUES (10, 1, 4);
    DELETE FROM reviews WHERE review_id = 10;
    SELECT * FROM products;
  2. Write an UPDATE trigger so editing a review's star value keeps the cached sum correct.
    review edit করার জন্য UPDATE trigger লিখুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans2.sql
    CREATE TRIGGER reviews_after_update
    AFTER UPDATE OF stars ON reviews
    BEGIN
        UPDATE products
           SET stars_sum = stars_sum + NEW.stars - OLD.stars
         WHERE product_id = NEW.product_id;
    END;
    UPDATE reviews SET stars = 3 WHERE review_id = 1;
    SELECT * FROM products;
  3. A reconciliation query — write SQL that finds products whose cached review_count disagrees with the actual count of rows in reviews.
    cached count এবং actual count মিলছে না — এমন product বের করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans3.sql
    SELECT p.product_id, p.review_count AS cached, COUNT(r.product_id) AS actual
    FROM products p
    LEFT JOIN reviews r ON r.product_id = p.product_id
    GROUP BY p.product_id, p.review_count
    HAVING p.review_count <> COUNT(r.product_id);
  4. Suggest one place where storing customer_full_name on every orders row is correct, not a bug.
    প্রতিটি orders row-এ customer_full_name রাখা কখন সঠিক?
    ✨ Show Answer (উত্তর দেখুন)

    Answer: When you need a frozen historical snapshot. If the customer later changes their name, the invoice should still reflect the name at the time of purchase. This is not denormalization — it is treating the value as a different fact (the name-on-the-day-of-sale) which simply happens to coincide with the live name today.

  5. Write a single SQL statement that fully recomputes review_count and stars_sum from scratch, as a nightly drift check.
    nightly recompute-এর জন্য SQL লিখুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans5.sql
    UPDATE products
       SET review_count = (SELECT COUNT(*) FROM reviews r WHERE r.product_id = products.product_id),
           stars_sum    = COALESCE((SELECT SUM(stars) FROM reviews r WHERE r.product_id = products.product_id), 0);
    SELECT * FROM products;
  6. List two queries on a Daraz-style site that should NOT be denormalized, and explain why.
    কোন দুটি query denormalize করা উচিত নয় — কেন?
    ✨ Show Answer (উত্তর দেখুন)

    Answer: (1) Customer billing address — must always be the latest, not a cached copy that drifts. (2) Real-time stock count — denormalizing into a cached column risks selling phantom inventory under concurrency. Both want strict consistency more than speed.

  7. Convert a normalized order_items(order_id, product_id, qty) + products(product_id, price) into a fact table for nightly reporting.
    উপরের টেবিলগুলো থেকে fact_order_items তৈরি করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans7.sql
    CREATE TABLE order_items (order_id INTEGER, product_id INTEGER, qty INTEGER);
    CREATE TABLE products    (product_id INTEGER PRIMARY KEY, name TEXT, price INTEGER);
    INSERT INTO products    VALUES (1,'Phone',25000), (2,'Book',450);
    INSERT INTO order_items VALUES (100,1,2), (100,2,3);
    
    CREATE TABLE fact_order_items AS
    SELECT oi.order_id, oi.product_id, p.name, oi.qty, p.price, oi.qty * p.price AS line_total
    FROM order_items oi JOIN products p ON p.product_id = oi.product_id;
    SELECT * FROM fact_order_items;
  8. Snowflake the dim_product dimension by extracting category into dim_category. Show the new schema.
    dim_product-কে snowflake করে dim_category বের করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans8.sql
    CREATE TABLE dim_category (category_key INTEGER PRIMARY KEY, category_name TEXT);
    CREATE TABLE dim_product  (
        product_key  INTEGER PRIMARY KEY,
        name         TEXT,
        category_key INTEGER REFERENCES dim_category(category_key)
    );
    INSERT INTO dim_category VALUES (1,'Electronics'),(2,'Books');
    INSERT INTO dim_product  VALUES (10,'Phone',1),(11,'Book',2);
    SELECT p.name, c.category_name
    FROM dim_product p JOIN dim_category c ON c.category_key = p.category_key;
  9. A junior engineer suggests replacing every JOIN in the OLTP database with denormalized columns. Give two reasons this is dangerous.
    প্রতিটি JOIN-কে denormalized column দিয়ে replace করা কেন বিপজ্জনক — দুটি কারণ দিন।
    ✨ Show Answer (উত্তর দেখুন)

    Answer: (1) Every write now updates many rows; INSERT/UPDATE latency explodes. (2) Drift becomes inevitable — without disciplined triggers and reconciliation, the cached copies diverge silently from the source of truth, producing wrong financial reports and customer-visible bugs.

  10. Identify which tables in a bKash transaction system you would NOT denormalize and why.
    bKash-এ কোন কোন টেবিল কখনো denormalize করবেন না — কেন?
    ✨ Show Answer (উত্তর দেখুন)

    Answer: The tx (transaction) and account_balance tables. Both demand strict, second-by-second correctness. A denormalized column drifting by even Tk 0.01 is an audit failure. Reads here are also already cheap — accounts are looked up by primary key.

  11. Write a query that uses the daily_revenue materialized table to find the top 3 revenue days.
    daily_revenue থেকে top 3 দিন বের করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans11.sql
    SELECT order_date, revenue
    FROM daily_revenue
    ORDER BY revenue DESC
    LIMIT 3;
  12. Describe a "100× faster" denormalization story you can defend in a code review.
    code review-তে defend করার মতো একটি "100× faster" denormalization গল্প বর্ণনা করুন।
    ✨ Show Answer (উত্তর দেখুন)

    Answer: "Our product page query joined products, reviews and order_items with two GROUP BYs. p99 latency was 480 ms. We added a products.review_count and products.stars_sum column, maintained by three triggers covering insert/update/delete, plus a nightly reconciliation script. p99 dropped to 4 ms — a 120× improvement on a page that loads 3 million times per day. Reconciliation has flagged drift exactly twice in 18 months, both caught and fixed within an hour."

    একটি বাস্তব গল্প — measure, change, measure again, এবং drift ধরার জন্য nightly reconciliation। এটিই defendable engineering।

Summary — Module 28

Normalization is a starting point, not the destination. Once you measure a real performance problem, controlled denormalization — cached aggregates, materialized views, star schemas — can deliver order-of-magnitude wins. The price is consistency obligations: triggers, reconciliation jobs and discipline. OLTP systems lean normalized; OLAP warehouses lean denormalized; most real companies run both, connected by ETL.

Normalization হলো শুরু, শেষ নয়। বাস্তব performance সমস্যা measure করার পর নিয়ন্ত্রিত denormalization — cached aggregate, materialized view, star schema — অনেক বড় গতি দিতে পারে। তবে এর মূল্য consistency-র দায়িত্ব। OLTP-তে normalize, OLAP-তে denormalize — অধিকাংশ কোম্পানি দুটিই ব্যবহার করে।

Next Module → Logical থেকে Physical schema — design কে আসল database-এ রূপ দেওয়া।