Denormalization & Real-World Trade-offs
Denormalization — তাত্ত্বিক বনাম বাস্তব
1. The Sin That Saves Production
Two modules ago we worked hard to reach 3NF. Now we are going to deliberately break it. Why? Because in production, the cost of a query is real money — slow pages lose customers, expensive queries melt servers, and the elegance of a perfectly normalized schema does not pay the AWS bill.
Denormalization is the controlled, intentional addition of redundancy back into a normalized schema, in exchange for faster reads. The keyword is controlled. Random duplication is a bug. Measured duplication, with code that keeps it in sync, is engineering.
2. Read-Heavy vs Write-Heavy — Know Your System
Every workload sits somewhere on a spectrum between two extremes. Where you sit decides how aggressively you should denormalize.
| Workload | Read : write ratio | Examples | Denormalize? |
|---|---|---|---|
| OLTP write-heavy | roughly 1 : 1 | bKash transactions, ride logs | Almost never. Normalize. |
| OLTP read-heavy | 50 : 1 or higher | Daraz product page, news feed | Selectively, on hot paths. |
| OLAP / analytics | 1000 : 1 or higher | Sales dashboards, BI reports | Yes — star/snowflake schemas. |
| Cache layer | 10000 : 1 | Profile cards, leaderboards | Yes — fully denormalized rows. |
আগে measure করুন, তারপর denormalize করুন। প্রতিটি denormalization একটি consistency obligation যোগ করে; বাস্তব slow query না থাকলে সেই খরচ লাভে পরিণত হবে না।
3. Cached Aggregate Columns — A 100× Win
Consider a Daraz-style products page that shows the average rating of every product. The pure-3NF way computes it on demand:
-- Pure 3NF query: scans every review, every time.
SELECT p.name,
COUNT(r.review_id) AS reviews,
ROUND(AVG(r.stars), 2) AS avg_stars
FROM products p
LEFT JOIN reviews r ON r.product_id = p.product_id
GROUP BY p.product_id, p.name;
With 200 reviews this is fast; with 200 million it is not. The denormalized solution: keep
review_count and avg_stars right on products, and use triggers
to keep them in sync.
CREATE TABLE products (
product_id INTEGER PRIMARY KEY,
name TEXT,
review_count INTEGER DEFAULT 0,
stars_sum INTEGER DEFAULT 0
);
CREATE TABLE reviews (
review_id INTEGER PRIMARY KEY,
product_id INTEGER REFERENCES products(product_id),
stars INTEGER
);
CREATE TRIGGER reviews_after_insert
AFTER INSERT ON reviews
BEGIN
UPDATE products
SET review_count = review_count + 1,
stars_sum = stars_sum + NEW.stars
WHERE product_id = NEW.product_id;
END;
INSERT INTO products(product_id, name) VALUES (1, 'Phone');
INSERT INTO reviews(product_id, stars) VALUES (1, 5), (1, 4), (1, 5);
-- The product-page query is now a single-row read, no JOIN, no aggregate:
SELECT name,
review_count,
ROUND(CAST(stars_sum AS REAL) / review_count, 2) AS avg_stars
FROM products WHERE product_id = 1;
cached aggregate-এর জন্য INSERT, DELETE এবং UPDATE — তিনটি ক্ষেত্রেই trigger লাগবে। একটি বাদ গেলে stale data শুরু হয়।
4. Materialized Views — Pre-Computed Reports
A materialized view is a query whose result is stored as a table. PostgreSQL,
Oracle and SQL Server support them natively (CREATE MATERIALIZED VIEW … REFRESH).
SQLite does not have the keyword, but the same effect is trivially achievable with a regular
table plus triggers — exactly as we just did.
CREATE MATERIALIZED VIEW), কিন্তু SQLite-এ regular table এবং trigger ব্যবহার করেই একই কাজ করা যায়।
Picture a daily revenue dashboard for a Pathao courier business. The truth lives in millions of
orders rows; the dashboard wants 365 numbers, one per day.
CREATE TABLE orders (
order_id INTEGER PRIMARY KEY,
order_date TEXT,
total_amount INTEGER
);
INSERT INTO orders(order_date, total_amount) VALUES
('2026-05-08', 1200), ('2026-05-08', 700),
('2026-05-09', 450), ('2026-05-09', 2100),
('2026-05-10', 800);
-- The "view": a regular table that stores the daily totals.
CREATE TABLE daily_revenue (
order_date TEXT PRIMARY KEY,
orders INTEGER,
revenue INTEGER
);
-- Refresh — for a single-writer system this is fine to run nightly.
DELETE FROM daily_revenue;
INSERT INTO daily_revenue(order_date, orders, revenue)
SELECT order_date, COUNT(*), SUM(total_amount)
FROM orders
GROUP BY order_date;
SELECT * FROM daily_revenue ORDER BY order_date;
| Refresh strategy | When to use | Cost |
|---|---|---|
| On-demand (manual) | Nightly batch reports. | Stale during the day. |
| Scheduled (cron / pg_cron) | Hourly dashboards. | Up to one hour stale. |
| Trigger-based (incremental) | Hot product cards, leaderboards. | Higher write cost. |
| Streaming pipeline | Real-time analytics, fraud detection. | External infrastructure. |
5. Star and Snowflake Schemas — OLAP Design
Online Analytical Processing (OLAP) systems live on the fully denormalized side. A star schema has one big fact table (one row per event, e.g. one row per sale) surrounded by small dimension tables (product, customer, store, time). Joins are at most one hop deep, and queries scan rows by date almost exclusively.
A snowflake schema further normalizes the dimensions (e.g., product → category → department). It saves space at the cost of extra joins. Most modern OLAP engines prefer the flat star — disks are cheap, CPU joins are not.
-- A toy star schema for a Bangladeshi retail dashboard.
CREATE TABLE dim_date (date_key TEXT PRIMARY KEY, year INTEGER, month INTEGER);
CREATE TABLE dim_product (product_key INTEGER PRIMARY KEY, name TEXT, category TEXT);
CREATE TABLE dim_store (store_key INTEGER PRIMARY KEY, city TEXT);
CREATE TABLE fact_sales (
date_key TEXT REFERENCES dim_date(date_key),
product_key INTEGER REFERENCES dim_product(product_key),
store_key INTEGER REFERENCES dim_store(store_key),
qty INTEGER,
amount INTEGER
);
INSERT INTO dim_date VALUES ('2026-05-10', 2026, 5);
INSERT INTO dim_product VALUES (1, 'Phone', 'Electronics'), (2, 'Book', 'Books');
INSERT INTO dim_store VALUES (1, 'Dhaka'), (2, 'Chittagong');
INSERT INTO fact_sales VALUES
('2026-05-10', 1, 1, 2, 50000),
('2026-05-10', 2, 1, 5, 2250),
('2026-05-10', 1, 2, 1, 25000);
-- Single-hop analytic query: revenue per category per city.
SELECT p.category, s.city, SUM(f.amount) AS revenue
FROM fact_sales f
JOIN dim_product p ON p.product_key = f.product_key
JOIN dim_store s ON s.store_key = f.store_key
GROUP BY p.category, s.city;
6. OLTP vs OLAP — Two Worlds, One Company
🟢 OLTP (operational)
- Many small writes per second.
- Row-oriented storage.
- Highly normalized (3NF / BCNF).
- Examples: bKash, Daraz checkout, login.
- Tools: PostgreSQL, MySQL, SQL Server.
🟡 OLAP (analytical)
- Few queries, each scanning huge ranges.
- Column-oriented storage.
- Heavily denormalized (star schema).
- Examples: monthly revenue, churn analysis.
- Tools: BigQuery, Snowflake, ClickHouse, DuckDB.
অধিকাংশ কোম্পানি OLTP-এর জন্য PostgreSQL ব্যবহার করে এবং nightly ETL দিয়ে data কপি করে BigQuery বা Snowflake-এ দেয়। এক truth, দুই shape।
7. Rules of Safe Denormalization
- Always start normalized. Denormalize only when a measured query is too slow.
- Document the duplication. Every cached column needs a comment naming the source-of-truth table and the maintenance trigger.
- Cover INSERT, UPDATE, DELETE. If any of the three is missing a maintenance hook, your aggregate will drift.
- Add a recompute job. Even with perfect triggers, run a nightly full recompute to detect drift early.
- Treat the source as truth. When the cache disagrees with the base tables, the base tables win — never the other way around.
- Re-test after every schema change. Adding a new column to
reviewsmust trigger a review of the cached aggregate.
8. Glossary (শব্দকোষ)
| Term | Meaning | বাংলায় |
|---|---|---|
| Denormalization | Intentional redundancy for read performance. | read দ্রুত করার জন্য ইচ্ছাকৃত পুনরাবৃত্তি। |
| Materialized view | A query result stored as a table. | একটি query-এর ফলাফল table-এ সংরক্ষিত। |
| Cached aggregate | A column holding a running SUM/COUNT/AVG. | SUM/COUNT/AVG-এর pre-computed কলাম। |
| OLTP | Online transaction processing — daily writes. | দৈনন্দিন write-নির্ভর system। |
| OLAP | Online analytical processing — large reads. | বড় read-নির্ভর analytics system। |
| Star schema | Central fact table joined to flat dimension tables. | একটি কেন্দ্রীয় fact + dimension টেবিল। |
| Snowflake schema | Star schema with normalized dimensions. | dimension গুলো নিজেরাও normalized। |
9. Practice Problems
Twelve problems on real denormalization decisions. Most have a runnable SQLite answer.
-
Add a trigger that decrements
review_countandstars_sumwhen a review is deleted.review delete হলেreview_countওstars_sumকমানোর জন্য trigger লিখুন।✨ Show Answer (উত্তর দেখুন)
ans1.sqlCREATE TRIGGER reviews_after_delete AFTER DELETE ON reviews BEGIN UPDATE products SET review_count = review_count - 1, stars_sum = stars_sum - OLD.stars WHERE product_id = OLD.product_id; END; INSERT INTO reviews(review_id, product_id, stars) VALUES (10, 1, 4); DELETE FROM reviews WHERE review_id = 10; SELECT * FROM products; -
Write an UPDATE trigger so editing a review's star value keeps the cached sum correct.review edit করার জন্য UPDATE trigger লিখুন।
✨ Show Answer (উত্তর দেখুন)
ans2.sqlCREATE TRIGGER reviews_after_update AFTER UPDATE OF stars ON reviews BEGIN UPDATE products SET stars_sum = stars_sum + NEW.stars - OLD.stars WHERE product_id = NEW.product_id; END; UPDATE reviews SET stars = 3 WHERE review_id = 1; SELECT * FROM products; -
A reconciliation query — write SQL that finds products whose cached
review_countdisagrees with the actual count of rows inreviews.cached count এবং actual count মিলছে না — এমন product বের করুন।✨ Show Answer (উত্তর দেখুন)
ans3.sqlSELECT p.product_id, p.review_count AS cached, COUNT(r.product_id) AS actual FROM products p LEFT JOIN reviews r ON r.product_id = p.product_id GROUP BY p.product_id, p.review_count HAVING p.review_count <> COUNT(r.product_id); -
Suggest one place where storing
customer_full_nameon everyordersrow is correct, not a bug.প্রতিটিordersrow-এcustomer_full_nameরাখা কখন সঠিক?✨ Show Answer (উত্তর দেখুন)
Answer: When you need a frozen historical snapshot. If the customer later changes their name, the invoice should still reflect the name at the time of purchase. This is not denormalization — it is treating the value as a different fact (the name-on-the-day-of-sale) which simply happens to coincide with the live name today.
-
Write a single SQL statement that fully recomputes
review_countandstars_sumfrom scratch, as a nightly drift check.nightly recompute-এর জন্য SQL লিখুন।✨ Show Answer (উত্তর দেখুন)
ans5.sqlUPDATE products SET review_count = (SELECT COUNT(*) FROM reviews r WHERE r.product_id = products.product_id), stars_sum = COALESCE((SELECT SUM(stars) FROM reviews r WHERE r.product_id = products.product_id), 0); SELECT * FROM products; -
List two queries on a Daraz-style site that should NOT be denormalized, and explain why.কোন দুটি query denormalize করা উচিত নয় — কেন?
✨ Show Answer (উত্তর দেখুন)
Answer: (1) Customer billing address — must always be the latest, not a cached copy that drifts. (2) Real-time stock count — denormalizing into a cached column risks selling phantom inventory under concurrency. Both want strict consistency more than speed.
-
Convert a normalized
order_items(order_id, product_id, qty)+products(product_id, price)into a fact table for nightly reporting.উপরের টেবিলগুলো থেকে fact_order_items তৈরি করুন।✨ Show Answer (উত্তর দেখুন)
ans7.sqlCREATE TABLE order_items (order_id INTEGER, product_id INTEGER, qty INTEGER); CREATE TABLE products (product_id INTEGER PRIMARY KEY, name TEXT, price INTEGER); INSERT INTO products VALUES (1,'Phone',25000), (2,'Book',450); INSERT INTO order_items VALUES (100,1,2), (100,2,3); CREATE TABLE fact_order_items AS SELECT oi.order_id, oi.product_id, p.name, oi.qty, p.price, oi.qty * p.price AS line_total FROM order_items oi JOIN products p ON p.product_id = oi.product_id; SELECT * FROM fact_order_items; -
Snowflake the
dim_productdimension by extractingcategoryintodim_category. Show the new schema.dim_product-কে snowflake করেdim_categoryবের করুন।✨ Show Answer (উত্তর দেখুন)
ans8.sqlCREATE TABLE dim_category (category_key INTEGER PRIMARY KEY, category_name TEXT); CREATE TABLE dim_product ( product_key INTEGER PRIMARY KEY, name TEXT, category_key INTEGER REFERENCES dim_category(category_key) ); INSERT INTO dim_category VALUES (1,'Electronics'),(2,'Books'); INSERT INTO dim_product VALUES (10,'Phone',1),(11,'Book',2); SELECT p.name, c.category_name FROM dim_product p JOIN dim_category c ON c.category_key = p.category_key; -
A junior engineer suggests replacing every JOIN in the OLTP database with denormalized columns. Give two reasons this is dangerous.প্রতিটি JOIN-কে denormalized column দিয়ে replace করা কেন বিপজ্জনক — দুটি কারণ দিন।
✨ Show Answer (উত্তর দেখুন)
Answer: (1) Every write now updates many rows; INSERT/UPDATE latency explodes. (2) Drift becomes inevitable — without disciplined triggers and reconciliation, the cached copies diverge silently from the source of truth, producing wrong financial reports and customer-visible bugs.
-
Identify which tables in a bKash transaction system you would NOT denormalize and why.bKash-এ কোন কোন টেবিল কখনো denormalize করবেন না — কেন?
✨ Show Answer (উত্তর দেখুন)
Answer: The
tx(transaction) andaccount_balancetables. Both demand strict, second-by-second correctness. A denormalized column drifting by even Tk 0.01 is an audit failure. Reads here are also already cheap — accounts are looked up by primary key. -
Write a query that uses the
daily_revenuematerialized table to find the top 3 revenue days.daily_revenueথেকে top 3 দিন বের করুন।✨ Show Answer (উত্তর দেখুন)
ans11.sqlSELECT order_date, revenue FROM daily_revenue ORDER BY revenue DESC LIMIT 3; -
Describe a "100× faster" denormalization story you can defend in a code review.code review-তে defend করার মতো একটি "100× faster" denormalization গল্প বর্ণনা করুন।
✨ Show Answer (উত্তর দেখুন)
Answer: "Our product page query joined
products,reviewsandorder_itemswith two GROUP BYs. p99 latency was 480 ms. We added aproducts.review_countandproducts.stars_sumcolumn, maintained by three triggers covering insert/update/delete, plus a nightly reconciliation script. p99 dropped to 4 ms — a 120× improvement on a page that loads 3 million times per day. Reconciliation has flagged drift exactly twice in 18 months, both caught and fixed within an hour."একটি বাস্তব গল্প — measure, change, measure again, এবং drift ধরার জন্য nightly reconciliation। এটিই defendable engineering।
Summary — Module 28
Normalization is a starting point, not the destination. Once you measure a real performance problem, controlled denormalization — cached aggregates, materialized views, star schemas — can deliver order-of-magnitude wins. The price is consistency obligations: triggers, reconciliation jobs and discipline. OLTP systems lean normalized; OLAP warehouses lean denormalized; most real companies run both, connected by ETL.