Redis Deep Dive — Data Structures, Pub/Sub, Persistence
Redis — in-memory data structure store
1. Redis Is Not Just a Cache
Most engineers meet Redis through one line in their backend — redis.get(key) in front of a
slow SQL query. That works. But underestimating Redis there is like buying a Toyota and using only the
cup holder. Redis is a complete data structure server that holds your data in RAM and
answers queries in microseconds. Companies use it as cache, queue, leaderboard, rate-limiter, session
store, real-time analytics engine — sometimes as the primary database.
Two facts to anchor everything else in this module:
- Redis is single-threaded for command execution. Every command runs to completion before the next starts. That makes most operations atomic for free.
- Redis stores everything in RAM. It can persist to disk for durability, but the working set must fit in memory. Capacity planning starts with "how big is my hot data?"
2. The Eight Core Data Structures
Every Redis key has a type. Choosing the right type is most of using Redis well. Here is the menu.
| Type | Holds | Typical Bangladeshi use case |
|---|---|---|
| String | Bytes — text, JSON, counter, even a JPEG. | Cache the rendered HTML of an article from Prothom Alo. |
| List | Ordered linked list with O(1) push/pop on either end. | Background job queue for SMS-sending workers. |
| Set | Unordered collection of unique strings. | "Users who liked this Facebook post" — fast intersection. |
| Sorted Set (ZSET) | Set + a score per element. Always sorted by score. | PUBG / Free Fire leaderboard, top-100 trending products. |
| Hash | String-to-string map under one key — like a row. | User profile fields stored under user:42. |
| Stream | Append-only log with consumer groups. | Order events feeding a fraud-detection worker. |
| Bitmap | Bits inside a string — set/get individual bits. | Daily-active-user tracking — 1 bit per user. |
| HyperLogLog | Probabilistic cardinality estimator (~12 KB, 0.81% error). | "Unique visitors today" without storing every userId. |
3. Strings, Lists, Sets, Hashes — The Bread and Butter
These four cover 80% of Redis use. Here is each one in three lines of redis-cli.
# Cache an article body for 10 minutes
SET article:1024 "<article>...</article>" EX 600
GET article:1024
TTL article:1024 # seconds left to live
# Atomic counter — page views
INCR pageviews:home
INCRBY pageviews:home 5
GET pageviews:home # "6"# Job queue — producers LPUSH, workers BRPOP (blocking)
LPUSH q:sms '{"to":"+8801711000001","text":"OTP 4821"}'
LLEN q:sms # queue depth
# Worker — block up to 30 seconds for next job
BRPOP q:sms 30# Track which users liked post 99
SADD likes:post:99 user:1 user:7 user:42
SISMEMBER likes:post:99 user:7 # 1 / 0
SCARD likes:post:99 # number of likes
# Friends-in-common — set intersection in one command
SINTER friends:user:1 friends:user:7# A user profile — one round trip to read/write many fields
HSET user:42 name "Arif" city "Dhaka" coins 120
HGET user:42 city # "Dhaka"
HINCRBY user:42 coins 10 # atomic — 130
HGETALL user:42user:42:name, user:42:city, ... uses ~10× the
memory of a single user:42 hash. Small hashes are encoded as a packed listpack and are
extraordinarily compact.
4. The Sorted Set (ZSET) — Redis's Killer Structure
A sorted set stores unique members each with a score (a 64-bit float). Members are kept in score order automatically. Lookup, insertion, and rank queries all take O(log N). It is the structure that powers leaderboards, top-K trending lists, time-windowed feeds, and even priority queues.
# Game leaderboard — score per player
ZADD lb:battlefield 2450 arif 3120 mim 1980 tanvir 3870 nadia
# Top 3 — highest first
ZREVRANGE lb:battlefield 0 2 WITHSCORES
# 1) "nadia" 3870
# 2) "mim" 3120
# 3) "arif" 2450
# Arif scored 500 more — atomic update
ZINCRBY lb:battlefield 500 arif
# What rank is Arif now (0-based, highest first)?
ZREVRANK lb:battlefield arif
# Players in score range 2000..3500
ZRANGEBYSCORE lb:battlefield 2000 3500 WITHSCORESORDER BY ... LIMIT 10 on
millions of rows under update load.
5. Streams, Bitmaps and HyperLogLog
Streams — durable append-only log
A Redis Stream is like a tiny Kafka inside Redis. Producers XADD events; consumers read with
XREAD (or, for fan-out with acknowledgements, XREADGROUP). Each entry has an
auto-generated id of the form <timestamp>-<seq>, so events are naturally time-sorted.
# Producer — log a new order event
XADD orders:events * orderId 7821 total 5400 city "Dhaka"
# Create a consumer group "fraud-checker"
XGROUP CREATE orders:events fraud-checker $ MKSTREAM
# Worker reads the next batch — up to 10 events, block 5s if empty
XREADGROUP GROUP fraud-checker worker-1 COUNT 10 BLOCK 5000
STREAMS orders:events >
# After processing, acknowledge so the message won't be redelivered
XACK orders:events fraud-checker 1700000000000-0Pub/Sub vs Streams
| PUB/SUB | Streams | |
|---|---|---|
| Storage | None — fire and forget | Durable in memory + AOF |
| Late subscriber sees old msgs? | No | Yes (replay from any id) |
| Acknowledge / retry | No | Yes (consumer groups) |
| Use case | Live chat, "someone is typing" | Order events, audit log, ETL |
Bitmaps — 1 bit per user
# Today is 2025-08-12. User 42 logged in:
SETBIT dau:2025-08-12 42 1
# Did user 42 log in today?
GETBIT dau:2025-08-12 42 # 1
# How many distinct users today?
BITCOUNT dau:2025-08-12
# Users active on BOTH days — bitwise AND across two bitmaps
BITOP AND dau:both dau:2025-08-12 dau:2025-08-13
BITCOUNT dau:bothHyperLogLog — count distinct in 12 KB
# Track unique visitors per article — fixed 12KB per HLL key, 0.81% error
PFADD uv:article:1024 user:1 user:7 user:42 user:1
PFCOUNT uv:article:1024 # ≈ 3 (duplicates collapsed)
# Combine many days for "monthly unique"
PFMERGE uv:august uv:2025-08-01 uv:2025-08-02 uv:2025-08-03Counting them with HyperLogLog: 12 KB. Trade exactness for size whenever exactness is unnecessary.
6. Pipelining vs MULTI/EXEC Transactions
A network round trip from your app server to Redis is typically 200–500 µs — a thousand times longer than the actual command execution. Sending 100 commands one by one waits 100 round trips. Both pipelining and transactions attack this, but they are not the same thing.
| Pipelining | MULTI/EXEC | |
|---|---|---|
| What it does | Batches commands into one network round trip. | Queues commands; all run atomically together. |
| Atomic? | No — other clients can interleave. | Yes — no other command runs between MULTI and EXEC. |
| Use when | Mass insert/load. | Multi-step state change that must succeed together. |
# Transfer 50 coins from arif to mim — atomic
MULTI
HINCRBY wallet:arif coins -50
HINCRBY wallet:mim coins 50
EXEC
# Either both HINCRBY happen or (on DISCARD) neither.7. Persistence — RDB, AOF and Hybrid
Redis lives in RAM, but RAM is volatile — a power cut and your data is gone. Redis offers three durability modes; you choose your balance of safety, restart speed and disk pressure.
RDB — point-in-time snapshot
- Compact single binary file — fast restart.
- Forks the process, dumps memory.
- Risk: minutes of data loss between snapshots.
AOF — append-only file
- Every write command is appended to a log.
- fsync policy =
everysecby default → ≤1 s loss. - Slower restart on huge AOF (rewrite compacts it).
None
- Pure cache mode — fast, fully volatile.
- Acceptable only when source of truth is elsewhere.
aof-use-rdb-preamble yes + appendonly yes.
# Save an RDB snapshot if 1+ keys changed in last 900s,
# or 100+ in 300s, or 10000+ in 60s.
save 900 1
save 300 100
save 60 10000
# AOF on, fsync once per second, hybrid format on.
appendonly yes
appendfsync everysec
aof-use-rdb-preamble yes8. Redis Cluster and Redis Sentinel
One Redis node holds at most a few hundred GB of RAM and serves at most a few hundred thousand ops/s. For more, scale horizontally with Redis Cluster; for high availability of a single shard, use Redis Sentinel.
Sentinel — HA for a single shard
A small set of Sentinel processes monitor a primary + N replicas. If the primary dies, they elect a replica as the new primary and tell clients about the new address. Good fit when your dataset fits in one node but downtime is unacceptable.
Cluster — sharding via hash slots
Redis Cluster splits the key space into 16384 hash slots. Each key's slot is
CRC16(key) mod 16384. Each cluster node owns a contiguous range of slots and replicates
them to one or more replicas. The client library reads the cluster topology and routes each command
directly to the right node — no proxy needed.
MSET a 1 b 2 works only if a and b live in the same slot.
Force this by wrapping the routing portion in {...} — e.g. order:{1234}:items
and order:{1234}:total hash to the same slot, so a transaction over both is allowed.
9. Idiomatic Patterns Every Backend Engineer Should Know
Cache-aside (lazy loading)
# Read path (pseudo-code)
# value = GET user:42
# if value is nil:
# value = SELECT * FROM users WHERE id=42 -- slow path
# SET user:42 value EX 600 -- repopulate cache for 10 min
# return value
SET user:42 '{"name":"Arif","city":"Dhaka"}' EX 600Distributed lock — SET key val NX EX
# Acquire — only succeeds if key did not exist (NX). Auto-expires in 30s.
SET lock:report:daily "<random-uuid>" NX EX 30
# Returns OK on first caller, nil on every other → only one worker runs.
# Release — only if WE still own it (compare-and-delete via Lua).
EVAL "if redis.call('get',KEYS[1])==ARGV[1] then
return redis.call('del',KEYS[1]) else return 0 end"
1 lock:report:daily "<random-uuid>"SET ... NX without EX is a deadlock waiting to happen — if the worker crashes
before DEL, the lock lives forever. Always pair NX with EX.
Rate limiting — INCR + EXPIRE
# Limit a phone number to 3 OTPs per minute
# key = otp:rl:<phone>:<minute-bucket>
INCR otp:rl:+8801711000001:202508121430
EXPIRE otp:rl:+8801711000001:202508121430 90
# If the returned value > 3 → 429 Too Many Requests.Leaderboard — already shown in §4 with ZSET.
Job queue (FIFO) — LIST + BRPOP
Producers LPUSH, workers BRPOP. The blocking variant means workers do not poll
— they wake up the instant a job arrives. For at-least-once semantics with retries, use Streams + consumer
groups instead.
10. Practice Problems
These are mostly conceptual. The redis-cli snippets in answers can be tried in the free Redis Cloud tier or a local redis-server.
-
Which Redis structure would you use for a Daraz "trending products in last hour" widget? Why?"গত এক ঘণ্টায় trending product" — কোন structure ব্যবহার করবেন?
Show Answer
Sorted Set (ZSET) keyed by
trending:<hour>, member = productId, score = view count.ZINCRBYon each view,ZREVRANGE 0 9 WITHSCORESfor the top 10. The hour-bucketed key naturally expires old data.ZSET — score হিসেবে view count, member হিসেবে productId। প্রতি ঘণ্টায় নতুন key ব্যবহার করলে পুরাতন data এমনিতেই বাদ যাবে।
-
Why is Redis "single-threaded for command execution" actually a feature, not a limitation?single-threaded হওয়া কেন সুবিধা?
Show Answer
Two big wins. (1) Every command is atomic with respect to other clients — no race conditions, no need for locks inside Redis. (2) Cache locality is perfect; CPU pipelines are never stalled by lock contention. The trade-off is one CPU per node — Redis Cluster scales across cores by running multiple shards.
প্রতিটি command atomic — race condition নেই। আর CPU cache পুরোপুরি কাজে লাগে। Multiple core ব্যবহারের জন্য Redis Cluster চালান।
-
A worker crashes 200 ms after acquiring
SET lock:foo "x" NX(no EX). What happens?EX ছাড়া lock নিয়ে worker crash করলে কী হয়?Show Answer
The lock key sits in Redis forever. No other worker will ever acquire it — permanent deadlock. Always use
NX EX 30(or similar) so the lock auto-releases.Lock চিরকাল আটকে থাকবে — পুরো system block হয়ে যাবে। তাই সবসময়
EXদিতে হবে। -
Which is more memory-efficient for storing 1M sessions of {userId, expiresAt, csrf}: 3M plain string keys, or 1M Hashes?১০ লাখ session কোন structure-এ ছোট হবে?
Show Answer
1M Hashes. Each plain string key has a fixed ~50–100 byte overhead (key string, expiry slot, dictionary entry). A small hash with three fields fits in one packed listpack — often under 80 bytes for the whole record. The 3M-key approach can use 5–10× more RAM for the same data.
-
In one sentence each: when do you choose Pub/Sub, and when do you choose Streams?Pub/Sub বনাম Streams — কখন কোনটি?
Show Answer
Pub/Sub: losing a message is acceptable and you want the lowest possible latency (live chat, "user is typing", presence). Streams: messages must not be lost, slow consumers must be able to catch up, and processing must be acknowledged (order events, audit log, ETL).
-
Show the redis-cli command(s) to atomically transfer 100 coins from user:7 to user:42, both stored as Hashes.user:7 → user:42 — ১০০ coin atomic transfer।
Show Answer
ans6.redisMULTI HINCRBY user:7 coins -100 HINCRBY user:42 coins 100 EXEC # In Cluster mode, force same slot with hash tags: # HINCRBY {wallet}:user:7 coins -100 # HINCRBY {wallet}:user:42 coins 100 -
Estimate: storing 50 million unique daily visitors as a Set vs as a HyperLogLog — what is the memory difference?৫ কোটি unique visitor — Set বনাম HLL-এ memory পার্থক্য কত?
Show Answer
A Set storing 50M short string ids uses roughly 3 GB. A HyperLogLog uses a fixed 12 KB regardless of cardinality, with ~0.81% error. That is a ~250,000× reduction. Use HLL whenever you only need the count, not the membership.
-
How many round trips does pipelining 1000
SETs make? How does that compare to MULTI/EXEC of the same 1000 SETs?১০০০টি SET — pipelining ও MULTI/EXEC-এ কতটি round trip?Show Answer
Both make essentially 1 round trip — the entire batch is sent before any reply is read. The difference is atomicity: pipelining allows other clients' commands to interleave, MULTI/EXEC does not. Use pipelining for bulk loading; use MULTI/EXEC when the commands must succeed as a unit.
-
A Redis instance is configured with only RDB (save 900 1 / 300 100 / 60 10000). The host loses power 4 minutes after the last snapshot. How much data is lost?RDB-only — শেষ snapshot-এর ৪ মিনিট পরের সব write হারাবে?
Show Answer
Up to 4 minutes of writes — anything after the last successful RDB snapshot. To shrink the loss window, enable AOF with
appendfsync everysecalongside RDB; loss drops to ≤1 second.শেষ ৪ মিনিটের write হারাবে। AOF চালু করলে এটি ≤ ১ সেকেন্ডে নেমে আসে।
-
In Redis Cluster, why does
MSET a 1 b 2 c 3sometimes fail withCROSSSLOT?CROSSSLOTerror কখন হয়?Show Answer
The keys hash to different slots, which live on different shards. A single command can only operate within one slot. Workarounds: (a) split into multiple MSETs; (b) use hash tags —
{user:42}:a,{user:42}:b— to force all keys into one slot. -
Design a 100 requests / 15 minutes per IP rate limiter using only Redis primitives.প্রতি IP-র জন্য ১৫ মিনিটে ১০০ request — কীভাবে বানাবেন?
Show Answer
ans11.redis# bucket = ip + floor(now / 900) → window-rounded key INCR rl:<ip>:<bucket> # returns the new count EXPIRE rl:<ip>:<bucket> 900 # first call sets TTL; later calls noop # If returned count > 100 → HTTP 429.প্রতি ১৫ মিনিটের bucket-এ INCR; প্রথম INCR-এ EXPIRE। count > 100 হলে block।
-
Why is using
KEYS *in production a fireable offence?production-এKEYS *কেন ব্যবহার করা যাবে না?Show Answer
KEYSis O(N) over the entire keyspace and runs on Redis's single command thread — every other client is blocked while it scans, possibly for seconds. UseSCANinstead: cursor-based, incremental, non-blocking.পুরো keyspace block হয়ে যায়; অন্য সব client wait করে। বিকল্প —
SCAN। -
Build a "users active on at least one of the last 7 days" report using Bitmaps.গত ৭ দিনে অন্তত একদিন active ছিল এমন user-সংখ্যা — Bitmap দিয়ে।
Show Answer
ans13.redisBITOP OR weekly:active dau:2025-08-06 dau:2025-08-07 dau:2025-08-08 dau:2025-08-09 dau:2025-08-10 dau:2025-08-11 dau:2025-08-12 BITCOUNT weekly:activeOR across each day's daily-active bitmap, then BITCOUNT.
-
In two sentences, when is Redis the wrong tool?Redis কখন বেমানান?
Show Answer
Answer: When the working set exceeds the memory you can afford (Redis is RAM-priced, not disk-priced) — or when the workload needs rich, ad-hoc analytical queries against denormalised data, like Postgres or BigQuery do. Redis is shaped for known-shape, latency-critical access patterns; it is not a substitute for SQL when you do not yet know the questions you will ask.
যখন data RAM-এ ধরে না, অথবা ad-hoc analytical query দরকার — তখন Postgres/BigQuery ভালো। Redis তৈরি দ্রুত, পরিচিত pattern-এর জন্য।
Summary — Module 45
Redis is an in-memory data structure server: Strings, Lists, Sets, Sorted Sets, Hashes,
Streams, Bitmaps and HyperLogLogs are not bolt-ons — they are first-class types. Pub/Sub fires events
without storage; Streams store them durably with consumer groups. Pipelining batches
for speed; MULTI/EXEC batches for atomicity. RDB snapshots restart
fast; AOF caps loss to one second; the hybrid format gives both. Sentinel
handles HA for one shard, Redis Cluster shards across many nodes via 16384 hash slots.
Master the patterns — cache-aside, distributed lock with SET NX EX, leaderboard with ZSET,
rate limit with INCR + EXPIRE — and you have the toolkit behind half the high-traffic
backends in the world.