Elasticsearch & Search Engines

Elasticsearch ও search engine — full-text search, log analytics ও vector retrieval

Read: ~40 min Intermediate 12 practice problems Query DSL

1. Why `LIKE '%bangla%'` Will Never Be Enough

You can search a SQL column with WHERE description LIKE '%shirt%'. It works on a thousand rows. It collapses on a million. And it cannot tell you that "shirts" should match "shirt", that "sirt" is a likely typo, that "jamaa" is the romanized form of "জামা", or that this document is a better match than that one.

A search engine like Elasticsearch is a database whose index is built for exactly this — turning text into searchable tokens, ranking results by relevance, scaling to billions of documents, and increasingly combining classical text retrieval with vector similarity for AI-era search.

মূল কথা: SQL-এর LIKE ছোট ডেটাতে চলে, কিন্তু lakh-koti document-এ ভেঙে পড়ে। Elasticsearch একটি বিশেষ ধরনের database — যেটি text-কে token-এ ভেঙে আগে থেকেই index তৈরি করে রাখে, এবং relevance score (BM25) দিয়ে ফলাফল ranking করে। শুধু text নয়, এটি logs, metrics এবং AI-এর জন্য vector embedding-ও খুঁজতে পারে।

In this module: how an inverted index works; analyzers (including icu_analyzer for Bangla); mappings (the all-important text vs keyword distinction); the Query DSL; aggregations; BM25 scoring; the ELK stack for log analytics; hybrid retrieval with vector search; and a tour of competitors (OpenSearch, Meilisearch, Typesense).

2. The Inverted Index — Reading Books Backwards

A relational index is "row → values": given row 42, find its name. A search index is the reverse: "value → rows". Given the token "laptop", give me every document that contains it. That is the inverted index.

একটি বইয়ের শেষে index page-এর কথা ভাবুন — সেখানে প্রতিটি keyword-এর পাশে কোন কোন page-এ সেটি আছে তার তালিকা। Search engine ঠিক সেই কাজটি বিশাল scale-এ করে: প্রতিটি token-এর জন্য সেটি যেসব document-এ আছে তাদের তালিকা (posting list) আগে থেকেই তৈরি থাকে। তাই query-র সময় শুধু সেই তালিকা পড়লেই হয়।

Document	Text
doc1	"Buy a red Bangla shirt"
doc2	"Red shirt for boys"
doc3	"A blue shirt and red pants"

After tokenization, lowercasing and stop-word removal, the inverted index looks like this:

Token	Posting list (doc, position)
`buy`	(doc1, 0)
`red`	(doc1, 2), (doc2, 0), (doc3, 4)
`bangla`	(doc1, 3)
`shirt`	(doc1, 4), (doc2, 1), (doc3, 2)
`boys`	(doc2, 3)
`blue`	(doc3, 1)
`pants`	(doc3, 5)

Now "red shirt" just looks up two posting lists, intersects them on doc_id, and optionally checks that the positions are adjacent (for phrase queries). All this happens in microseconds even with billions of documents — because the work is logarithmic in vocabulary size, not document count.

3. Analyzers — How Text Becomes Tokens

An analyzer is the pipeline that turns a string into a list of tokens for the inverted index. It runs at both index time (when you ingest a document) and query time (when you search), and they must agree.

An analyzer has three stages:

Character filter — strip HTML, fix curly quotes, normalize Unicode.
Tokenizer — split text into tokens (whitespace, n-gram, ICU, Standard, etc.).
Token filter — lowercase, remove stop words, stem (running → run), normalize Bangla diacritics.

একটি analyzer তিনটি ধাপে কাজ করে: (১) Character filter — অপ্রয়োজনীয় symbol সরায়, (২) Tokenizer — text-কে token-এ ভাঙে, (৩) Token filter — lowercase করে, stop word ফেলে দেয়, stemming করে। Bangla-এর জন্য Elastic-এর analysis-icu plugin ব্যবহার করুন — এটি Unicode standard অনুযায়ী Bangla text সঠিকভাবে tokenize ও normalize করতে পারে।

Analyzer	Use it for	Example tokens of "Quick-running cats!"
`standard`	General-purpose multilingual.	`quick`, `running`, `cats`
`whitespace`	Code, IDs, log lines where punctuation matters.	`Quick-running`, `cats!`
`english`	English text with stemming + stopwords.	`quick`, `run`, `cat`
`edge_ngram`	Search-as-you-type / autocomplete.	`q`, `qu`, `qui`, `quic` ...
`icu_analyzer`	CJK, Arabic, Bangla — Unicode-aware.	Bangla: `বাংলা`, `শার্ট`, `লাল`

create_index.json

// PUT /products — index with a Bangla-aware analyzer
{
  "settings": {
    "analysis": {
      "analyzer": {
        "bangla_icu": {
          "type":      "custom",
          "tokenizer": "icu_tokenizer",
          "filter":    ["lowercase", "icu_normalizer", "icu_folding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name":        { "type": "text", "analyzer": "bangla_icu" },
      "category":    { "type": "keyword" },
      "price":       { "type": "integer" },
      "created_at":  { "type": "date" }
    }
  }
}

4. Mappings — `text` vs `keyword`

This is the single most-misunderstood concept in Elasticsearch and the source of about half of all production bugs. Both are strings. They are not the same.

📖 `text`

Goes through an analyzer.
Stored as tokens in the inverted index.
For full-text search with match.
Cannot be sorted, aggregated on, or used in term queries reliably.
Example: a product description, a blog post body.

🏷️ `keyword`

NOT analyzed — stored as the exact original string.
For filtering, sorting, aggregating.
Used with term / terms queries.
Example: a country code, a status enum, a user-id, a tag.

Rule of thumb: মানুষ যেটি পড়বে এবং তার মধ্যে শব্দ খোঁজা হবে — সেটি text। কম্পিউটার যেটি ঠিক একই রূপে match করবে, sort/group করবে — সেটি keyword। Status code, country, tag, user-id — সবই keyword। Description, title — text। অনেক সময় একটি field-এ দুটোই দরকার হয় — সেক্ষেত্রে name field-এ name.keyword সাব-field রাখুন।

The classic bug You ingest "category": "Mobile Phone". You query {"term": {"category": "Mobile Phone"}} and get zero results. Why? Because category is mapped as text, so it was stored as tokens ["mobile", "phone"]. term looks for the exact token "Mobile Phone" — which doesn't exist. Use keyword for categories.

5. The Query DSL — JSON-Powered Search

Elasticsearch queries are JSON documents POSTed to the _search endpoint. There are two categories: queries (which compute a relevance score) and filters (which yes/no exclude documents and are cacheable).

The basic clauses

match — full-text search, runs the same analyzer as the field.
match_phrase — words must appear adjacent and in order.
term — exact token match (use only on keyword / numeric).
range — between two values (numbers, dates).
wildcard — "laptop*" patterns. Slow on big indexes.
bool — combine clauses with must, should, must_not, filter.

match = relevance score-সহ full-text search। term = exact match (status, id-এর মতো keyword field-এ)। range = সংখ্যা/তারিখের মধ্যে। bool = এদের সবগুলোকে একসাথে যুক্ত করার logical wrapper — must (AND, score করে), filter (AND, score করে না, cache হয়), should (OR, score বাড়ায়), must_not (NOT)।

search.json

// GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "red bangla shirt" } }
      ],
      "filter": [
        { "term":  { "category": "clothing" } },
        { "range": { "price":    { "gte": 200, "lte": 2000 } } }
      ],
      "should": [
        { "match": { "name": "cotton" } }
      ],
      "must_not": [
        { "term": { "status": "out_of_stock" } }
      ]
    }
  },
  "size": 10,
  "sort": ["_score", { "created_at": "desc" }]
}

Read this query in plain English: "Find products whose name analytically matches 'red bangla shirt', in category 'clothing', priced 200–2000 BDT, not out of stock; bonus relevance if they also mention cotton."

Phrase and prefix variants

phrase.json

// match_phrase — words must be adjacent and in order
{ "query": { "match_phrase": { "name": "bangla shirt" } } }

// match_phrase_prefix — last word can be a prefix (autocomplete)
{ "query": { "match_phrase_prefix": { "name": "bangla shi" } } }

// multi_match — search the same query across many fields, with boost
{ "query": { "multi_match": {
    "query":  "bangla shirt",
    "fields": ["name^3", "description", "tags^2"]
} } }

6. Aggregations — SQL `GROUP BY` on Steroids

Search engines aren't just for finding documents — they are surprisingly capable analytics engines. Elasticsearch aggregations are like SQL GROUP BY, but compose into trees: you can bucket logs by hour, then within each bucket compute average response time by service, then within that find the top 5 endpoints. All in one round trip.

Metric aggregations compute numbers: avg, sum, min, max, cardinality, percentiles.
Bucket aggregations split documents into groups: terms (group by field), date_histogram (group by time interval), range, histogram.
Pipeline aggregations run on top of other aggs: moving averages, derivatives.

Aggregation = GROUP BY-এর শক্তিশালী রূপ। Bucket aggregation document-গুলোকে গ্রুপে ভাগ করে (যেমন প্রতি ঘণ্টা, প্রতি category)। Metric aggregation প্রতি গ্রুপের ভেতরে গণনা করে (avg response time, sum amount)। এদের nested করেও ব্যবহার করা যায় — যেমন "প্রতি ঘণ্টায় প্রতি service-এর average latency"।

aggs.json

// Per-day request count, per-day average latency, top 5 endpoints
{
  "size": 0,
  "query": { "range": { "@timestamp": { "gte": "now-7d/d" } } },
  "aggs": {
    "per_day": {
      "date_histogram": { "field": "@timestamp", "calendar_interval": "day" },
      "aggs": {
        "avg_latency":  { "avg": { "field": "latency_ms" } },
        "p99_latency":  { "percentiles": { "field": "latency_ms", "percents": [99] } },
        "top_endpoints": {
          "terms": { "field": "endpoint.keyword", "size": 5 }
        }
      }
    }
  }
}

Set "size": 0 to skip returning matching documents — you only want the aggregations.

7. BM25, the ELK Stack, and the Vector Era

How relevance is scored — BM25

When you run match, every matching document gets a _score. Elasticsearch's default scorer is BM25 (Best Matching 25), a refinement of TF-IDF that handles document length better. The intuition:

The more often a query term appears in a document, the higher the score — but with diminishing returns (term frequency saturation).
Rare terms in the corpus carry more weight than common ones (inverse document frequency).
Long documents are slightly penalised — a 5-word match in a 10-word title means more than the same match in a 10,000-word article.

BM25 মানে — যে শব্দ আপনার query-তে আছে সেটি document-এ যত বেশি বার আসবে, score তত বেশি হবে; কিন্তু একটি সীমা আছে। যে শব্দ পুরো corpus-এ বিরল, সেটির ওজন বেশি (যেমন "elasticsearch" বনাম "the")। ছোট, focused document বড় document-এর চেয়ে preference পায়। এই formula-ই Elasticsearch-এর default ranking-এর ভিত্তি।

The ELK / Elastic stack — log analytics in production

Most production deployments don't use Elasticsearch alone — they use it as the storage and search layer of the ELK stack (now called the Elastic Stack):

E — Elasticsearch: stores and indexes the data.
L — Logstash (or the lightweight Beats family): collects logs from servers, parses them, ships them to Elasticsearch.
K — Kibana: web UI for searching, dashboards, alerts.

ELK = Elasticsearch + Logstash + Kibana। বাস্তবে: প্রতিটি server-এ Filebeat (Beats family-র সদস্য) চলতে থাকে, যা log file পড়ে Elasticsearch-এ পাঠায়। Kibana-তে আপনি live dashboard দেখতে পাবেন — কোন endpoint slow, কোন error rate বাড়ছে, কোন service-এ p99 latency বেশি। অনেক company-র SRE team পুরোপুরি ELK-এর উপর নির্ভরশীল।

Figure 48.1 — A typical ELK / Elastic Stack pipeline for production log analytics.

Hybrid retrieval — BM25 + vector similarity

Modern AI-driven search combines two retrievers:

Lexical (BM25): excellent at exact terminology, product codes, names. Cannot handle paraphrase ("how do I unsubscribe" vs "cancel my account").
Vector / dense retrieval: encode every document into a high-dimensional vector using an embedding model (OpenAI, Cohere, BGE). At query time, encode the query into a vector and find documents with the closest vectors using cosine or dot-product similarity. Excellent at paraphrase and semantic similarity, but can hallucinate matches that share no terms.

Hybrid search = run both, then combine the rankings (Reciprocal Rank Fusion, or a weighted sum of normalized scores). This is the foundation of most production RAG (retrieval-augmented generation) systems today.

Hybrid search: BM25 (text-ভিত্তিক) এবং vector search (semantic) — দুটোকে একসাথে চালিয়ে ranking মিশিয়ে দেওয়া। BM25 exact term-এ ভালো, vector paraphrase-এ ভালো। দুটো মিশিয়ে দিলে আপনি দুই দুনিয়ার সুবিধা পান। আজকের প্রায় সব AI chatbot / RAG system এই ভাবেই knowledge retrieve করে।

hybrid.json

// kNN vector search + classic BM25 in one query
{
  "query": { "match": { "body": "how do I cancel my subscription" } },
  "knn": {
    "field":           "body_vector",
    "query_vector":    [0.12, -0.04, 0.88, /* ... 768 dims ... */],
    "k":               50,
    "num_candidates":  200
  },
  "rank": { "rrf": { "window_size": 100 } },
  "size": 10
}

Competitors and forks

Engine	Position
OpenSearch	Amazon-led fork of Elasticsearch 7.10 (after the 2021 license change). Same Query DSL, Apache-2 licensed, default in AWS.
Meilisearch	Tiny, single-binary, instant search-as-you-type. Great for dashboards, less for log analytics.
Typesense	Similar niche to Meilisearch — fast, friendly defaults, simple ops.
Apache Solr	The other Lucene-based veteran. Mature, used by Wikipedia and many enterprise stacks.
Vespa	Yahoo's open-source engine, exceptional at hybrid lexical + vector at huge scale.

8. Glossary (শব্দকোষ)

Term	Meaning	বাংলায়
Inverted index	Map from token to the list of documents containing it.	প্রতিটি token-এর জন্য সে কোন কোন document-এ আছে তার তালিকা।
Analyzer	Pipeline of char-filters → tokenizer → token-filters that turns text into tokens.	text-কে token-এ রূপান্তর করার pipeline।
icu_analyzer	Unicode-aware analyzer; the right choice for Bangla, CJK, Arabic.	Unicode standard অনুযায়ী tokenize করে — Bangla-এর জন্য আদর্শ।
`text`	Analyzed string — for full-text `match` queries.	analyze করা string — full-text search-এর জন্য।
`keyword`	Unanalyzed string — for filtering, sorting, aggregating.	analyze না-করা exact string — filter, sort, aggregate-এর জন্য।
BM25	The default relevance scoring formula. TF-IDF with length normalization.	default relevance scoring; TF-IDF-এর উন্নত রূপ।
ELK / Elastic Stack	Elasticsearch + Logstash/Beats + Kibana — production log analytics.	production log analytics-এর জন্য তিনটি tool-এর combo।
Hybrid retrieval	Combining BM25 with vector similarity ranking.	BM25 ও vector similarity দুটোকে একসাথে ranking-এ ব্যবহার।

9. Practice Problems

For each problem, write or read the JSON Query DSL. There is no SQL runner here — read carefully and check the answer.

প্রতিটি প্রশ্নের জন্য JSON query লিখুন বা বুঝুন। উত্তর মিলিয়ে নিন।

In one sentence, what is the fundamental data structure that makes Elasticsearch fast at full-text search?

এক বাক্যে বলুন — Elasticsearch-এর full-text search-এর গতি কোন data structure থেকে আসে?

✨ Show Answer (উত্তর দেখুন)

Answer: The inverted index — a map from each indexed token to the posting list of documents (and positions) where it appears, allowing lookups to scale with vocabulary size rather than document count.

Inverted index — প্রতিটি token-এর জন্য সেটি কোন কোন document-এ আছে তার আগে থেকে তৈরি posting list, যা vocabulary size-এ scale করে, document count-এ নয়।
Should the field order_status (values like "pending", "shipped") be mapped as text or keyword? Why?

order_status field-কে কি text না keyword mapping-এ রাখবেন?

✨ Show Answer

Answer: keyword. The values are categorical, not human prose — they will be filtered with term, sorted, and used in terms aggregations. text would be analyzed (lowercased, possibly stemmed) and would silently break term queries.

keyword। এটি একটি category-type field — filter, sort ও aggregation-এ ব্যবহার হবে।
Write a query that finds products whose name matches "wireless headphone" and whose price is between 1000 and 5000.

name = "wireless headphone" এবং price 1000–5000 BDT-এর মধ্যে এমন product খুঁজুন।
✨ Show Answer
ans3.json
```
{
  "query": {
    "bool": {
      "must":   [{ "match": { "name": "wireless headphone" } }],
      "filter": [{ "range": { "price": { "gte": 1000, "lte": 5000 } } }]
    }
  }
}
```
Difference between match and match_phrase?

match এবং match_phrase-এর পার্থক্য কী?

✨ Show Answer

Answer: match finds documents that contain the analyzed terms in any order, anywhere. match_phrase requires the terms to appear adjacent and in the same order.

match: কোনো order ছাড়াই token গুলো document-এ থাকলেই হবে। match_phrase: token গুলো ঠিক একই order-এ পাশাপাশি থাকতে হবে।
Write an aggregation query that returns the top 5 most-purchased category values, with the average price in each.

শীর্ষ ৫টি কেনা category এবং প্রতি category-এর average price।
✨ Show Answer
ans5.json
```
{
  "size": 0,
  "aggs": {
    "top_categories": {
      "terms": { "field": "category", "size": 5 },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    }
  }
}
```
Why does the query {"term": {"name": "Mobile Phone"}} often return nothing on a text-mapped name field?

name field text mapping-এ থাকলে term query কেন কাজ করে না?

✨ Show Answer

Answer: The text analyzer broke "Mobile Phone" into the tokens ["mobile", "phone"] at index time. term does no analysis — it looks up the exact token "Mobile Phone", which never existed. Use match for analyzed text, or add a name.keyword sub-field for exact lookups.

Index-এর সময় "Mobile Phone" token হিসেবে ["mobile", "phone"]-এ ভেঙে গিয়েছে। term analyze করে না, তাই exact token না পেয়ে কিছুই return করে না।
Briefly: what does the icu_analyzer add over the default standard analyzer for Bangla content?

Bangla content-এর জন্য icu_analyzer default standard-এর তুলনায় কী যোগ করে?

✨ Show Answer

Answer: ICU uses Unicode-aware word segmentation and normalization, so it correctly splits Bangla compound words, normalizes nukta and other diacritics, and folds visually similar characters. The default analyzer often fails on Indic scripts and CJK and produces poor token boundaries.

ICU Unicode-সচেতন word boundary detection এবং normalization (যেমন nukta, diacritic) করে। default analyzer Bangla-তে token সীমা ঠিকমতো ধরতে পারে না।
Explain in two sentences what BM25 rewards and what it penalizes.

BM25 কী reward করে এবং কী penalize করে — দুই বাক্যে।

✨ Show Answer

Answer: BM25 rewards documents that contain rare query terms many times (with diminishing returns). It penalizes very long documents — a 5-word match in a short title is more meaningful than the same match buried in a 10,000-word article.

BM25 reward করে — বিরল query term অনেক বার থাকা (কিন্তু saturated)। Penalize করে — অতিরিক্ত লম্বা document।
In the ELK stack, what does Logstash do that Elasticsearch alone could not?

ELK stack-এ Logstash কী করে যা Elasticsearch একা করতে পারে না?

✨ Show Answer

Answer: Logstash (or Beats) handles ingestion: it reads logs from many sources, parses unstructured lines (e.g., grok patterns over Nginx access logs), enriches them (GeoIP lookup of client IPs, parsing user-agent strings), and forwards the cleaned events to Elasticsearch. Elasticsearch is a storage + search engine — not a log shipper or parser.

Logstash log collection ও parsing করে — Nginx, app log লাইন structured JSON-এ রূপান্তর করে, GeoIP/User-agent parse করে, তারপর Elasticsearch-এ পাঠায়।
In one sentence, what is hybrid retrieval and why is it popular for AI applications?

Hybrid retrieval কী, এবং AI application-এ এটি কেন জনপ্রিয়?

✨ Show Answer

Answer: Hybrid retrieval runs both BM25 (lexical, exact-term) and vector similarity (semantic, paraphrase-aware) search and fuses their rankings — popular for RAG because BM25 catches exact product codes and names while vector search catches paraphrased questions like "cancel my plan" matching documents about "unsubscribe".

BM25 (exact term) এবং vector similarity (semantic) — দুটোকে একসাথে চালিয়ে ranking মিশিয়ে দেওয়া। RAG / AI chatbot-এ এটি জনপ্রিয় কারণ একসাথে exact term ও paraphrase দুটোই handle হয়।
Name three engines that compete with or fork Elasticsearch.

Elasticsearch-এর তিনটি প্রতিযোগী বা fork-এর নাম বলুন।

✨ Show Answer

Answer: OpenSearch (Amazon's fork), Meilisearch, Typesense, Apache Solr, Vespa — any three.

OpenSearch (AWS-এর fork), Meilisearch, Typesense, Apache Solr, Vespa — যেকোনো তিনটি।
When should you NOT use Elasticsearch?

কখন Elasticsearch ব্যবহার করা উচিত নয়?

✨ Show Answer

Answer: When you need strict transactional consistency (ACID transactions across documents), strong relational integrity (foreign keys, joins), or it is your single source of truth for critical financial data. Elasticsearch is eventually-consistent, has no real joins, and refresh is not synchronous — those properties make it the wrong place for primary OLTP storage. Use Postgres/MySQL as the system of record and stream into Elasticsearch for search.

যখন strict ACID transaction, foreign key, complex JOIN বা single source of truth দরকার — তখন। Elasticsearch eventually-consistent — তাই OLTP-এর জন্য নয়। Postgres-এ data রাখুন, Elasticsearch-এ search-এর জন্য stream করুন।

Summary — Module 48

Elasticsearch is a JSON document database whose primary index is an inverted index, making full-text search and analytics fast at billion-document scale. Text is processed by analyzers (use icu_analyzer for Bangla); fields map to either text (analyzed, for match) or keyword (exact, for filters and aggregations). The Query DSL composes match, term, range, bool with must / should / must_not / filter. Aggregations bucket and metric documents like SQL GROUP BY on steroids. Relevance is scored by BM25. The ELK stack is the standard production log-analytics pipeline; modern AI search adds vector retrieval on top, fusing scores for hybrid retrieval.

Inverted index Elasticsearch-এর প্রাণ। Analyzer text-কে token-এ ভাঙে। text মানুষের পড়ার জন্য, keyword exact match ও aggregation-এর জন্য। Query DSL = JSON-এ লেখা search; bool দিয়ে আপনি যেকোনো জটিল প্রশ্ন তৈরি করতে পারেন। Aggregation = SQL GROUP BY-এর শক্তিশালী রূপ। ELK = Elasticsearch + Logstash + Kibana, log analytics-এর de-facto standard। আজকের যুগে BM25 + vector mix করে hybrid retrieval — RAG ও AI search-এর ভিত্তি।

Next Module → Time-Series Databases — InfluxDB, TimescaleDB, hypertables, downsampling.

Prev: Neo4j & Cypher Next: Time-Series DB

1. Why LIKE '%bangla%' Will Never Be Enough

2. The Inverted Index — Reading Books Backwards

3. Analyzers — How Text Becomes Tokens

4. Mappings — text vs keyword

📖 text

🏷️ keyword

5. The Query DSL — JSON-Powered Search

The basic clauses

Phrase and prefix variants

6. Aggregations — SQL GROUP BY on Steroids

7. BM25, the ELK Stack, and the Vector Era

How relevance is scored — BM25

The ELK / Elastic stack — log analytics in production

Hybrid retrieval — BM25 + vector similarity

Competitors and forks

8. Glossary (শব্দকোষ)

9. Practice Problems

Summary — Module 48

1. Why `LIKE '%bangla%'` Will Never Be Enough

4. Mappings — `text` vs `keyword`

📖 `text`

🏷️ `keyword`

6. Aggregations — SQL `GROUP BY` on Steroids