Search: inverted indexes and Elasticsearch

A normal database index maps a row to its values. An inverted index flips this: it maps each term (word) to a postings list, the sorted list of document IDs that contain the term, often with positions and counts. To find 'red cotton kurta', the engine looks up three postings lists and intersects them, which is fast even over hundreds of millions of documents.

Before indexing, text goes through an analyzer: a tokenizer splits it into terms, and filters lowercase them, remove stop words, apply stemming ('running' -> 'run'), handle synonyms ('tv' = 'television') and, for Indian products, transliteration ('kurta', 'kurtha', the Devanagari word). The same analyzer runs on the query, so query terms match indexed terms.

Results are ranked by relevance. The default in Lucene, Elasticsearch and OpenSearch is BM25: a term scores higher if it appears often in the document (with diminishing returns), rarely across all documents (rare words matter more), and in a shorter field. Real products add business signals such as popularity, rating, stock, distance and personalisation on top.

Elasticsearch and OpenSearch wrap Lucene into a distributed system: an index is split into primary shards (each a Lucene index), each with replicas on other nodes. Lucene writes immutable segments and merges them in the background; new documents become searchable after a refresh, every 1 second by default, so search is near-real-time, not instantly consistent.

1. 1Model the document: denormalise what search needs into one flat document per item (product name, brand, category path, price, rating, in_stock, seller city). Search engines do not do joins well.
2. 2Define mappings: 'text' fields are analysed for full-text search; 'keyword' fields are stored exactly for filters, sorting and aggregations (brand, category, size).
3. 3Index: each document is analysed into terms, added to an in-memory buffer and the translog (for durability), then written as a new immutable segment on refresh (default 1 s). Background merges combine small segments into larger ones.
4. 4Query: the coordinating node sends the query to one copy of every shard (query phase); each shard returns its top-k IDs and scores; the coordinator merges them and fetches the full documents for the final page (fetch phase).
5. 5Filter before scoring: structured conditions (in_stock = true, price < 1000, city = Pune) are filters, which are cached as bitsets and do not affect the score; free text goes in the scored 'match' part.
6. 6Rank in stages: BM25 retrieves the top few hundred candidates cheaply, then a function score or a learning-to-rank model re-ranks them using popularity, conversion rate and personalisation.
7. 7Handle typos and prefixes: fuzzy matching (edit distance 1 or 2), edge n-grams or the completion suggester for autocomplete, and synonym lists curated from search logs.
8. 8Keep it in sync: the source of truth (Postgres, MySQL) publishes changes via an outbox or CDC (Debezium) to Kafka; an indexer consumes and upserts documents by ID, so replays are idempotent. Rebuild a full index by reindexing into a new index and switching an alias.

// Elasticsearch query: filters are cached and unscored, text is scored with BM25
POST /products/_search
{
  "query": {
    "bool": {
      "must":   [{ "multi_match": { "query": "red kurta",
                    "fields": ["title^3", "brand", "description"],
                    "fuzziness": "AUTO" } }],
      "filter": [{ "term":  { "in_stock": true } },
                 { "range": { "price": { "lte": 800 } } }]
    }
  },
  "size": 20
}