Design AI-powered semantic search

Vector search understands meaning ('shoes for flat feet' matches 'arch support sneakers') but is weak on exact tokens: model numbers, brand names, sizes and rare words often embed close to similar but wrong items ('iPhone 15' near 'iPhone 14'). BM25 is the opposite: precise on terms, blind to synonyms and intent. Running both and merging gives clearly better recall than either alone; measure it on your own evaluation set rather than assuming. The simplest robust merge is Reciprocal Rank Fusion: each document scores Σ 1/(k + rank) over the lists it appears in, with k ≈ 60. It needs no score normalisation, which matters because BM25 scores and cosine similarities are on different scales. A learned blend (a ranker that sees both scores as features) does better once you have click data. Fusion only produces candidates; the final order comes from the learning-to-rank model, which adds signals that text similarity cannot know: conversion rate, price competitiveness, delivery speed to this pincode, return rate. Keep both retrievers in one engine so the same filters apply to both lists and you avoid a second network fan-out.

HNSW finds the nearest vectors in the whole index. If you then apply filters (in stock, deliverable to pincode 560034, price ≤ 1500, category kurtas), a restrictive filter may remove 99% of results, and the top 300 neighbours leave only 3 valid products. Three approaches. Post-filtering with a larger k (fetch 3,000, filter to 300) is simple but slow and still fails for very selective filters. Pre-filtering (compute the allowed set, then do exact search inside it) is accurate when the allowed set is small, say under 50k documents, and brute force over that set is fast. Filtered HNSW traversal, supported by modern engines (Lucene-based kNN with filters, Vespa, Qdrant, pgvector iterative scans), skips disallowed nodes during the graph walk and falls back to exact search when the filter is very selective. In practice: route by category so each category's vectors are a separate partition (most queries are category-specific after query understanding), apply the engine's filtered kNN for stock, price and region, and keep delivery serviceability coarse (region buckets) in the index with an exact check on the final 40 results.

An off-the-shelf multilingual embedding model is a fine start, but marketplace relevance is specific: 'haldi outfit' should mean yellow ethnic wear, and 'joote' should mean shoes. Fine-tune a bi-encoder (separate query and product towers sharing weights, or a dual encoder) on the marketplace's logs: positive pairs are (query, product purchased or added to cart after that query); negatives are other products in the same batch plus hard negatives (shown at a high position but not clicked, or same category but wrong attribute). Use clicks carefully: position bias means top results get clicked because they are on top, so weight purchases more than clicks and downweight clicks at position 1. Include product images through a vision encoder or captions for fashion, where text is poor. Distill to a small model so query embedding stays under ~10 ms. Most importantly, query and product embeddings must come from the same model version; mixing versions silently ruins results, so every vector carries embedding_version and the search orchestrator only queries an index whose version matches its query encoder.

Calling an LLM on every search would add 300-1,000 ms and cost more than the margin on many orders, at 30k queries/s. But search traffic is heavily skewed: the top ~1M distinct queries cover most of the volume. So run an LLM in batch every night over new and frequent queries to produce a structured interpretation: corrected spelling, transliteration to the catalog's script, category, attribute filters, price range and a few synonyms. Validate it with data: keep a rewrite only if it improves click-through in a small online test or matches the categories users actually bought from. Store results in a key-value table for sub-millisecond lookup. For unseen tail queries, use a small fine-tuned model (or the same embedding model plus a classifier head) with a strict 30 ms timeout and fall back to the raw query. Show extracted filters as removable chips; if the system wrongly read 'under 1500' as a size, the user can fix it in one tap. This pattern, an expensive model offline and cheap models online, is the most useful idea in AI-era search.

A separate vector DB (Milvus, Pinecone) may have better ANN features, but you must run two queries, apply filters consistently in two places and keep two indexes in sync. A single engine with both BM25 and HNSW (OpenSearch/Elasticsearch, Vespa) keeps filtering, freshness and fan-out in one place. Choose the separate system only if vector scale or features clearly need it.

int8 scalar quantisation cuts memory 4× with a small recall loss, recovered by re-scoring the top few hundred with full-precision vectors. Product quantisation cuts memory further (16-32×) with a larger recall loss. At 150M items, int8 plus re-scoring is the usual balance; disk-based indexes (DiskANN-style) are an option if RAM cost dominates.

A cross-encoder that reads query and product together is the most accurate text relevance model, but scoring 100 pairs per query at 30k QPS needs a large GPU fleet. A gradient-boosted ranker over cheap features is fast on CPU. Use the cross-encoder offline to label data or online only for the top 20 of tail queries.

Re-embedding on every change is wasteful because most changes are price and stock. Separate fast-changing filterable fields (partial updates) from slow-changing text (re-embed), and batch embeddings every few seconds for GPU efficiency.

HNSW search is CPU and memory-bandwidth bound. Pre-scale replicas days before, raise the candidate cache TTL so head queries are served from Redis, reduce ef_search slightly under load (trading a little recall for latency), and protect the cluster with admission control at the orchestrator.

Index RAM grows linearly. Move to more shards, product quantisation or disk-based ANN, and index only active, in-stock-recently products in the hot tier, with a cold tier for the long tail.

The orchestrator sets a 40 ms deadline on query embedding. On failure it runs BM25 with rewrites only. Users get slightly worse results, not errors. Head queries still hit the candidate cache.

A new model looks good offline but drops conversion. Because indexes are blue/green by embedding_version, rollback is switching the alias back to the old index and the old query encoder together; the old index is kept for a week after cut-over.

A seller uploads 5M products and lag grows. Separate topics or priorities for price/stock (must stay under 1 minute) and new-product embedding (can lag 15 minutes); scale embedding workers on consumer lag.