RAG: chunking, embeddings and hybrid retrieval

An LLM only knows what was in its training data. RAG gives it your data at question time: an ingestion pipeline splits documents into chunks, turns each chunk into an embedding vector, and stores the text, vector and metadata. At query time the system retrieves the best chunks for the question and asks the model to answer using only those chunks, with citations.

Retrieval quality decides answer quality. Vector (semantic) search finds chunks with similar meaning even when words differ ('refund' vs 'money back'). Keyword search (BM25 or Postgres full-text) finds exact terms that embeddings handle badly: order ids, error codes, product names, section numbers. Hybrid retrieval runs both and merges the lists, usually with Reciprocal Rank Fusion (RRF), and often a reranker model reorders the top results.

RAG is cheaper and fresher than fine-tuning for knowledge: update a document and the next answer reflects it after re-indexing, with no training run. Fine-tuning is for changing behaviour or style, not for adding facts that change.

1. 1Ingest: when a document is uploaded or changed, publish an event; an async worker extracts text (PDF, HTML, DOCX, OCR for scans), cleans it and keeps structure such as headings, tables and page numbers.
2. 2Chunk: split into pieces of roughly 300 to 800 tokens with 10 to 20% overlap, preferring natural boundaries (headings, paragraphs). Attach metadata to each chunk: tenant_id, document_id, title, section path, page, access-control list, updated_at. Prepending the document title and section heading to each chunk's text improves retrieval.
3. 3Embed: call an embedding model in batches (for example 100 chunks per request) to get vectors of a few hundred to a few thousand dimensions. Store the model name with each vector: changing the embedding model means re-embedding everything, because vectors from different models are not comparable.
4. 4Store: text, vector and metadata, either in Postgres with pgvector (a tsvector column with a GIN index for keywords plus an HNSW index for vectors) or in a search engine such as OpenSearch, Elasticsearch, Vespa or Qdrant. Enforce tenant isolation in the store itself, for example Postgres Row-Level Security, not only in application WHERE clauses.
5. 5Query understanding: optionally rewrite the user's question into a standalone search query (resolve 'it' and 'that order' from the chat history), detect language, and extract filters (product, date range).
6. 6Hybrid retrieval: run vector search (top 50) and keyword search (top 50) with the same tenant and permission filters, then merge with RRF: score = sum over lists of 1 / (60 + rank). RRF needs no score normalisation, which is why it is the common default.
7. 7Rerank: send the top 30 to 50 merged chunks to a cross-encoder reranker (a model that reads question and chunk together) and keep the best 5 to 10. This usually gives a large precision gain for tens of milliseconds.
8. 8Assemble the prompt: instructions to answer only from the provided context and say 'I don't know' otherwise, then the chunks with ids, then the question. Ask for citations by chunk id and check that cited ids were actually provided.
9. 9Keep it fresh: on document update or delete, re-chunk and replace that document's chunks atomically (delete old chunk ids, insert new) and invalidate cached answers that used them.
10. 10Evaluate: keep a set of 50 to 500 real questions labelled with the chunks that should be found; track recall@10 for retrieval and faithfulness for answers on every change to chunking, embeddings or ranking.

-- Hybrid search in Postgres + pgvector. RLS on chunks filters by tenant automatically.
-- $1 = query embedding, $2 = query text
WITH vec AS (
  SELECT id, RANK() OVER (ORDER BY embedding <=> $1) AS r
  FROM chunks
  ORDER BY embedding <=> $1
  LIMIT 50
),
kw AS (
  SELECT id, RANK() OVER (ORDER BY ts_rank(tsv, q) DESC) AS r
  FROM chunks, websearch_to_tsquery('english', $2) AS q
  WHERE tsv @@ q
  ORDER BY ts_rank(tsv, q) DESC
  LIMIT 50
)
SELECT c.id, c.document_id, c.content,
       COALESCE(1.0 / (60 + vec.r), 0) + COALESCE(1.0 / (60 + kw.r), 0) AS rrf
FROM vec
FULL OUTER JOIN kw USING (id)
JOIN chunks c USING (id)
ORDER BY rrf DESC
LIMIT 10;