Chunking, hybrid search and re-ranking

In the previous lesson, RAG worked on three tiny files. Real systems break in the retrieval step: the right passage exists but is not found, or it is found but buried at position 7 when you only send the top 3. This lesson covers the three tools that fix most retrieval problems: better chunking, hybrid search and re-ranking.

You need the pipeline from *Retrieval-augmented generation (RAG) end to end*.

A chunk is the unit you embed and retrieve. If chunks are too big, one vector has to represent many topics, so it matches nothing well, and you waste tokens sending unrelated text. If chunks are too small, a chunk loses its context: "It costs Rs 180" is useless without knowing what "it" is.

Analogy: when you revise from a textbook, a whole chapter is too much to find one fact, and a single sentence torn out of the page may not make sense. A paragraph or a section with its heading is usually right.

import re


def sliding_window(text, size=500, overlap=100):
    """Fixed-size chunks that share `overlap` characters with the previous one."""
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start : start + size])
        start += size - overlap
    return chunks


def split_by_headings(markdown):
    """One chunk per section; each chunk keeps its heading as context."""
    parts = re.split(r"(?m)^(?=#{1,3} )", markdown)
    return [p.strip() for p in parts if p.strip()]


def add_context(heading_path, text):
    """Prefix a chunk with where it came from, so it makes sense on its own."""
    return f"{heading_path}\n{text}"


def reciprocal_rank_fusion(rankings, k=60):
    """Merge several ranked lists of ids. Each list: best id first."""
    scores = {}
    for ranking in rankings:
        for position, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + position)
    return sorted(scores, key=scores.get, reverse=True)

Three chunkers' building blocks plus the rank-merging function used later in this lesson.

sliding_window is the baseline: size=500, overlap=100 means each chunk repeats the last 100 characters of the previous one, so a sentence cut at a boundary still appears whole in one chunk. split_by_headings is usually better for documents that have headings. add_context shows a very effective trick: prefix each chunk with its title or heading path (for example Refund policy > Digital items). The prefix is embedded with the chunk, so the chunk matches queries about its topic even when the paragraph itself never repeats the topic name.

Store facts about each chunk next to it: source file, page, section, date, category, language, who may see it. Then you can filter before searching: only the bakery category, only documents from 2026, only files this user may read. Filtering is cheap, exact and removes whole classes of wrong answers. In a vector database this is a where filter (see *Vector databases: pgvector, Chroma and friends*). Metadata also gives you citations: "policies.txt, page 2".

Embeddings capture meaning, which is great for "Do you deliver?" matching "pickup only". But they are weak on exact tokens: product codes, error codes, names, IDs and rare words. A query like BB-4471 or ORA-00942 has no "meaning" to match. Classic keyword search (BM25, the ranking used by search engines for decades) is the opposite: excellent on exact words, blind to synonyms.

Hybrid search runs both searches and merges the results. The two scores are on different scales (BM25 scores are unbounded, cosine is between -1 and 1), so adding them directly is tricky. Reciprocal Rank Fusion (RRF) avoids the problem by ignoring scores and using only rank: each list gives a document 1 / (k + rank) points, and you add up the points. A document that is near the top of both lists wins. k=60 is the usual default.

The reciprocal_rank_fusion function above is the whole algorithm. Here it is used with the rank-bm25 library (pip install rank-bm25) and the rag.py from the previous lesson.

import re

import numpy as np
from rank_bm25 import BM25Okapi

from chunking import reciprocal_rank_fusion
from rag import build_index, embed


def tokenize(text):
    return re.findall(r"\w+", text.lower())


chunks, matrix = build_index("docs")
bm25 = BM25Okapi([tokenize(c["text"]) for c in chunks])


def hybrid_search(question, k=3, pool=10):
    # 1. keyword ranking
    kw_scores = bm25.get_scores(tokenize(question))
    kw_rank = list(np.argsort(-kw_scores)[:pool])
    # 2. vector ranking
    q = embed([question], "RETRIEVAL_QUERY")[0]
    vec_rank = list(np.argsort(-(matrix @ q))[:pool])
    # 3. merge both lists
    merged = reciprocal_rank_fusion([kw_rank, vec_rank])
    return [chunks[int(i)] for i in merged[:k]]


if __name__ == "__main__":
    for hit in hybrid_search("Is order code BB-4471 ready?"):
        print(hit["source"], "|", hit["text"][:70])

Runs BM25 and vector search over the same chunks and merges them with RRF. Run it from the folder that has rag.py, chunking.py and docs/.

Many databases do this for you: PostgreSQL full-text search next to pgvector, Elasticsearch/OpenSearch, and several vector databases have a built-in hybrid mode. Know the idea so you can use whichever your stack offers.

Vector search is fast because it compares precomputed vectors, but it judges the question and the passage separately. A re-ranker reads the question and one passage together and scores how well the passage answers it. That is slower but much more accurate. So the usual pattern is two stages:

1. Retrieve a wide pool cheaply (for example the top 20 from hybrid search). 2. Re-rank those 20 and keep the best 3 to 5 for the prompt.

Re-rankers come in three flavours: a cross-encoder model (for example from the sentence-transformers library, runs locally), a hosted re-rank API, or simply an LLM asked to score passages. The last needs no new library, so it is a good first version.

You are a search relevance judge.

Question: [user question]

Below are numbered passages. For each passage, give a score from 0 to 10 for how directly it answers the question (10 = contains the answer, 0 = unrelated).

[1] [passage text]
[2] [passage text]
[3] [passage text]

Return only JSON in this shape: {"scores": [{"id": 1, "score": 7}, ...]}

What to expect: Ask for JSON (see *Structured output with JSON schemas*) so you can sort programmatically. Keep only passages with a high score, and send the top few to the answering step.