Design a document ingestion and embedding pipeline

Every stage will run more than once: workers crash, queues redeliver, users upload the same file twice, connectors report the same change repeatedly. Make each stage a pure function of its inputs and key its outputs deterministically. The raw file is stored by content hash, so identical uploads are free. Extraction output is keyed by (document, version, parser_version). Chunk ids are hash(document, version, ordinal), so re-running the chunk stage overwrites the same rows instead of adding duplicates. The embedding cache is keyed by (model, chunk text hash), so edited documents only pay for changed chunks. The important part is publishing: never mix chunks of two versions in search. Write all chunks of the new version first, invisible to queries, then flip documents.current_version in one database transaction; queries filter on the current version or join through it. Old-version chunks are deleted afterwards. If the pipeline fails halfway, the old version stays fully searchable and the new one is retried. Use one workflow per (document, version) with that pair as the workflow id, so concurrent duplicate events cannot start two pipelines for the same version, and a newer version cancels an older one still in progress.

Most RAG quality problems start here, not in the model. PDFs are a drawing format: text can come out in the wrong reading order, with headers and footers on every page, hyphenated words split, and tables flattened into nonsense. Use a layout-aware parser (for example Unstructured, Apache Tika plus a layout model, or a cloud document AI service) that returns typed blocks: title, heading, paragraph, list, table, figure caption, with page numbers. Detect scanned pages by the absence of a text layer and only OCR those; OCR for Indian scripts (Devanagari, Tamil, Bengali) needs models trained on them, and quality must be checked per language. Clean: remove repeated headers and footers, fix hyphenation, normalise Unicode. Chunk by structure: split at headings, then at paragraph boundaries to a target of ~300-800 tokens with small overlap; never split a table row from its header; prepend the heading path ('Leave Policy > Maternity > Eligibility') so each chunk carries its context. Keep the chunker versioned and measure: a change to chunking should be evaluated on a retrieval eval set (recall@k on golden questions) before re-processing millions of documents, because 'looks better' is not evidence.

Two kinds of work share the pipeline: interactive uploads where a user is waiting, and bulk work (connector backfills of 50M files, re-indexing) where throughput matters and latency does not. If they share one FIFO queue, a new enterprise tenant's backfill delays everyone's uploads by hours. Use separate queues per lane with reserved worker capacity for interactive work, and fair scheduling across tenants within the bulk lane, so one tenant cannot take all workers. Each stage scales on its own queue depth (KEDA-style), since extraction is CPU-heavy, OCR is an external API with its own quota, and embedding is limited by GPU or provider tokens per minute. The embedding stage is usually the choke point: batch chunks (100-256 per request) to get high throughput, run a token-bucket limiter shared across workers to stay under provider limits, and on 429s slow down instead of retry-storming. When the index write path is slow, upstream stages must stop pulling (bounded queues, not unbounded buffering). Poison documents (encrypted PDFs, zip bombs, 10,000-page scans, malformed files) get strict limits on size, pages and processing time, and after N failures go to a dead-letter queue with the error, and the user sees a clear 'failed: password protected' status.

Vectors from different embedding models are not comparable, so changing the model means re-embedding everything, possibly billions of chunks, while search keeps working. Do it blue-green. Create a new index (or collection) for model B. Start dual-writing: new and updated documents are embedded with both models. Run a bulk backfill in the low-priority lane reading extracted JSON from S3 (no re-parsing or OCR needed, which is why you keep it), re-chunking only if the chunker also changed, and embedding with model B at a controlled rate. Track progress per tenant. Before switching, run the retrieval eval set against both indexes and confirm B is better; then switch the read alias per tenant (or for a percentage of traffic) and watch answer-quality metrics. Keep index A for a rollback window, then delete it and stop dual-writing. Cost estimate before starting: 1B chunks × 500 tokens × $0.02/M ≈ $10,000 via API, plus index build time, which is often the slower part. The same mechanism handles chunker or parser upgrades, because document_versions records which parser, chunker and model produced each version.

A queue per stage (SQS or Kafka topics) is simple and scales well, but tracking a document's overall status, retries and cancellation is hand-built. A durable workflow per document gives status, retries, timeouts and cancellation of superseded versions for free, at the cost of running Temporal. At millions of documents per day either works; many teams use workflows for orchestration and queues for heavy fan-out.

APIs need no GPUs and are cheap for small models, but bring rate limits, data-residency questions and dependence on one vendor. Self-hosting an open embedding model gives control and predictable cost at high volume and keeps data in-region, but you run GPU capacity. For a big backfill a self-hosted fleet often wins.

Fixed-size chunking is simple and predictable but splits sentences and tables. Structure-aware chunking gives better retrieval and citations but needs a good parser and more code. Use structure-aware with a fixed-size fallback for unstructured text.

OCR becomes the biggest cost and slowest stage. Skip pages with a text layer, downscale images sensibly, cache by page hash, use a self-hosted OCR model for bulk backfills, and OCR only on demand for rarely used archives.

Heavy upserts and deletes slow vector index segments and compaction. Batch writes, publish versions in bulk, and schedule bulk re-indexing in off-peak hours with write-rate limits.

The embedding stage queue grows; interactive lane keeps its reserved quota share, bulk lane pauses. Documents stay on their previous version, so search is stale but correct. Alert on interactive lag above 2 minutes.

Because document_versions records parser_version, you can find every affected document, fix the parser, and re-run from extraction for just those, through the bulk lane.

Query-time filtering on deleted ids hides the content within seconds anyway; the reconciler removes orphans nightly, and a metric counts orphans so failures are visible.