Design an AI content moderation pipeline

Running a multimodal LLM on 230M items a day would be slow and very expensive, and unnecessary: most content is obviously fine. A cascade orders checks by cost. Stage 1, nearly free: hash matching for known illegal media, blocklists of URLs and scam phone patterns, rate and account signals (a 1-day-old account posting 200 comments with links). Stage 2, cheap ML: small text classifiers per policy (multilingual transformers distilled to run in a few milliseconds on CPU) and image classifiers on GPU. These produce calibrated scores. Stage 3, expensive: video frame analysis, ASR and OCR for text hidden in images and audio, then an LLM or vision-language model that reads the actual policy text and the evidence, only for the ~5% of items in the uncertain band. Stage 4, humans, for the items the models disagree on, high-reach items and appeals. Each stage can only make confident decisions; everything else moves down. Because the LLM reads the written policy, a new policy (say, deepfake rules during elections) can be enforced within hours by updating its prompt, while classifiers are retrained over the following weeks using the labels the LLM and reviewers produce.

Each classifier outputs a score, and the action depends on two thresholds per policy: above the high threshold auto-action, below the low threshold allow, in between escalate. The right thresholds differ by policy because the costs differ. For child sexual abuse material, missing one item is unacceptable, so the low threshold is very low and false positives are accepted (they go to specialised reviewers). For hate speech or harassment, wrongly removing a political opinion has real costs, so auto-removal only happens above ~95% precision and the band for humans is wider. Thresholds are set from data: take a labelled sample per language, plot precision and recall against threshold, and pick points that meet the policy's precision target and the review team's capacity (a wider band means more human work). Re-check per language: a model trained mostly on Hindi and English may be badly calibrated on Tamil or Bhojpuri, so measure each language separately and route low-resource languages to humans more often. Recall cannot be measured from removals alone; sample allowed content randomly each week and have reviewers label it to estimate what was missed.

Harm grows with views. A borderline post seen by 20 followers is a small problem; the same post shown to 2M people through the recommendation feed is a big one. So the system separates 'published' from 'distributed'. After the sync gate, content is visible to the author and followers (limited reach) within a second, which keeps creators happy. It becomes eligible for the recommendation feed only after the async checks pass, typically within minutes. Then reach milestones trigger re-evaluation with stricter thresholds and, at very high reach, a mandatory human look for sensitive categories (news, health claims, communal topics). This is a strong design point because it removes the false choice between 'check everything slowly before posting' and 'post first and hope'. The same idea drives review priority: severity × predicted reach, with SLAs from minutes (imminent harm, legal orders with hour-level deadlines under India's IT Rules) to a day (low-severity spam).

Bad actors adapt: they add noise or borders to images to beat hashes, write 'f r e e 5 0 0 0 r s' or use look-alike Unicode characters, put scam phone numbers inside images, or mass-report a rival creator. Defences: perceptual (not cryptographic) hashes with a Hamming-distance threshold; text normalisation (Unicode folding, transliteration, removing separators) before classifiers; OCR and ASR so text in images and audio is moderated like captions; report weighting by reporter accuracy history to blunt brigading; and network signals (many accounts posting near-identical content). Every decision row stores model versions, policy version, scores and who decided, so you can answer 'why was this removed?' for an appeal, a regulator or a court, and you can re-run old content when a policy changes. Reviewer decisions become labels, but reviewers also make mistakes, so QA staff re-review a sample, golden items with known answers are mixed into queues to measure each reviewer's accuracy, and disagreement rates per policy show where the policy text itself is unclear. Reviewer wellbeing is a system requirement: blur and greyscale by default, limits on exposure to graphic queues.

Checking everything deeply before publishing gives the best safety but adds minutes of delay that creators hate. Publishing instantly risks spreading harm. The middle path: a fast sync gate for clear violations, instant limited reach, and wide distribution only after async checks.

Classifiers are cheap, fast and calibrated but need labelled data per policy and retraining for new policies. LLMs read policy text and adapt quickly, handle context and sarcasm better, but cost far more per item, are slower and less consistent. Use classifiers for volume and LLMs for the uncertain band and new policies, and distil LLM labels back into classifiers.

More automation lowers cost and speeds action, but errors at scale damage trust and may be legally risky for political speech. Keep humans for high-severity, high-reach, low-confidence and appeal cases; measure automation precision continuously before widening it.

A riot or a major match can multiply reports 10×. Priority by severity × reach keeps the important items moving; temporarily tighten auto-thresholds for the affected topic (with logging), add surge reviewers, and auto-dedupe near-identical content so one decision applies to thousands of copies via hash clustering.

At 10× video volume, frame analysis dominates GPU spend. Sample frames adaptively (scene changes, not every second), skip deep analysis for trusted creators with low reach, and run audio and frame models only after a cheap first pass.

Fail to the safe intermediate state: content is published to the author only ('processing') and released when checks complete. Known-bad hash matching is a separate, very simple service with its own replicas so it is never skipped.

Shadow-run new models on live traffic first and compare decisions; roll out per policy and language with automatic rollback if removal rate or appeal-overturn rate jumps. Because decisions are logged with model versions, wrongly removed content can be found and restored in bulk.

Borderline items fall back to the human queue and limited-reach state; no item becomes widely distributed without the check it needs. Queue SLAs and surge staffing absorb the gap.