Design a news aggregator (Inshorts / Google News)

Freshness depends on how fast we discover new articles, so the scheduler is the heart of ingestion. Each source has an adaptive interval: if the last few polls found new items, poll sooner (down to 1 min for top publishers); if not, back off (up to 30-60 min). Use conditional GET with ETag and Last-Modified so most polls return a cheap 304. Better than polling is push: support WebSub (PubSubHubbub) and a publisher API so large partners notify us instantly; many Indian news sites also publish Google News sitemaps that list the last 48 hours of articles, which is a dense source. Implement scheduling as a priority queue of (next_fetch_at, source_id), sharded by source so each scheduler owns a slice; Redis sorted sets work well at this scale. Politeness matters: a per-domain token bucket caps concurrent fetches so we never hurt a small publisher's server, and we honour robots.txt and crawl-delay. Freshness SLO is measured as time from publisher's published_at to visibility in our feed, tracked per source tier, with alerts when a top source has not produced anything for an unusual time, which usually means their feed format changed and extraction broke.

Two different problems. Duplicate detection finds the same text republished: agency copy from PTI or ANI appears on hundreds of sites with small edits. SimHash over shingles gives a 64-bit fingerprint, and articles within a Hamming distance of about 3 are treated as duplicates; we keep the highest-authority copy as canonical and others as alternate sources. Clustering is harder: different articles about the same event ('RBI keeps repo rate unchanged' from ten outlets in four languages). Use a multilingual sentence embedding of title and first paragraphs, then an online, incremental algorithm: for each new article, find nearest recent cluster centroids with an ANN search limited to the last 72 hours and the same broad category, and join if similarity is above a tuned threshold and they share key entities (people, organisations, places from NER). Otherwise start a new cluster. Online clustering drifts, so a periodic batch job re-merges clusters that have converged and splits ones that grew too broad. Multilingual embeddings let a Hindi and an English article join the same story, which is a big product win in India.

Inshorts-style 60-word summaries used to be written by editors; today an LLM can draft them, but news has high stakes for errors. Summarise per cluster, not per article, and feed the model 3-5 high-authority articles so the summary reflects consensus. Constrain it: a fixed word limit, no opinions, every number and name must appear in the sources. Run a cheap automatic check that extracts entities and numbers from the summary and verifies they appear in the source text; fail means regenerate or send to an editor. Sensitive categories (elections, communal incidents, deaths, legal cases) go to a human review queue before publishing. Translation to regional languages can be done by the same model, with a glossary for names. Cost is controlled because we summarise ~100k clusters a day, not 2M articles, and only regenerate when a cluster changes meaningfully (new facts, not just a new source). Track model, prompt version and source article IDs with every summary so you can audit any complaint. This is an async pipeline: the story appears with the headline first and the summary fills in seconds later.

Most of the ranking work is shared across users. A ranking job runs every 30-60 seconds and scores every active cluster on importance (number and authority of sources, how fast new sources are joining, a breaking-news signal), freshness (exponential decay with a half-life of a few hours) and engagement (click-through and dwell in the first minutes). It writes ranked lists to Redis for each (section, language, region) combination, maybe a few thousand lists of ~500 stories each. The per-user step is then cheap: merge a handful of lists, re-weight by the user's topic interests, filter read items, and apply diversity. That keeps p99 well under 150 ms and lets the system survive a 10x spike on budget day because the expensive work does not grow with users. Logged-out traffic is served from the CDN with a 60-second TTL. For breaking news, the ranking job can be triggered immediately when a cluster's source count jumps sharply, and the notification service picks the same event to send pushes.

Polling works for every publisher but costs requests and adds delay. Push (WebSub, publisher API) is instant and cheap but only big partners adopt it. Use both: push where available, adaptive polling for the long tail.

Fully personal ranking with a large model per request is expensive and adds little for news, where importance is mostly shared. Shared lists re-weighted per user give most of the benefit at a fraction of the cost and survive spikes.

Extractive summaries (pick the best 2-3 sentences) never invent facts but read poorly and cannot translate. Abstractive LLM summaries read well and can be multilingual but can hallucinate. We pick LLM plus factual checks plus human review for sensitive categories.

A loose threshold merges different stories (two separate accidents in different states); a strict one shows the same story five times. Tune per category and use entity overlap as a second check; error on the side of splitting, then batch re-merge.

A major event makes thousands of publishers post within minutes. Ingestion queues grow; clustering becomes hot on one cluster. Kafka absorbs the burst, workers autoscale on consumer lag, and updates to a single cluster are batched (update centroid and counts every few seconds rather than on every article).

Reads jump 5-10x on results day. Shared ranked lists in Redis with replicas, CDN caching for anonymous users, and degrading personalisation (serve section lists directly) keep latency stable.

Site redesigns break HTML extraction silently. Monitor per-source article yield and text length; on anomaly, fall back to the RSS description and title, and alert the on-call to update the extractor rule.

Automatic fact checks catch most errors. If a wrong summary goes live, editors can pull it with one click; the cluster falls back to the original headline. Keep audit logs of model and inputs.

Feed service keeps serving the last ranked lists from Redis (they are stale by minutes, not broken). Alert if lists are older than 5 minutes; freshness SLO is breached but availability is kept.