Design a URL shortener (TinyURL / bit.ly)

There are three common options. (1) Hash the long URL (MD5/SHA-256) and take 7 base62 characters. It is simple, but different URLs can collide, so you need a check-and-retry loop on every write, and the same URL always maps to the same code, which breaks per-user analytics. (2) Random 7-character codes with a conditional insert. With 3.5 trillion possible codes and 6B used, the collision chance per insert is about 0.2%, so retries are rare and cheap; this is a fine answer. (3) A counter: a unique 64-bit number encoded in base62. It never collides and needs no retry. The weak point of a single counter is that it is a bottleneck and single point of failure, so each Link API server leases a block of 1M IDs from a small coordinator (Postgres row with SELECT ... FOR UPDATE, or etcd). At 40 writes/s a block lasts hours. If a server crashes, the unused part of its block is simply lost, which is fine with 3.5 trillion codes. Plain counters create guessable codes (aaaaaab, aaaaaac), which lets people scrape private links. Fix it by passing the number through a reversible scramble (for example a Feistel network or multiply by a large odd number modulo 62^7) before encoding. You keep uniqueness and lose predictability.

A 301 (Moved Permanently) tells browsers and proxies they may cache the redirect forever. Repeat visitors then never reach your servers: great for load, bad for analytics and for disabling a phishing link, because a cached 301 cannot be taken back. A 302 (or 307) is temporary: every click reaches you (or your CDN), so you can count it and you can change or kill the link at any time. bit.ly-style services use 301 for SEO reasons in some cases but most link products that sell analytics use 302. A good answer: default to 302 with a short Cache-Control (for example max-age=0 for browsers), and let the CDN cache redirects for 60 seconds using surrogate keys so you can purge one code instantly when it is disabled. Clicks served by the CDN are counted from CDN logs (Cloudflare/Akamai log push to Kafka), so analytics still work. This turns a 20k/s peak into a few thousand origin requests per second, and keeps the 'disable link' feature honest within about a minute.

The redirect must never wait for analytics. The redirect service puts the click into an in-memory buffer and a background thread sends batches to Kafka every 50-100 ms (acks=1 is acceptable here; losing a tiny fraction of clicks during a crash is a business decision you should state). If Kafka is down, the buffer is bounded and drops events rather than blocking redirects, and a counter metric tracks drops. Consumers enrich events (GeoIP country, bot detection by user agent and rate) and insert into ClickHouse in batches of 10k-100k rows, which is what ClickHouse is good at. A materialized view keeps daily rollups per code, so a stats query for one link reads a few hundred rows, not millions. Store an IP hash, not the raw IP, to reduce personal data under India's DPDP Act and GDPR. Exactly-once counting is not needed: use the event ID for de-duplication in the rollup if the product sells billing-grade click counts, otherwise at-least-once is fine.

The core query is 'get long_url by code', a single-key lookup over 6B rows. DynamoDB or Cassandra gives this with partitioning built in and predictable 5-10 ms reads at any size. Postgres can work too: 3 TB fits in a few shards, and it is a fine answer for a startup at 10M links. The KV store wins at this scale because you never have to plan a resharding. Keep the owner's 'my links' list as a secondary index or a separate Postgres table if you need rich filtering.

Counter blocks guarantee no collisions and need no read-before-write, but need a coordinator and a scramble step to avoid enumeration. Random codes need no coordinator at all and are naturally unguessable, but need a conditional write and a rare retry. Both are good answers; say which you pick and why. Pure hashing of the URL is the weakest because collisions grow with data and it cannot give two users separate links for the same URL.

Returning the same code for the same long URL saves a little storage but needs an extra index on long_url (large, hot) and mixes analytics from different users. Most products create a new code per request and only dedupe per owner if asked. Storage is cheap; clarity of analytics is worth more.

A link shared during an IPL final can get 50k clicks/s. All of them hit one Redis key and one DynamoDB partition. Fixes: let the CDN cache the redirect for 30-60 s (absorbs almost all of it), keep a small in-process LRU cache in each redirect server (1-2 s TTL), and read Redis replicas. The DB is only hit on cache misses, so it stays safe.

At 200k clicks/s, writing each event to a database row would melt it. Kafka with 64+ partitions keyed by code absorbs it, and ClickHouse batch inserts handle millions of rows/s. If cost matters, pre-aggregate per code per minute in the consumer (Flink or simple in-memory windows) and store only counts plus a sampled raw stream.

Redirects fall back to DynamoDB. At 20k/s peak that is within DynamoDB on-demand limits, but latency rises from ~2 ms to ~10 ms. Protect the DB with request coalescing (one DB read per code at a time per server) and the in-process LRU. Redirects keep working; this is why the cache is an optimisation, not the source of truth.

Each Link API server holds a leased block of 1M IDs, which at 40 writes/s across 10 servers lasts many hours. Creation keeps working until blocks run out, giving time to fail over the coordinator (Postgres replica promotion or etcd leader election). Redirects are not affected at all.

Click events buffer in memory up to a limit, then are dropped and counted. Redirects never fail because of analytics. Alert on the drop metric; analytics shows a small gap for the outage window.