Design Pastebin

Putting 10 KB to 10 MB blobs into a relational database works at first but hurts later: rows become huge, backups and replication slow down, the buffer cache fills with paste bodies instead of indexes, and storing 36 TB/year in Postgres is expensive. Object storage is built for this: 11 nines of durability, no capacity planning, cheap per GB, and lifecycle rules for tiering. The metadata row stays small (~500 B) and holds the S3 key, size and content hash, so listing, expiry and permission checks never touch S3. One trade-off to raise: S3 charges per request (about $0.005 per 1,000 PUTs), and at 10M pastes/day that is ~$50/day just for PUTs. Many pastes are under 1-2 KB, so a common optimisation is to store small bodies inline in the metadata row and only use S3 above a threshold such as 4 KB. Another is content-hash de-duplication: if two pastes have the same SHA-256, they can point to the same object (with reference counting for deletes). Mention write order: write S3 first, then metadata. If the metadata write fails you leave an orphan object, which a periodic sweep removes; the reverse order would create metadata pointing to nothing.

Expiry has two parts: making expired pastes invisible immediately, and actually deleting the bytes later. Visibility is easy: the read path checks expires_at on every request, and the CDN Cache-Control max-age is set to the remaining lifetime, so the edge never serves a paste past expiry. Deletion is a background job. With DynamoDB you get native TTL that deletes rows (within roughly 48 hours) and emits a stream event that a Lambda uses to delete the S3 object. With Postgres, a job scans an index on expires_at in small batches (DELETE ... WHERE expires_at < now() LIMIT 5000) to avoid long locks and replication lag. For S3, you can also use prefixes per expiry bucket (e.g. expires/2026-10-25/) with lifecycle rules that delete whole prefixes, which avoids millions of single DELETE calls. Burn-after-read is a special case: use a conditional update (status from active to consumed) so two simultaneous readers cannot both see it, and do not let the CDN cache it.

Anonymous paste sites are abused for phishing payloads, malware command-and-control configs and dumps of stolen data. A real design needs: rate limits per IP and per account (token bucket in Redis), CAPTCHA after suspicious bursts, and an asynchronous scanner that checks URLs against Safe Browsing, runs spam classifiers, and detects secrets with regex and entropy rules similar to GitHub secret scanning (AWS access keys start with AKIA, private keys have PEM headers). When a secret is found in a public paste, hide it and optionally notify the provider. Keep the scan asynchronous so creates stay fast, but make reads check a 'hidden' flag. Also rate limit 404s per IP, because a scraper guessing keys produces mostly 404s; that is the signal to block it.

Unlisted pastes rely on the key being secret, so random 8-character keys from a cryptographic generator are better than counters, even scrambled ones. Collisions are rare (billions used out of 218 trillion) and handled by a conditional insert and retry. Counter-based keys are a fine choice for URL shorteners where links are meant to be shared publicly.

Rendering highlighted HTML on the server costs CPU on every cache miss and makes the response bigger. Sending raw text plus a JavaScript highlighter (Prism, highlight.js) moves the work to the browser and lets the CDN cache one immutable raw object. Server-side rendering is only worth it for SEO of public pastes.

Immutable content is perfect for long CDN caching, but an owner deleting a paste, or a takedown, needs it gone now. Cache with a moderate max-age (e.g. 1 hour) plus explicit purge by URL on delete. Purge APIs complete in seconds on modern CDNs.

A paste linked from a big forum can get 10k+ reads/s. Because it is immutable, the CDN absorbs nearly all of it. Make sure view counting uses Redis INCR (or sampled CDN logs) instead of updating the DB row per view, which would create a hot row.

Billions of rows over years. Shard by key (hash), or use DynamoDB. Delete expired rows continuously, and archive 'never expire' pastes that have not been read in a year to a cheaper tier with their metadata.

The client gets an error and retries with a new key. The orphan S3 object is invisible because nothing points to it. A weekly reconciliation job lists objects older than a day without metadata and deletes them.

Reads of already-cached pastes continue from the CDN. Enable cross-region replication for content if the SLA requires it; otherwise accept degraded reads for uncached pastes and queue creates locally (or fail creates with a clear error).