Design object storage (like S3)

Three full copies give good durability but cost 3x storage. Erasure coding with Reed-Solomon 10+4 splits data into 10 fragments and adds 4 parity fragments; any 10 of the 14 rebuild the data. Overhead is 1.4x and it survives any 4 lost fragments, which is better than 3-way replication (survives 2). Fragments must be placed in independent failure domains: different disks, servers, racks and, for zone-level durability, 3 availability zones (for example 5+5+4 across zones, so losing a full zone leaves ≥ 10). Durability then depends on how fast we repair: the chance of losing 5 fragments of the same stripe before repair finishes. With ~1-2% annual disk failure and repair within hours, the probability is tiny, which is how providers reach '11 nines'. Costs of erasure coding: writes touch 14 nodes, small objects waste space (so small objects are packed together into stripes), and degraded reads need decoding, which costs CPU. A common pattern is to write new data replicated for speed and convert cold data to erasure coding in the background.

The data path scales by adding disks, but every request touches metadata, and LIST needs sorted keys. The object index is a range-partitioned, ordered key-value store keyed by (bucket, key, version). Each range is a Raft group of 3-5 replicas, giving strong consistency: once a PUT's metadata commit is acknowledged, every later GET and LIST sees it. This is how S3 has offered strong read-after-write consistency since 2020. Range partitioning makes prefix LIST a scan of one or a few shards, but it creates hot spots when keys are sequential (logs/2026-10-25-00001…). Shards split automatically when they get too hot or large, and the router keeps a cached map of ranges to shards. Advising customers to add randomness at the start of keys is the old fix; automatic splitting is the modern one. A per-bucket write rate around 3,500 PUT/s per prefix is the type of limit you set while a shard splits. Deletes and overwrites are metadata changes first; data is freed later by garbage collection.

A 5 TB object cannot be one HTTP request: a dropped connection would restart everything. Multipart upload splits it into parts of 5 MB to 5 GB (up to 10,000 parts). The client calls InitiateUpload and gets an upload_id, uploads parts in parallel (each part is stored as normal chunks with its own ETag), and retries only failed parts. CompleteUpload sends the ordered list of part numbers and ETags; the metadata service validates them and writes one object entry whose layout is the concatenation of the part layouts. No data is copied at completion, so it is fast. The final ETag is not a plain MD5 of the content but derived from the part MD5s, a detail worth knowing. Abandoned uploads keep using storage, so a lifecycle rule (abort incomplete uploads after N days) and GC clean them up. The same part structure makes parallel range GETs efficient for downloads.

At thousands of servers, failure is normal: several disks fail daily and bits rot silently. Each storage node verifies checksums continuously (scrubbing), reading every byte every few weeks. When a disk fails or a fragment is corrupt, the repair service finds affected stripes (through the placement map, not by scanning metadata), reads 10 healthy fragments, rebuilds the missing ones and writes them to new disks. Repair is prioritised by risk: a stripe with 3 missing fragments is fixed before one with 1 missing. Repair traffic is rate-limited so it does not hurt customer latency, but never so slow that durability drops. Garbage collection handles deletes and orphans: deleting an object removes the metadata entry and emits an event; GC later frees the fragments. Because fragments are inside append-only volume files, GC marks them dead and compacts volumes when the dead fraction passes a threshold (for example 30%), copying live data to new volumes.

Replication: simple, fast reads and writes, cheap repair, but 3x cost. Erasure coding: ~1.4x cost and higher durability, but more CPU, more network fan-out per write and expensive repair. Use replication for small and hot objects, erasure coding for large and cold data, or convert in the background.

Eventual consistency was historically easier (cached metadata, async index updates) but surprised users (read old data after write, LIST missing new keys). Strong consistency via Raft-replicated metadata adds a few ms per write and needs careful cache invalidation, but removes a whole class of client bugs. Modern designs choose strong.

Object stores use a flat key space; '/' is just a character and 'folders' are a LIST with delimiter. This makes sharding simple, but renaming a 'folder' means copying every object. A file system with real directories makes rename cheap but sharding hard. Say this when asked about rename.

Sequential keys send all writes to the last range shard. Fix with automatic shard splitting by load, and by spreading new ranges quickly; for extreme cases, hash-prefix the key internally while keeping a secondary ordered index for LIST.

Billions of 1 KB objects waste space with erasure coding and cost one metadata entry each. Pack small objects into larger stripes or store them replicated, and price per request so customers batch small files.

One object read by thousands of clients overloads its 10 storage nodes. Front ends keep a short-lived cache of hot fragments, and a CDN in front of public content takes most of the load.

Fragments are spread so each zone holds at most 4-5 of 14; the remaining zones hold ≥ 10, so reads continue (with decoding). Metadata Raft groups have replicas in 3 zones and keep a majority. Writes continue using placements in the two remaining zones until the zone returns.

Slow nodes hurt tail latency. Readers request 1-2 extra fragments (or hedge after a short delay) and use the first 10 to arrive. Nodes with high latency are marked degraded and drained.

Raft elects a new leader in a few seconds. In-flight writes either committed or not; clients retry PUTs safely because PUT of the same key is idempotent (last writer wins).