Design file storage and sync (Google Drive / Dropbox)

Files are split into chunks and each chunk is named by its SHA-256 hash. This gives three wins. Resumable upload: a failed upload restarts from the last chunk, not from zero, which matters on Indian mobile networks. Delta sync: on edit only chunks with new hashes are sent. Deduplication: if the server already has a hash (from this user or anyone), the chunk is not uploaded again. Fixed-size chunks (Dropbox uses 4 MB) are simple but an insert at the start of a file shifts every boundary and changes all hashes. Content-defined chunking (a rolling hash such as Rabin or FastCDC cuts where hash mod N == 0) keeps boundaries stable around inserts, so only nearby chunks change; it is used by backup tools. For very large files that change in small places (VM images, databases) an rsync-style rolling checksum can send even less. Cross-user dedup has a privacy trap: if 'already have this hash' is visible to clients, an attacker can test whether someone stored a known file. Mitigations: dedup only within a user or team, or always require upload of small files. Chunks are encrypted at rest with per-chunk keys wrapped by a master key.

Clients should never diff their whole tree against the server. Instead, every change in a namespace appends a row to an ordered journal with a sequence number that increases by one inside that namespace. A device keeps a cursor: for each namespace it can see, the last seq it applied. Sync becomes 'give me everything after my cursor', which is cheap, ordered and resumable. The cursor is opaque to the client, so the server can change its format. Notifications carry no data, only 'namespace X has new changes'; the client then pulls. This makes push best-effort: if a notification is lost, the next long poll or a periodic check (every few minutes) still catches up. Sharding by namespace means each journal lives on one shard, so appending a journal row in the same transaction as the tree change is atomic, and there is no outbox problem for correctness. The Kafka event is only a hint for notifications and workers. Old journal entries are compacted after, say, 90 days; a device with a cursor older than that gets a 'reset' and re-lists.

Two devices can edit the same file offline. Every commit carries parent_rev, the version the device started from. The metadata service does an optimistic concurrency check: if parent_rev is not the current latest_rev, the commit is rejected. The client then keeps both versions and creates a 'conflicted copy' file, which is what Dropbox and Drive do, because the service cannot merge arbitrary binary files and silently losing someone's work is the worst outcome. Folder operations need more care. A move is a single update of parent_id in one transaction, so it is atomic. Cycles (moving folder A into its own child) are checked by walking up the parent chain before commit. Name collisions use the unique (parent_id, name) constraint; the loser gets a suffix like '(1)'. Deletes go to trash, and an edit to a file deleted on another device restores it rather than losing the edit. Clients must also handle case-insensitive file systems (Windows, macOS) where 'Report.pdf' and 'report.pdf' collide locally even if the server allows both.

Sharing a single file can be an ACL row on the file. Sharing a folder is harder: the folder may hold 100k files, and permissions must apply to all of them and to future children. The clean model is to make a shared folder its own namespace. The owner's tree and every member's tree contain a mount point that points to that namespace. Members' devices add the namespace to their cursor map and sync its journal like their own. Access checks happen at the namespace level (one lookup, cached in Redis), not per file. Removing a member deletes their mount and their cursor entry; their devices then remove local copies. Link sharing creates a token mapped to (node, role, expiry); downloads via link use short-lived pre-signed CDN URLs. Moving files between namespaces (from personal to shared) is a cross-shard operation; implement it as copy-metadata-then-delete with an idempotent workflow, because chunk data is content-addressed and does not need copying.

Fixed 4 MB chunks are simple and fast to compute and work well for most office files and photos, which are usually rewritten whole. Content-defined chunking saves more bandwidth for large files with inserts, at the cost of CPU and variable chunk sizes. Many products use fixed chunks plus special handling for large files.

Pure polling is simple but wastes requests and adds latency. Pure push needs reliable delivery and replay. The hybrid (push a small 'something changed' signal, then pull from the journal with a cursor) is robust: lost pushes only cost latency, never correctness.

Syncing full copies to every device makes offline work easy but fills small laptop and phone disks. On-demand (placeholder) files download content only when opened. Mobile apps are almost always on-demand; desktops let users choose per folder.

A company-wide folder with 1M files and 10k members means one namespace shard handles every change and fans out notifications to 10k users. Batch notifications (coalesce per namespace for ~1 s), and paginate change lists. Split very large team spaces into several namespaces.

A few power users or automated tools create millions of small files. Rate-limit per user, use bulk APIs, and move very heavy namespaces to dedicated shards.

Chunks already uploaded stay in storage. The client retries only missing chunks and then commits. Chunks never referenced by any commit are removed by garbage collection after a grace period (for example 7 days).

Devices fall back to periodic long-poll or polling of /changes. Sync slows from seconds to a minute or two but nothing is lost, because the journal is the source of truth.

A buggy or ransomware-infected client may sync mass deletes or encrypted versions. Version history and trash allow restore; detect abnormal patterns (thousands of deletes or rewrites in minutes) and offer a 'rewind folder to time T' feature.