Design an image upload and processing pipeline

Proxying 3 MB uploads through API servers wastes their memory, bandwidth and threads, and a slow 2G client can hold a connection for a minute. Pre-signed URLs let the client upload straight to object storage while the API stays small. The URL must be tightly scoped: one object key chosen by the server (never by the client), a short expiry (5-15 minutes), and a content-length range and content type in the signature (S3 POST policies support size limits). Use multipart upload for files over ~5-10 MB so a failed part is retried alone. The server still treats the object as untrusted: the content-type header can lie, so workers check magic bytes and decode with strict limits on pixel count to stop decompression bombs (a tiny PNG that expands to 50,000 × 50,000 pixels and exhausts memory). Rate-limit upload slots per user and per IP so a bot cannot fill your bucket. Incomplete multipart uploads and pending rows that never complete are cleaned up by lifecycle rules and a sweeper job.

Treat the pipeline as at-least-once: storage events can be delivered twice, workers crash mid-job, and the moderation API times out. Make every step idempotent. Output keys are deterministic (v/{image_id}/360.webp), so a retry simply overwrites with the same bytes. Status changes use conditional updates (set ready only if status is processing and version matches), so a slow duplicate worker cannot overwrite a newer result such as rejected. Use a visibility timeout longer than the worst-case job time, and extend it for large images. Split slow and fast work if needed: a quick stage that validates, strips EXIF and creates a thumbnail (so the user sees something within 2-3 seconds), and a heavier stage that produces AVIF and zoom variants. After N failed attempts, move the message to a dead-letter queue with the error, alert on DLQ size, and provide a replay tool. Workers autoscale on queue depth; during a sale, a backlog grows harmlessly instead of failing uploads.

Pre-generating a fixed set of variants makes delivery trivial (static files, perfect caching) but wastes storage on variants nobody requests and requires a backfill job whenever designers add a new size. On-demand resizing (an imgproxy or Thumbor-style service behind the CDN) creates any size from the original when first requested and lets the CDN cache it, so new sizes need no backfill; the cost is CPU on cache misses and a risk of abuse, since attackers can request endless size combinations to burn CPU and pollute the cache. Protect it by signing URLs (the app gets a template with an HMAC) or by allowing only a fixed list of widths. A common hybrid: pre-generate the 2-3 variants that serve 90% of traffic (feed and thumbnail) at upload time, and use on-demand resizing for the rest. Put an origin shield in front of the resizer so each new variant is generated once, not once per CDN edge. Format negotiation (AVIF where supported, else WebP, else JPEG) usually saves 30-50% bytes versus JPEG, which matters on Indian mobile networks.

Moderation must sit before public visibility but should not make every upload slow. Run a fast classifier (NSFW, violence, weapons, prohibited items) inside the pipeline; it typically takes tens of milliseconds on GPU or a bit more on CPU. Use two thresholds: above the high one, reject automatically; between thresholds, publish to the owner only and queue for human review; below, publish. Track the review queue size and the share of uploads it receives, because a model update that doubles borderline cases will swamp reviewers. Perceptual hashes (pHash, dHash) map visually similar images to hashes with small Hamming distance, unlike SHA-256 which changes with one pixel. Use them to block re-uploads of images already removed, and in commerce to cluster the same product photo uploaded by hundreds of resellers, which helps catalogue quality and saves storage. For scale, index hashes with LSH bands or a BK-tree, or use image embeddings in a vector index when you need semantic similarity.

Direct upload scales without app servers handling bytes and is cheaper. Proxying gives simpler validation before storage, but ties up servers on slow clients. Direct upload plus asynchronous validation is the standard choice.

Processing in the upload request gives an instant final result but makes uploads slow and fragile under load. Asynchronous processing with a fast first stage gives a quick thumbnail and absorbs spikes in the queue.

Keeping originals costs storage but lets you regenerate variants for new sizes or formats (for example when adopting AVIF). Move them to cheaper storage classes instead of deleting them.

Resizing on the phone before upload cuts bytes and upload time a lot on mobile networks, at the cost of not having the full-resolution original. For commerce and social apps that is almost always fine.

Sellers upload catalogues before a big sale. The queue absorbs the spike; scale workers on queue depth and keep AVIF generation in a lower-priority stage so thumbnails stay fast.

Thousands of edges miss at once. An origin shield with request collapsing turns them into one fetch from S3 or the resizer.

Batch requests to the model, autoscale GPU workers on queue depth, and fall back to 'visible only to owner until reviewed' rather than failing the upload.

The message reappears after the visibility timeout and another worker redoes the job; deterministic output keys make the retry safe.

Images stay in 'pending review' and are visible only to the owner; the pipeline retries with backoff. Do not fail open for public surfaces.

Multipart upload resumes the missing parts with a refreshed URL; abandoned uploads are cleaned up by lifecycle rules and a sweeper that expires pending rows after 24 hours.

After N attempts it goes to the DLQ instead of looping forever; strict decode limits and running decoders in a sandbox stop one file from taking down the fleet.