Object storage and media pipelines

Object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, MinIO self-hosted) stores immutable blobs addressed by a bucket and key, such as s3://media/u/42/p/9f3a.jpg. It has a flat namespace, an HTTP API (PUT, GET, DELETE, multipart upload), effectively unlimited capacity, and very high durability: S3 is designed for 99.999999999% (eleven nines) durability by storing data redundantly across multiple availability zones.

It is cheap and scales without limit, but it is not a file system or a database: you cannot update a byte range in place (you rewrite the object), listing is slow for huge prefixes, and per-request latency is tens of milliseconds. Since December 2020, S3 gives strong read-after-write consistency for PUTs and LISTs; older material saying it is eventually consistent is out of date.

A media pipeline is everything around the blob: the client uploads the original straight to object storage, an event triggers workers that validate, scan, transcode and generate thumbnails, results are written back to object storage, the database records metadata and status, and a CDN serves the outputs to users close to them.

1. 1Client asks the API for an upload: POST /uploads with file size and type. The API checks auth, quota and type, creates a metadata row (status UPLOADING) and returns a pre-signed PUT URL valid for, say, 15 minutes.
2. 2Client uploads directly to object storage. Files over about 100 MB use multipart upload: split into parts (for example 8 to 64 MB), upload parts in parallel, retry only failed parts, then complete. On weak mobile networks, smaller parts and resumable uploads matter more than raw speed.
3. 3Object storage emits an event (S3 Event Notification to SQS, SNS or EventBridge) when the object is created. A worker picks it up; the queue absorbs bursts.
4. 4Processing workers validate the file (real type by magic bytes, size limits), virus scan documents, strip EXIF location data from photos, and produce derivatives: image thumbnails in several widths and WebP or AVIF; video transcoded with FFmpeg into an HLS or DASH ladder (for example 240p to 1080p) in 2 to 6 second segments.
5. 5Writes are idempotent: output keys are derived from the input key and preset (p/9f3a/720.webp), so a retried job overwrites the same objects rather than creating duplicates.
6. 6The worker updates the metadata row (status READY, dimensions, duration, keys of derivatives) and emits an event so feeds and notifications can show the item.
7. 7Serving: clients fetch through a CDN (CloudFront, Akamai, Cloudflare). Objects are immutable with content-hash or versioned keys, so they can be cached for a year. Private content uses signed CDN URLs or cookies with short expiry.
8. 8Lifecycle and cost: move originals to infrequent-access or archive tiers (S3 Standard-IA, Glacier) after 30 to 90 days, delete abandoned multipart uploads, and watch egress, which is often the biggest bill. A CDN in front cuts egress from the origin.

# API: authorise and hand back a pre-signed URL; the bytes never pass through the API
import boto3, uuid
s3 = boto3.client("s3")

def create_upload(user_id: int, content_type: str, size: int) -> dict:
    if content_type not in {"image/jpeg", "image/png"} or size > 20 * 1024 * 1024:
        raise ValueError("unsupported file")
    key = f"orig/{user_id}/{uuid.uuid4().hex}"
    db.insert_photo(user_id=user_id, key=key, status="UPLOADING")
    url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": "media", "Key": key, "ContentType": content_type},
        ExpiresIn=900,  # 15 minutes
    )
    return {"upload_url": url, "key": key}