Design ephemeral preview environments per pull request

The simplest model gives each PR a full copy of all 40 services, plus its own Postgres, Kafka and Redis. It is perfectly isolated and easy to reason about, but at 300 PRs it means about 2,400 vCPU, thousands of pods, slow startup (every service and its dependencies must boot and become healthy), and constant flakiness because one broken unrelated service breaks every preview. A shared baseline with request routing deploys only the changed services into the PR namespace. A stable baseline environment runs the main branch of every service. Each request carries a routing key (x-preview: pr-142), set at the ingress from the hostname and propagated through every hop via trace context baggage. The service mesh (Istio, Linkerd with header routing) or an in-house router sends a request to the PR's version of a service if one exists, and to baseline otherwise. This is how tools like Signadot and Uber's and Lyft's staging approaches work. The cost is that propagation must be universal: any service that drops the header breaks routing, and async paths (Kafka) need the key in message headers with consumers that filter or re-route. Data isolation is also weaker, because baseline services share a database. Use full copies for small systems (under ~10 services) and the shared model beyond that.

Data is where previews get hard. Never copy production: it contains PII and payment data, and regulations (and common sense) forbid it. Use synthetic seed data or a sanitised snapshot built by a weekly job that anonymises names, phones and addresses and keeps referential integrity; restore it per preview in about a minute from S3. For speed, keep the database small (1-5 GB) or use copy-on-write clones (Neon branches, Aurora fast clones, ZFS/LVM snapshots) that create a writable copy in seconds regardless of size. Each PR runs its own migrations against its own database or schema, so a destructive migration in one PR cannot break others. If baseline services share a database, a PR that changes the schema of a shared table must deploy that owning service in its namespace with its own DB. Kafka is the other trap: if the PR's consumer and baseline's consumer are in the same consumer group, they steal each other's messages. Either give each preview its own topics (prefix pr-142.orders.events) on a shared cluster, or route by header: consumers ignore messages whose routing key belongs to another preview, and the PR version consumes only its own.

Previews are only used if they are fast. The 10-minute target breaks down into build, deploy and data. Build: compute affected services from the change and build only those, with a remote Docker/BuildKit layer cache so a code-only change rebuilds one layer in under a minute. Deploy: keep node capacity warm (a small buffer of idle nodes or overprovisioning pods with low priority that get preempted) so new pods are not waiting 2-3 minutes for a node. Data: snapshot restore or copy-on-write clone instead of running seed scripts. Cancel in-flight jobs when a new commit arrives. Cost: run preview node pools on spot instances (acceptable because previews are disposable), set resource requests small with ResourceQuota per namespace, scale previews to zero after 2 hours of no traffic and wake them on the next request (a KEDA HTTP add-on or an activator that holds the request while pods start), shut everything down at night and weekends, and tag every resource with pr= and team= so cost per PR is visible. Forgotten environments are the classic money leak, so the reaper is not optional.

A Terraform-created stack per PR (own RDS, MSK, cluster) is the strongest isolation and tests infrastructure changes too, but takes 15-30 minutes and costs many times more. Namespaces in a shared cluster start in minutes and cost little; isolation is softer (noisy neighbours, shared control plane). Use namespaces for app previews and a separate, rarer pipeline for infrastructure changes.

Running helm upgrade from CI is simple and fast. GitOps adds a reconcile loop: the desired state is in Git, drift is corrected, and the cluster can be rebuilt from Git. At 300 environments, ApplicationSets with a pull-request generator give automatic create and prune with less custom code.

A Postgres pod per preview is simple and isolated but uses memory for 300 instances. A schema per preview on a shared preview database server is cheaper but one heavy test can slow everyone. Copy-on-write branching services give isolation with low cost.

Hundreds of previews wake up or deploy at 10:00. Node autoscaling (Karpenter) needs minutes; keep a warm buffer, stagger wake-ups, and prioritise builds for recently active PRs.

If baseline (main branch) is broken, every preview looks broken. Deploy baseline only from green main builds, with its own smoke tests, and keep the previous version ready to roll back.

Thousands of ingress rules and certificates slow the controller. Use one wildcard certificate and one wildcard DNS record, with host-based routing done in the proxy from a map, not one ingress object per service per preview.

The job is idempotent: re-running helm/Argo sync converges to the desired state. The PR check shows the failure with a link to logs; the previous working version stays live if it existed.

Pods are rescheduled within minutes; preview databases need a persistent volume or are re-seeded automatically. Previews are explicitly best-effort; reviewers see a 'waking up' page instead of an error.

Webhooks can be lost. The nightly reaper lists open PRs from the GitHub API and deletes any namespace or cloud resource whose PR is closed, using the pr tag, and alerts when orphan count is non-zero.