Design a chaos engineering platform

The platform must be safer than the failures it injects. Layer the protections. Before a run: policy checks refuse production runs during freezes (Big Billion Days, IPL final, month-end payment peaks), during active incidents, for services whose error budget is nearly spent, or without the owning team's approval; target locks stop two experiments overlapping on one service. Blast radius is limited by default to a few pods or a few percent of traffic, and widened only step by step (1% → 5% → one zone) after passes. During a run: automated abort rules on SLO metrics (success rate, latency, business metrics such as orders per minute), evaluated every few seconds, plus a human kill switch available to anyone on call. Faults must revert without the control plane: every fault is applied with a lease, agents renew it by heartbeat, and when the lease lapses the agent removes the fault on its own; a dead man's switch, not a request. Agents also record applied faults on local disk so a restarted agent cleans up after itself. Finally, start in staging and preview environments, run production experiments in business hours with people watching, and make every action auditable.

Different faults need different mechanisms. Process and pod faults: the control plane calls the Kubernetes API to delete pods or the agent kills a container process; this tests restarts, readiness probes and replica counts, but clean crashes are the least interesting failure. Resource faults: the agent runs stress processes in the target container's cgroup for CPU, memory or disk pressure. Network faults: the agent enters the pod's network namespace and uses tc/netem for latency, jitter, packet loss and bandwidth limits, and iptables to drop traffic to specific IPs or ports (blackholing Redis, Kafka or the LLM provider's endpoint), which tests timeouts and circuit breakers. Application-level faults: the service mesh (Envoy/Istio fault injection) returns HTTP 503 or adds delay for a percentage of requests on a route, even only for requests with a test header, which allows very precise targeting. Infrastructure faults: cloud fault services (AWS Fault Injection Service) stop instances, simulate AZ power loss or throttle APIs. Open-source tools such as Chaos Mesh and LitmusChaos implement many of these as Kubernetes custom resources; the platform can drive them and add guardrails, scheduling and reporting on top rather than reinventing the injectors.

A chaos run is an experiment, so it needs a hypothesis and a measurement that can say no. Define steady state in terms users and the business feel: checkout success rate, p99 latency, orders per minute, stream start failures, not CPU or pod restarts. The hypothesis says these stay within bounds while the fault is active. Measure a baseline before injecting; if the system is already unhealthy, cancel. The best design compares an experiment group with a control group at the same time, which removes noise from daily traffic patterns: inject the fault into a small set of pods or a small slice of traffic (routed by the mesh), keep a same-sized control slice untouched, and compare their metrics. Netflix's ChAP worked this way, routing a tiny fraction of traffic into experiment and control clusters. Abort rules come from the same metrics with tighter windows: for example abort if the experiment group's error rate exceeds control by 0.5 points for 30 s. Use SLO burn rate for longer runs: abort if the run is burning budget more than ~10× faster than allowed. Store graphs and the decision with the run, so the report says clearly whether the hypothesis held.

Chaos is valuable only if it changes the system. Every failed hypothesis produces a finding: the weakness, the evidence, the owner, and a ticket. After the fix, the same experiment re-runs to verify it and then joins an automated suite. Teams keep experiments as YAML in their service repos, reviewed like code; CI syncs them to the platform. A nightly suite runs the standard catalogue against staging or preview environments with a load test running in parallel: kill 50% of a consumer group's pods (lag rises, autoscaler reacts, nothing is lost), make Redis unreachable (rate limiter falls back to local limits), add 500 ms to Postgres (admission control sheds load instead of cascading timeouts), return 503 from the LLM provider for 10 minutes (gateway fails over), kill a Temporal worker (workflows resume), cut the API off from Kafka (outbox buffers and the relay catches up). Results feed a resilience scorecard per service. Game days add the human side: a surprise fault in a controlled window tests alerts, runbooks and on-call response, which automated runs cannot. The platform measures success by bugs found and fixed before customers found them, not by number of runs.

Existing injectors are mature; building your own netem wrappers adds risk without value. What companies usually build is the layer on top: guardrails tied to their incident and freeze systems, SLO-based abort rules, scheduling, ownership and reporting. Adopt injectors, build the control plane where the off-the-shelf one does not fit.

Staging is safe but differs from production in traffic, data and configuration, so many weaknesses only appear in production. Production chaos finds real problems but needs strong guardrails and tiny blast radius. Do both: broad automated suites in staging, small, supervised experiments in production for services that have passed in staging.

Network and node faults test real behaviour of clients, kernels and libraries but are blunt. Mesh-level faults target precisely (one route, one header, 1% of requests) but skip lower-layer behaviour such as TCP retransmits. Use both for different questions.

At hundreds of concurrent runs, querying raw metrics every 5 s overloads Prometheus and slows abort detection. Use pre-computed recording rules for SLIs, a dedicated query path, and push-based alerts for abort conditions.

If every production run needs a meeting, teams stop running them. Auto-approve experiments that passed several times in staging, stay within a small blast radius and run in business hours; require approval only for widening scope.

Network rules applied with tc/iptables persist in the pod's namespace after the agent dies. The restarted agent reads its local record of applied faults and removes them; if the node is unreachable, the control plane cordons it and the pods are rescheduled, which removes the namespace and the fault with it.

A CPU-starved node may not process the revert quickly. Faults have hard maximum durations enforced by the agent's own timer, and the control plane escalates by deleting the target pods, which always removes pod-scoped faults.

The kill switch stops all runs; the incident process takes over; the run's audit trail shows exactly what was injected and when. The post-incident review checks why guardrails did not catch it, and the blast radius policy is tightened.