Scalability basics
In one line
A scalable system handles more load by adding resources, usually by keeping app servers stateless behind a load balancer and moving state into databases, caches and queues that can grow on their own.
What it is
Scalability is a system's ability to handle growing load (more users, requests or data) while keeping performance acceptable, by adding resources rather than redesigning.
Vertical scaling (scale up) means a bigger machine: more CPU, RAM or faster disks. It is simple but has a hard ceiling and one machine is still a single point of failure. Horizontal scaling (scale out) means more machines sharing the work. It has no hard ceiling but requires the work to be split, which is easy for stateless servers and hard for databases.
Why interviewers ask
Almost every design question eventually asks 'what happens at 10x traffic?'. Interviewers want to see that you know the standard moves (stateless services, load balancing, caching, replication, sharding, async work), in what order you would use them, and that latency and throughput are different things.
How it works
- 1Start with one server running the app and the database. It works until CPU, memory or disk becomes the bottleneck.
- 2Separate the database onto its own machine so app and database can be sized independently.
- 3Make the app tier stateless: no user sessions in local memory; keep them in a shared store like Redis or in signed tokens. Now any server can handle any request.
- 4Put a load balancer in front of several app servers. It spreads requests (round robin, least connections) and stops sending traffic to servers that fail health checks. Add auto scaling to add servers when CPU or request rate rises.
- 5Take load off the database: cache hot reads (Redis, CDN for static files), add read replicas, and move slow work such as emails or image resizing to background workers via a queue.
- 6When a single primary database can no longer handle the writes or the data size, shard it: split data across several databases by a key such as user_id.
- 7Measure throughout: throughput (requests per second) and latency percentiles (p50, p99). Scale the component that is actually saturated.
Worked example · Growing a web app from 100 to 10,000 requests per second
- At 100 req/s, one server at 30% CPU is fine. Each request takes 20 ms of CPU, so one 8-core server can do about 8 x (1000 / 20) = 400 req/s at most.
- At 1,000 req/s that server would need 2.5x its capacity. Make it stateless, put 4 servers behind a load balancer (about 1,600 req/s total, leaving headroom).
- The database now sees 1,000 queries per second, 90% of them reads. A cache with a 90% hit rate on those reads cuts database reads from 900 to 90 per second.
- At 10,000 req/s, add app servers (about 30 to 40), add 2 read replicas for cache misses, and send emails through a queue. Writes are 1,000 per second, still fine for one well-tuned primary, so sharding can wait.
Picture it
A busy restaurant can buy a bigger stove (vertical) or hire more cooks with identical stations (horizontal). Extra cooks only help if any cook can make any dish from a shared pantry, which is what making servers stateless means.
Key patterns
Stateless app tier
When: Any web or API service that must scale out
How: Keep sessions and uploads out of local memory and disk; store them in Redis, a database or object storage so any instance can serve any request.
Load balancing with health checks
When: More than one server
How: Distribute requests with round robin or least connections and remove servers that fail health checks.
Cache and replicate reads
When: Read-heavy workloads
How: Serve hot data from a cache and a CDN; send remaining reads to read replicas.
Async processing
When: Slow work that the user need not wait for
How: Put a job on a queue, return immediately, and let workers process it at their own pace.
Partition data
When: Writes or data size exceed one database
How: Shard by a key with an even spread so each database holds a slice.
Real-world use
Netflix on AWS
Runs many stateless microservices in auto scaling groups across multiple availability zones, and adds or removes instances as viewing traffic rises and falls through the day.
Stack Overflow
Known for serving a very large site from a small number of powerful, vertically scaled servers with heavy caching, showing scale-up can go a long way.
Wikipedia
Serves most page views from caching layers in front of the application servers, since articles are read far more often than edited.
AWS Elastic Load Balancing and Kubernetes
Load balancers spread traffic across healthy instances, and Kubernetes' Horizontal Pod Autoscaler adds pods when CPU or custom metrics pass a target.
In a project
An e-commerce site slows to 5-second page loads during a flash sale that brings 20x normal traffic.
- Profile first: find that the single app server is at 100% CPU and the database at 80% from repeated product-page queries.
- Move sessions from server memory to Redis so the app tier is stateless, then run it behind a load balancer with auto scaling from 3 to 30 instances.
- Cache product pages and product data in Redis with a short TTL, and serve images through a CDN.
- Send order confirmation emails and invoice generation to a queue processed by background workers.
- Load-test at 30x normal traffic before the next sale.
Impact: The next flash sale peaks at 25x traffic with p99 page load under 400 ms, database CPU stays under 50%, and the app tier scales back down afterwards so costs stay close to normal.
Trade-offs
Use it when
- Vertical: early stage, or a database that is hard to split; quickest win
- Horizontal: stateless services, high availability needs, load beyond one machine
- Auto scaling: traffic with daily or seasonal peaks
Avoid it when
- Sharding or microservices before a single database and monolith are actually saturated; it adds a lot of complexity
- Horizontal scaling of a stateful server without moving its state out first
- Scaling the wrong tier: adding app servers when the database is the bottleneck
Common mistakes
- Confusing latency (how long one request takes) with throughput (how many requests per second).
- Adding app servers while sessions still live in server memory, so users get logged out randomly.
- Forgetting that the database, not the app tier, is usually the first real bottleneck.
- Looking only at averages; p99 latency is what the slowest users feel.
- Jumping straight to sharding without trying caching, indexing and read replicas first.
Interview questions
Quick revision
- Vertical = bigger box (simple, ceiling). Horizontal = more boxes (needs stateless or partitioning).
- Stateless app servers + load balancer + auto scaling is the default web tier.
- Order of moves: cache, read replicas, async queues, then sharding.
- Latency = time per request; throughput = requests per second.
- Watch p99, not just the average.
- Remove single points of failure with redundancy across availability zones.
- Scale the tier that is actually saturated.
Check yourself
1. What must be true before you can add more app servers behind a load balancer without breaking users?
2. Which is a drawback of vertical scaling?
3. A database receives 1,000 reads per second and a cache in front of it achieves a 95% hit rate. How many reads per second still reach the database?
4. Which metric best shows what your slowest users experience?
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.