Design a distributed job scheduler

With 100M pending jobs you cannot scan the table. The core structure is an index on next_run_at, so 'jobs due before now' is a cheap range scan. To spread work across scheduler instances, give every job a shard_id (hash of job_id modulo 128) and index (shard_id, next_run_at). Each scheduler instance owns a set of shards via leases in etcd and only scans those, so no two instances look at the same rows. An alternative without leases is the Postgres pattern SELECT ... FOR UPDATE SKIP LOCKED LIMIT 500, which lets many schedulers pick disjoint rows from the same table; it is simple and works well up to thousands of dispatches per second per database. Scanning slightly ahead (now + 2 s) and holding runs in an in-memory timing wheel lets the scheduler release them exactly on time instead of waiting for the next poll. For very far-future one-off jobs (reminders in 6 months), a two-tier design helps: keep them in cold storage bucketed by hour, and a loader moves the next hour's jobs into the hot table, keeping the hot index small.

The scheduler can guarantee that each scheduled time produces exactly one run record, thanks to the unique constraint on (job_id, scheduled_for). It cannot guarantee that the job's side effect happens exactly once. A worker can call the target, the target can complete the work, and the worker can crash before it records success; the reaper will then retry. Any 'exactly-once' claim here is wrong. So the contract is: dispatch is de-duplicated, execution is at-least-once, and targets must be idempotent. Pass run_id (plus attempt) as an idempotency key, and have targets store processed keys with a unique constraint. Leases with heartbeats detect dead workers; choose lease length relative to job duration (60 s lease, 20 s heartbeat). For long jobs, heartbeats carry progress so a retry can resume. A worker that loses its lease (e.g. network partition) must stop, and its completion is rejected if the lease token no longer matches (fencing). This is exactly the same model as Temporal activities and SQS visibility timeouts, which you can mention as proof that the pattern is standard.

Humans write '0 0 * * *' far more often than '17 3 * * *', so traffic spikes at midnight and on the hour; in India, many billing, settlement and reminder jobs fire at 00:00 IST. 100k runs due in the same second can overload the queue and, worse, the downstream services they call. Mitigations: let job owners declare a window ('any time between 2:00 and 2:30') and spread runs with deterministic jitter (hash(job_id) mod window); rate-limit dispatch per target service; and keep per-tenant queues so one team's 50k jobs do not delay another team's 10. Time zones need care: store the cron expression with an IANA time zone and compute the next run in that zone, then convert to UTC. India has no DST, but global customers do: a 2:30 AM job may not exist on spring-forward day and may happen twice on fall-back; document the rule (skip or run once). After an outage, recurring jobs have missed runs. A catch-up policy per job decides: run once now (most common), run every missed occurrence (for idempotent per-period work like hourly aggregation), or skip.

Polling a database index is simple, durable and easy to query for history, but adds up to one poll interval of delay and puts scan load on the DB. Delay queues (SQS delay up to 15 minutes, Redis sorted sets scored by time, Kafka with time-bucketed topics) push jobs at the right time but are limited in delay length or durability. Use the database as the source of truth and optionally a timing wheel or Redis sorted set as a near-term buffer.

For simple 'call this endpoint at this time', a custom scheduler is reasonable. For multi-step jobs with retries per step, compensation and human waits, Temporal or AWS Step Functions already solve durable state, timers and retries; building that yourself is months of work and hard to get right.

Pulling from a queue gives natural backpressure: workers take work only when they have capacity. Pushing (scheduler calls workers directly) lowers latency but needs load awareness and retries. Pull via queues is the safer default.

At 10× load the runs table grows by 200M rows/day. Partition runs by day and archive to S3, shard jobs by job_id across several Postgres clusters, and keep the hot due-job index small with the two-tier hot/cold design.

The scheduler can produce work faster than targets can absorb. Per-target concurrency limits and rate limits in the dispatcher, plus queue depth alerts, prevent the scheduler from DDoSing internal services at midnight.

Its etcd leases expire in ~10 s and other instances take over its shards. They scan from the oldest due job, so nothing is skipped; at worst some runs start ~10-15 s late. Runs already inserted are not duplicated because of the unique key.

New jobs cannot be accepted and no new runs are dispatched; running jobs continue but cannot record completion. Use a replicated Postgres with automatic failover (~30 s). After recovery, due jobs are dispatched according to their catch-up policy.

A job that always crashes its worker is retried up to the limit with backoff, then moved to dead state and alerted. Without max attempts it would loop forever and waste capacity.