Consensus and leader election

Many systems need exactly one of something: one primary database, one scheduler that runs a cron job, one owner of a Kafka partition, one current config. Choosing that one machine and making every other machine agree, even when networks are slow and machines crash, is the consensus problem. Leader election is its most common use.

Consensus algorithms such as Paxos, Raft and ZAB (used by ZooKeeper) work with a majority quorum. A cluster of 2f+1 nodes tolerates f failures: 3 nodes survive 1 failure, 5 survive 2. A value is decided only when a majority accepts it, and any two majorities overlap in at least one node, so two conflicting decisions cannot both win. A minority side of a network partition stops making progress instead of diverging.

In practice you use consensus as a service: etcd (behind Kubernetes), ZooKeeper (classic Kafka, HBase, Hadoop), Consul, or databases with Raft built in such as CockroachDB, TiDB, YugabyteDB and Kafka's KRaft mode. The important skills are knowing what they guarantee, how to elect a leader on top of them with leases, and how to fence a stale leader that does not know it has lost its role.

1. 1Raft splits time into numbered terms. Each node is a follower, a candidate or the leader. Followers expect regular heartbeats from the leader.
2. 2If a follower hears nothing for its election timeout (randomised, for example 150-300 ms in the Raft paper; often around 1 s in etcd deployments), it increments the term, becomes a candidate, votes for itself and asks others for votes.
3. 3Each node votes for at most one candidate per term, and only for a candidate whose log is at least as up to date as its own. A candidate with votes from a majority becomes leader. Random timeouts make split votes rare.
4. 4The leader accepts client writes, appends them to its log and replicates them to followers. An entry is committed once a majority has stored it; then it is applied to the state machine (for example a key-value map) and the client gets success.
5. 5If the leader crashes, a new election happens; the new leader is guaranteed to have all committed entries, so no acknowledged write is lost.
6. 6During a partition, only the side with a majority can elect a leader and commit. An old leader on the minority side may still think it is leader for a short time, but it cannot commit new writes. Reads from it can be stale unless reads also go through a quorum check or a lease.
7. 7Write latency = time to reach a majority. Across three regions that means one inter-region round trip per write, which is why most consensus clusters live in one region across 3 AZs.
8. 8To elect a leader for your own service, use the consensus store: create a key with a lease (etcd) or an ephemeral node (ZooKeeper). The holder is leader while it keeps renewing; if it dies, the lease expires and another instance takes over.
9. 9Fence every leader action. A paused leader (long GC pause, VM freeze) can wake up after its lease expired and keep writing. Give each new leadership a monotonically increasing fencing token (etcd revision, ZooKeeper zxid, Raft term) and make the storage reject writes with a token lower than the highest it has seen.

-- Storage-side fencing: the resource accepts work only from the newest leader.
CREATE TABLE scheduler_fence (
  name          text PRIMARY KEY,
  highest_token bigint NOT NULL
);

-- Every write by a leader first passes through the fence in the same transaction.
BEGIN;
UPDATE scheduler_fence
   SET highest_token = :my_token
 WHERE name = 'campaign-scheduler'
   AND highest_token <= :my_token;   -- 0 rows updated => a newer leader exists, abort

UPDATE campaigns
   SET state = 'enqueued'
 WHERE id = :campaign_id AND state = 'due';
COMMIT;