Durable workflows and job scheduling

Some work is a single short task: resize an image, send an email. A job queue (SQS, Sidekiq, Celery, a Postgres table) handles it well: enqueue, a worker picks it up, retries on failure, dead-letters after N attempts. Some work runs on a schedule: generate invoices at midnight, retry failed payouts every 15 minutes. A job scheduler stores schedules and fires jobs at the right time, exactly one fire per schedule even with many scheduler nodes.

Other work is a multi-step process that runs for minutes to weeks: onboarding (verify email, KYC check, human approval, create account), order fulfilment, loan disbursal, refunds, an AI agent calling tools. Built with cron jobs and status columns, these become fragile state machines: half-finished records, duplicated steps after retries, and nobody knows where a stuck process is.

Durable execution engines (Temporal, its predecessor Uber Cadence, AWS Step Functions, Azure Durable Functions, Restate) let you write the whole process as one function. The engine records the result of every step (activity) in an event history. If the worker crashes, another worker replays the history to rebuild the function's state and continues from the next step, without re-running completed ones. Timers ('wait 3 days') and waits for external signals ('manager approved') are durable and cost nothing while waiting.

Data pipeline orchestrators (Airflow, Dagster, Argo Workflows) are a related but different tool: they schedule DAGs of batch tasks, usually on time intervals, and are a poor fit for per-user business processes with millions of concurrent instances.

1. 1Simple jobs: put a job record in a durable queue with an id, payload and attempt count. Workers lease a job (SQS visibility timeout, or SELECT ... FOR UPDATE SKIP LOCKED in Postgres), process it, and delete or mark it done. If the worker dies, the lease expires and another worker retries. So delivery is at-least-once: jobs must be idempotent.
2. 2Retries: exponential backoff with jitter (1 s, 2 s, 4 s ... capped), a maximum attempt count, then a dead-letter queue with the error for inspection and replay.
3. 3Scheduling: store schedules in a database with next_run_at indexed. Scheduler nodes poll for due rows and claim them atomically (an UPDATE that sets the next run time, or a lease), then enqueue the job. Only the node that wins the claim fires it. Add jitter so 10,000 'midnight' jobs do not all hit the database at 00:00:00.
4. 4Durable workflows: split code into a workflow (deterministic orchestration: which step next, waits, branches) and activities (side effects: API calls, DB writes, LLM calls), each with its own timeout and retry policy.
5. 5The engine stores an event history per workflow: activity scheduled, completed with result, timer started, signal received. Workers are stateless; any worker can continue any workflow by replaying its history.
6. 6Determinism rule: workflow code must give the same decisions on replay, so it must not read the clock, random numbers or the network directly; use the SDK's time, side-effect and activity APIs.
7. 7Long waits use durable timers and signals; a workflow can sleep 48 hours or wait for an approval click without holding a thread.
8. 8Failures that cannot be retried trigger compensations (saga): refund the payment, release the seat, cancel the shipment, in reverse order.
9. 9Versioning: changing workflow code while old instances are mid-flight requires version markers (Temporal's GetVersion / patching), or the replay of old histories will not match the new code.

// Temporal workflow (Go SDK). Workflow code must be deterministic:
// no time.Now(), no random, no direct network calls. Side effects live in activities.
func OnboardingWorkflow(ctx workflow.Context, in Signup) error {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval: time.Second, BackoffCoefficient: 2, MaximumAttempts: 8,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    if err := workflow.ExecuteActivity(ctx, SendVerifyEmail, in.UserID).Get(ctx, nil); err != nil {
        return err
    }

    // Wait up to 24h for a signal; the wait costs nothing and survives restarts.
    verified := workflow.GetSignalChannel(ctx, "email-verified")
    timer := workflow.NewTimer(ctx, 24*time.Hour)
    sel := workflow.NewSelector(ctx)
    ok := false
    sel.AddReceive(verified, func(c workflow.ReceiveChannel, _ bool) { c.Receive(ctx, nil); ok = true })
    sel.AddFuture(timer, func(workflow.Future) {})
    sel.Select(ctx)
    if !ok {
        return workflow.ExecuteActivity(ctx, CloseSignup, in.UserID).Get(ctx, nil)
    }

    // Idempotency key = workflow id, so a retried activity cannot create two accounts.
    key := workflow.GetInfo(ctx).WorkflowExecution.ID
    return workflow.ExecuteActivity(ctx, ProvisionAccount, in.UserID, key).Get(ctx, nil)
}