Elson TanSomeone asks my agent to summarise a 60-page PDF. The answer will not be ready before the request times out, so the work has to keep going after the user has gone. Now I need somewhere to park that job, something to pick it up, and a promise that two workers will never grab it at once.
I have built that four times. Always the same shape: a jobs table with a status column, a worker
polling it, a lock so nothing gets picked up twice, and a retry counter nobody quite trusts.
Every version leaked somewhere. The lock expires while the work is still running, or the retry counter increments on a crash before the work is recorded, and you get a double charge.
The version I stopped rewriting uses Durable Object alarms. One object per task. The alarm is the scheduler.
What you get for free
An alarm is just a timestamp you store with the object. When the time arrives, the platform wakes
the object up and calls its alarm() method. Cloudflare is specific about what that promises:
alarms run
at least once, and get retried with exponential backoff until they succeed.
Read that carefully, because three of the four things I kept hand-rolling are already in there.
The object only runs one thing at a time, so nothing can pick up the same job twice. No lock. The alarm is the schedule, so nothing sits in a loop asking an empty table whether there is work yet. And if the handler throws, the platform backs off and calls again, so the retry logic is not mine to get wrong.
import { DurableObject } from 'cloudflare:workers'
export class TaskDO extends DurableObject {
sql: SqlStorage
constructor(ctx: DurableObjectState, env: Env) {
super(ctx, env)
this.sql = ctx.storage.sql
this.sql.exec(`
CREATE TABLE IF NOT EXISTS task(
id TEXT PRIMARY KEY,
payload TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'pending',
created_at INTEGER NOT NULL
);
`)
}
async enqueue(id: string, payload: unknown, delayMs = 0) {
this.sql.exec(
`INSERT OR REPLACE INTO task (id, payload, created_at) VALUES (?, ?, ?)`,
id,
JSON.stringify(payload),
Date.now()
)
await this.ctx.storage.setAlarm(Date.now() + delayMs)
}
async alarm() {
const task = this.sql.exec(`SELECT * FROM task WHERE status = 'pending' LIMIT 1`).toArray()[0]
if (!task) return
// Record the attempt before doing the work. A crash mid-run must not
// look like an attempt that never happened.
this.sql.exec(`UPDATE task SET attempts = attempts + 1 WHERE id = ?`, task.id)
try {
await this.run(JSON.parse(task.payload as string))
this.sql.exec(`UPDATE task SET status = 'done' WHERE id = ?`, task.id)
} catch (err) {
if ((task.attempts as number) >= 4) {
this.sql.exec(`UPDATE task SET status = 'dead' WHERE id = ?`, task.id)
return
}
throw err // let the platform back off and call us again
}
}
}The line I get wrong every time I write this from memory is the attempt counter. Increment it before the work, not after. If you increment on success you cannot tell a task that has never run from one that has crashed six times, and the dead-letter branch never fires.
Why does everything fire at once?
Because you scheduled it that way. If a thousand objects all set an alarm for the top of the hour, a thousand objects wake at the top of the hour, and whatever they call downstream sees a thousand simultaneous requests.
The fix is to add randomness to the delay, which AWS documents thoroughly in Timeouts, retries and backoff with jitter. In my platform the heartbeat scan that checks agent follow-up conditions runs on a rolling window rather than a fixed clock time, for exactly this reason.
const base = 60_000
const jitter = Math.floor(Math.random() * base * 0.5)
await this.ctx.storage.setAlarm(Date.now() + base + jitter)Half the base interval is enough spread for a scan of a few thousand objects. It is not enough if your fan-out is in the millions, at which point you want a real queue in front.
One object per task, or one object per queue?
Per task. I have run both and the shared-queue version reintroduces the problem the alarm was supposed to remove.
| One object per task | One shared queue object | |
|---|---|---|
| Concurrency | One worker per task, free | Serialised across all tasks |
| Retry isolation | Per task | A poison task blocks everything behind it |
| Storage lifecycle | Delete the object when done | One database that grows forever |
| Cross-task ordering | Not available | Free |
| Cost per idle task | Zero | Zero |
The shared version looks cheaper because there is one object instead of thousands. In practice serialisation is the thing that hurts. A single task that takes ninety seconds stalls every task queued behind it, and the alarm cannot help because there is only one alarm.
Cloudflare’s own Queues product moved onto Durable Objects and reported a roughly 10x throughput improvement, which is a reasonable signal that the primitive holds up under real load.
Where this stops being the right tool
Alarms give you a timer, not a queue. Three things they do not give you:
- Ordering across tasks: each object knows only its own alarm. If task B must run after task A, you need the parent to chain them, and now you have written a workflow engine.
- Fan-in: waiting for twenty tasks to finish before running a twenty-first means somewhere has to count completions. That somewhere is another object, and it needs its own alarm as a timeout in case one task never reports.
- Backpressure: nothing stops you creating a million objects. Storage is metered per object and a lifecycle policy is your job, not the platform’s.
Ordering and fan-in are where I would reach for Workflows or a real queue. Everything below that, which in my experience is most background work, is one object and one timer.
Sources
- Durable Objects Alarms, a wake-up call for your applications, Cloudflare Blog
- Durable Objects alarms API, Cloudflare
- Durable Objects aren’t just durable, they’re fast: a 10x speedup for Cloudflare Queues, Cloudflare Blog
- Durable Objects SQL storage API, Cloudflare
- Durable Objects limits, Cloudflare
- Timeouts, retries and backoff with jitter, Amazon Builders’ Library