A Postgres tableis my job queue

Velric generates coding missions with LLMs, plants bugs in them per candidate, and grades the results. Generation is the expensive part: a single mission takes 30 seconds or more of model time, and demand arrives in spikes, a company onboards, a cohort of candidates starts, everyone clicks generate at once. The app runs as six clustered instances plus a dispatcher on a 12-core VM, and all seven processes want to start generations. Without admission control, a spike either melts the model budget or piles up requests until everything times out.

The obvious answer is a queue service. The answer I shipped is a Postgres table, and I’d ship it again.

The table is the queue

Every generation request inserts a row into a jobs table with status pending. That’s the enqueue. The dequeue is the part people get wrong across multiple instances, and the fix is one SQL idiom: UPDATE ... SET status = ‘generating’ WHERE id = ? AND status = ‘pending’, then check how many rows came back. If zero, another instance won the race, and you walk away. No advisory locks, no SELECT FOR UPDATE SKIP LOCKED even, just an optimistic claim where the WHERE clause is the lock. Seven processes can all try to fire the same job and exactly one succeeds.

Two layers, soft and hard

Admission control runs at two layers that are configured to agree. The dispatcher enforces a soft global cap of 40 concurrent generations, counted from the table itself, and fires the oldest pending jobs first into whatever capacity remains. Downstream, the generation service holds a hard semaphore at the same limit, the seatbelt for the day the soft layer miscounts. The comment in the code says it plainly: the dispatcher is soft and fair, the semaphore is the hard seatbelt.

Above the cap sits load shedding: when queue depth hits 200 pending jobs, new requests get an honest 503 instead of a promise the system can’t keep. Shedding never touches in-flight work. And each user is capped at three active missions, so one enthusiastic client can’t occupy the whole queue.

The sweeper that saves the throughput

The failure mode that actually bites queue-in-a-table designs is silent: a worker dies mid-job, the row stays in generating forever, and every dead row permanently occupies a capacity slot. Do nothing and the system throttles itself toward zero throughput over days, with no errors anywhere. So a sweeper runs on every dispatch tick and flips any job that’s been generating longer than ten minutes to failed, freeing the slot. The dispatch tick itself runs every ten seconds with an overlap guard, because a single tick can outlast the interval, and overlapping ticks would transiently double-count in-flight capacity.

One more defensive choice I like in hindsight: if counting in-flight jobs fails, the counter reports the system as full rather than empty. When your monitoring is broken, the safe lie is “no capacity,” never “infinite capacity.”

Proving a planted bug is real

The strangest subsystem I built there is the sabotage engine: it takes a working gold solution and asks GPT-4o to break it in a specific, pedagogically useful way, producing a debugging challenge unique to each candidate. The problem is that LLMs are unreliable saboteurs, sometimes the “bug” they plant doesn’t actually break anything. The fix is executable proof: run the sabotaged code against the test suite and require every test to fail, then run the gold solution and require every test to pass. A mutation that survives both checks is a real, solvable challenge. One that doesn’t is a trick question, and trick questions are how assessment platforms lose candidates’ trust.

Execution happens in throwaway workspaces: a temp directory per run, dependencies synthesized on the fly (a package.json for Jest, a requirements.txt for pytest, and for SQL an entire disposable Postgres database created and dropped per run, tested with pgTAP), a hard SIGKILL timeout, and one invariant enforced in every result parser: zero tests executed counts as failure. An empty test file that “passes” is the oldest trick in take-home assessments, and the graders refuse to be impressed by it.

The warm pool

Thirty seconds is too long to stare at a spinner, so candidates are usually served from a pre-generated pool: least-served mission first, cloned into a per-user instance on serve, retired automatically after 50 serves (100 for technical missions) so nothing goes stale, gated on QA status so an unreviewed mission never reaches a candidate, and guarded by a domain map so a marketing mission never leaks to an engineer. Generation still happens, it just happens ahead of demand instead of inside the request.

What I’d tell you to copy

Under a few hundred jobs a minute, you don’t need queue infrastructure, you need queue discipline: an atomic claim, a global cap counted from the source of truth, honest load shedding, and above all a sweeper, because the silent failure of table-based queues is capacity leaking away one dead job at a time. Postgres was already there, already durable, already observable with plain SQL. The eight BullMQ queues Velric does run handle a different shape of work, but for admission control, the table won.