Retry a job's final status write until it lands
CI / ci (push) Successful in 2m58s
CI / ci (pull_request) Successful in 2m39s

A worker whose final SucceedJob or FailJob write failed logged it and
moved on, leaving the job running. For a suggestion batch whose own end
write had also failed, both rows stayed running, so orphan reconciliation
kept treating the batch as active and it blocked manual and replenish
batches until a restart. The runner now retries the final write with the
claim backoff until it lands; on shutdown a failed attempt is left for
the next Start to fail as interrupted.

Document that orphaned batches are reconciled each time a batch is
started, before the active batch is selected, not at boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 00:12:52 -04:00
co-authored by Claude Opus 5.5
parent a6643aaa94
commit d5cd47a775
4 changed files with 174 additions and 37 deletions
+7 -4
View File
@@ -283,10 +283,13 @@ automatic** (D-021).
`grounding`, `done`). It fails with `no_regulars`, `ai_failed` (the
skill or `normalize_recipe` failed; no Side quest is made up instead,
D-017), `search_not_configured` or `search_failed`, and the batch is set
`failed`; Side quests stored before the failure stay in the deck. At
boot, after the job runner fails the jobs a previous process left
running, every batch still `pending` or `running` with no job queued or
running is set `failed`, so it does not block the next.
`failed`; Side quests stored before the failure stay in the deck. If
recording the batch's end fails, its job fails; the job runner retries
a job's final status write with backoff until it lands. Each time a
batch is started, before the active batch is selected, every batch
still `pending` or `running` with no job queued or running (the process
stopped, or recording its end failed) is set `failed`, so it does not
block the next.
`GET /api/v1/suggestions/deck` returns `{suggestions: [{id, dinner_id,
name, photo_url, difficulty, method, minutes_total, cuisine, reason,
cited_regular_ids, status, dinner_status, decided_at, created_at}]}`: