Bound the suggestion batch's end write like a job's final write
CI / ci (push) Successful in 2m40s
CI / ci (pull_request) Successful in 2m39s

runBatch recorded the batch's end with an uncancellable context and no
deadline, before its handler returned, so a writer connection that
stayed busy could hold the worker, and shutdown, open indefinitely.

jobs.FinalWriteContext now gives every final write one bounded attempt
context (FinalWriteTimeout, 5 seconds, not cancelled with ctx); the
runner's record and finishBatch both use it. On timeout at shutdown the
handler returns, the job and batch stay running, the next Start fails
the job as interrupted and the next StartBatch fails the orphaned batch.

TestBlockedBatchEndDoesNotHoldShutdown holds the writer while a batch
reaches finishBatch and asserts the worker stops within the bound and
the batch is reconciled after restart. TestBlockedFinalWriteDoesNotHoldShutdown
no longer sets a runner field, so it compiles against earlier heads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 00:40:52 -04:00
co-authored by Claude Opus 5.5
parent 76ec60d219
commit f95a31472b
6 changed files with 153 additions and 29 deletions
+3 -2
View File
@@ -645,7 +645,6 @@ func TestFinalWriteErrorIsRetried(t *testing.T) {
func TestBlockedFinalWriteDoesNotHoldShutdown(t *testing.T) {
st := migratedStore(t)
r := newRunner(t, st, 1)
r.recordTimeout = 100 * time.Millisecond
started, finish := make(chan struct{}), make(chan struct{})
k := register(t, r, "count", func(ctx context.Context, p countPayload, publish func(countResult) error) (countResult, error) {
close(started)
@@ -676,9 +675,11 @@ func TestBlockedFinalWriteDoesNotHoldShutdown(t *testing.T) {
stop()
close(stopped)
}()
// The final write's attempt ends after 5 seconds; well past that, the
// worker is waiting for the writer with no bound.
select {
case <-stopped:
case <-time.After(5 * time.Second):
case <-time.After(15 * time.Second):
conn.Close()
<-stopped
t.Fatal("worker waited for a held writer through shutdown")