From cced89cc2a6a5490993c8c7bafd011443bc21538 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 18:59:17 -0400 Subject: [PATCH] Report every rollback check failure and stop deploy chains on the first error The rollback check's trap now attempts its clean-up whatever happens, keeps a failed clean-up as a nonzero status and prints "check FAILED" on every nonzero exit. Both checks install their trap before the first command that can fail. Deploy step 1, both rollback paths and the script-writing block now chain with && so a failed checkout, fetch, build or write stops the block with its status, instead of a later command restarting the wrong release or masking the failure. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 55 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 16 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 351d25d..c0cdb15 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -495,13 +495,16 @@ Every deploy builds the image on the host from the checked-out commit. keeps running the previous SHA. ```sh -cd /home/overseer/mealquest -git fetch origin -git rev-parse --short HEAD # the SHA being replaced: note it for rollback -git checkout --detach origin/main # or a release tag +cd /home/overseer/mealquest && +git fetch origin && +git rev-parse --short HEAD && # the SHA being replaced: note it for rollback +git checkout --detach origin/main && # or a release tag docker compose -f deploy/compose.yaml build ``` + Each command runs only if the one before it succeeded. If `echo $?` + then prints anything but `0`, stop: nothing has restarted. + 2. **Verify a restorable pre-deploy point — before the restart.** The restart applies any new migrations, and an older SHA refuses the upgraded schema, so the only rollback is restoring the database to a @@ -517,8 +520,6 @@ set -eu # any failed step ends the verification with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data fail() { echo "$*" >&2; exit 1; } point= -rm -rf /tmp/mqcheck -mkdir /tmp/mqcheck # On the way out, run both clean-ups whatever happens, keep any failure, # and print the restore point only if every check and clean-up succeeded. trap 'st=$? @@ -527,6 +528,8 @@ sudo sh -c "rm -f $D/predeploy-check.db*" || st=1 if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi exit "$st"' EXIT +rm -rf /tmp/mqcheck +mkdir /tmp/mqcheck # a. Replication is running: litestream up, no errors in its log, and the # newest LTX file (listed) is from the latest writes. Each command's own @@ -626,9 +629,14 @@ cd /home/overseer/mealquest && ( set -eu # any failed step ends the check with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data P= +# On the way out, remove /tmp/mqcheck whatever happens, keep any failure, +# and report every nonzero exit. +trap 'st=$? +rm -rf /tmp/mqcheck || st=1 +if [ "$st" -ne 0 ]; then echo "check FAILED (status $st): stop and fix the cause" >&2; fi +exit "$st"' EXIT rm -rf /tmp/mqcheck mkdir /tmp/mqcheck -trap 'st=$?; rm -rf /tmp/mqcheck; [ "$st" -eq 0 ] || echo "check FAILED (status $st): stop and fix the cause" >&2; exit "$st"' EXIT sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/mealquest.db sudo chown -R "$USER" /tmp/mqcheck git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only @@ -645,8 +653,9 @@ PY ``` The check runs in a subshell that stops at the first failed step — the -snapshot or its integrity check, `git ls-tree` or the comparison — and -removes `/tmp/mqcheck` on the way out without changing the status. Read it +snapshot or its integrity check, `git ls-tree` or the comparison. On the +way out it always attempts to remove `/tmp/mqcheck`, and a failed removal +fails the check too; every nonzero exit prints `check FAILED`. Read it like this: - `unknown to the previous SHA: none` and `echo $?` prints `0` — no @@ -655,18 +664,26 @@ like this: prints `0` — the older build does not know them: the migration-crossing path below is required. - `check FAILED`, any other error, or a nonzero `echo $?` — stop and fix - the cause, then run the check again. Do not pick a path. + the cause, then run the check again. Do not pick a path, even if an + `unknown to the previous SHA:` line was printed before it. If the + clean-up failed, remove `/tmp/mqcheck` by hand first. **No migration crossed** — rebuild and restart the older SHA: ```sh -git checkout --detach -docker compose -f deploy/compose.yaml build -sudo systemctl restart mealquest -docker compose -f deploy/compose.yaml ps +git checkout --detach && +docker compose -f deploy/compose.yaml build && +sudo systemctl restart mealquest && +docker compose -f deploy/compose.yaml ps && curl -fsS http://10.60.1.14:8090/health ``` +Each command runs only if the one before it succeeded, so the health +check answers only for the older SHA. If `echo $?` then prints anything +but `0`, the rollback is not done: the first error names the failed step, +and if it came before the restart, production is still on the release +being rolled back from. + **A migration crossed** — the older binary refuses the newer schema, so the database is restored first and the older SHA started after. Changes made since the pre-deploy restore point are lost; say so to the household. @@ -674,10 +691,13 @@ since the pre-deploy restore point are lost; say so to the household. 1. Check out and build the older SHA. Nothing restarts yet. ```sh -git checkout --detach +git checkout --detach && docker compose -f deploy/compose.yaml build ``` + If `echo $?` then prints anything but `0`, stop: the database has not + been touched and nothing has restarted. + 2. Run "Restore the database" steps 1–4, adding `-timestamp ` (recorded in step 2 of "Deploy") to the `restore` command. Its step 5 then starts the older @@ -838,7 +858,7 @@ running) and, with the cause understood, return the sidecars to `/tmp` is temporary, so write the scripts before every check: ```sh -cat > /tmp/mqcount.py <<'EOF' +cat > /tmp/mqcount.py <<'EOF' && import sqlite3, sys, hashlib c = sqlite3.connect(sys.argv[1]) tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")] @@ -868,6 +888,9 @@ print("snapshot", dst, "integrity ok") EOF ``` +The second file is written only if the first was; `echo $?` must then +print `0` before either script is used. + With the stack stopped, copy the database and run it: ```sh