Report every rollback check failure and stop deploy chains on the first error
CI / ci (push) Successful in 3m12s
CI / ci (pull_request) Successful in 3m12s

The rollback check's trap now attempts its clean-up whatever happens,
keeps a failed clean-up as a nonzero status and prints "check FAILED" on
every nonzero exit. Both checks install their trap before the first
command that can fail. Deploy step 1, both rollback paths and the
script-writing block now chain with && so a failed checkout, fetch, build
or write stops the block with its status, instead of a later command
restarting the wrong release or masking the failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 18:59:17 -04:00
co-authored by Claude Opus 5.5
parent dc4b284bd8
commit cced89cc2a
+39 -16
View File
@@ -495,13 +495,16 @@ Every deploy builds the image on the host from the checked-out commit.
keeps running the previous SHA.
```sh
cd /home/overseer/mealquest
git fetch origin
git rev-parse --short HEAD # the SHA being replaced: note it for rollback
git checkout --detach origin/main # or a release tag
cd /home/overseer/mealquest &&
git fetch origin &&
git rev-parse --short HEAD && # the SHA being replaced: note it for rollback
git checkout --detach origin/main && # or a release tag
docker compose -f deploy/compose.yaml build
```
Each command runs only if the one before it succeeded. If `echo $?`
then prints anything but `0`, stop: nothing has restarted.
2. **Verify a restorable pre-deploy point — before the restart.** The
restart applies any new migrations, and an older SHA refuses the
upgraded schema, so the only rollback is restoring the database to a
@@ -517,8 +520,6 @@ set -eu # any failed step ends the verification with a nonzero status
D=/var/lib/docker/volumes/mealquest-data/_data
fail() { echo "$*" >&2; exit 1; }
point=
rm -rf /tmp/mqcheck
mkdir /tmp/mqcheck
# On the way out, run both clean-ups whatever happens, keep any failure,
# and print the restore point only if every check and clean-up succeeded.
trap 'st=$?
@@ -527,6 +528,8 @@ sudo sh -c "rm -f $D/predeploy-check.db*" || st=1
if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point"
else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi
exit "$st"' EXIT
rm -rf /tmp/mqcheck
mkdir /tmp/mqcheck
# a. Replication is running: litestream up, no errors in its log, and the
# newest LTX file (listed) is from the latest writes. Each command's own
@@ -626,9 +629,14 @@ cd /home/overseer/mealquest && (
set -eu # any failed step ends the check with a nonzero status
D=/var/lib/docker/volumes/mealquest-data/_data
P=<previous SHA>
# On the way out, remove /tmp/mqcheck whatever happens, keep any failure,
# and report every nonzero exit.
trap 'st=$?
rm -rf /tmp/mqcheck || st=1
if [ "$st" -ne 0 ]; then echo "check FAILED (status $st): stop and fix the cause" >&2; fi
exit "$st"' EXIT
rm -rf /tmp/mqcheck
mkdir /tmp/mqcheck
trap 'st=$?; rm -rf /tmp/mqcheck; [ "$st" -eq 0 ] || echo "check FAILED (status $st): stop and fix the cause" >&2; exit "$st"' EXIT
sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/mealquest.db
sudo chown -R "$USER" /tmp/mqcheck
git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only
@@ -645,8 +653,9 @@ PY
```
The check runs in a subshell that stops at the first failed step — the
snapshot or its integrity check, `git ls-tree` or the comparison — and
removes `/tmp/mqcheck` on the way out without changing the status. Read it
snapshot or its integrity check, `git ls-tree` or the comparison. On the
way out it always attempts to remove `/tmp/mqcheck`, and a failed removal
fails the check too; every nonzero exit prints `check FAILED`. Read it
like this:
- `unknown to the previous SHA: none` and `echo $?` prints `0` — no
@@ -655,18 +664,26 @@ like this:
prints `0` — the older build does not know them: the migration-crossing
path below is required.
- `check FAILED`, any other error, or a nonzero `echo $?` — stop and fix
the cause, then run the check again. Do not pick a path.
the cause, then run the check again. Do not pick a path, even if an
`unknown to the previous SHA:` line was printed before it. If the
clean-up failed, remove `/tmp/mqcheck` by hand first.
**No migration crossed** — rebuild and restart the older SHA:
```sh
git checkout --detach <previous SHA>
docker compose -f deploy/compose.yaml build
sudo systemctl restart mealquest
docker compose -f deploy/compose.yaml ps
git checkout --detach <previous SHA> &&
docker compose -f deploy/compose.yaml build &&
sudo systemctl restart mealquest &&
docker compose -f deploy/compose.yaml ps &&
curl -fsS http://10.60.1.14:8090/health
```
Each command runs only if the one before it succeeded, so the health
check answers only for the older SHA. If `echo $?` then prints anything
but `0`, the rollback is not done: the first error names the failed step,
and if it came before the restart, production is still on the release
being rolled back from.
**A migration crossed** — the older binary refuses the newer schema, so the
database is restored first and the older SHA started after. Changes made
since the pre-deploy restore point are lost; say so to the household.
@@ -674,10 +691,13 @@ since the pre-deploy restore point are lost; say so to the household.
1. Check out and build the older SHA. Nothing restarts yet.
```sh
git checkout --detach <previous SHA>
git checkout --detach <previous SHA> &&
docker compose -f deploy/compose.yaml build
```
If `echo $?` then prints anything but `0`, stop: the database has not
been touched and nothing has restarted.
2. Run "Restore the database" steps 1–4, adding
`-timestamp <pre-deploy restore point>` (recorded in step 2 of
"Deploy") to the `restore` command. Its step 5 then starts the older
@@ -838,7 +858,7 @@ running) and, with the cause understood, return the sidecars to
`/tmp` is temporary, so write the scripts before every check:
```sh
cat > /tmp/mqcount.py <<'EOF'
cat > /tmp/mqcount.py <<'EOF' &&
import sqlite3, sys, hashlib
c = sqlite3.connect(sys.argv[1])
tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
@@ -868,6 +888,9 @@ print("snapshot", dst, "integrity ok")
EOF
```
The second file is written only if the first was; `echo $?` must then
print `0` before either script is used.
With the stack stopped, copy the database and run it:
```sh