Check each deploy verification command's own status and print the restore point last
CI / ci (push) Successful in 3m13s
CI / ci (pull_request) Successful in 3m28s

- Capture the docker compose ps status before testing its output, and
  scan the Litestream log with grep's status read explicitly: 0 is a
  logged error, 1 is no match, anything else fails the verification.
- Run both clean-ups on exit whatever happens, keep any failure, and
  print the pre-deploy restore point only after every check and both
  clean-ups succeeded.
- State that the live snapshot expects the existing -wal and -shm, list
  their ownership after the snapshot, and say to stop and check the
  service state if a root-owned sidecar appears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 18:51:47 -04:00
co-authored by Claude Opus 5.5
parent 2b37d8ef26
commit dc4b284bd8
+46 -13
View File
@@ -516,15 +516,31 @@ cd /home/overseer/mealquest && (
set -eu # any failed step ends the verification with a nonzero status set -eu # any failed step ends the verification with a nonzero status
D=/var/lib/docker/volumes/mealquest-data/_data D=/var/lib/docker/volumes/mealquest-data/_data
fail() { echo "$*" >&2; exit 1; } fail() { echo "$*" >&2; exit 1; }
point=
rm -rf /tmp/mqcheck rm -rf /tmp/mqcheck
mkdir /tmp/mqcheck mkdir /tmp/mqcheck
trap 'st=$?; rm -rf /tmp/mqcheck; sudo sh -c "rm -f $D/predeploy-check.db*" || st=1; [ "$st" -eq 0 ] || echo "verification FAILED (status $st): do not restart" >&2; exit "$st"' EXIT # On the way out, run both clean-ups whatever happens, keep any failure,
# and print the restore point only if every check and clean-up succeeded.
trap 'st=$?
rm -rf /tmp/mqcheck || st=1
sudo sh -c "rm -f $D/predeploy-check.db*" || st=1
if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point"
else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi
exit "$st"' EXIT
# a. Replication is running: litestream up, no errors in its log, and the # a. Replication is running: litestream up, no errors in its log, and the
# newest LTX file (listed) is from the latest writes. # newest LTX file (listed) is from the latest writes. Each command's own
[ -n "$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream)" ] || fail "litestream is not running" # status is checked: grep exits 0 on a match, 1 on none, 2+ on an error.
log=$(docker compose -f deploy/compose.yaml logs --since 24h litestream) id=$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream) || fail "docker compose ps failed"
if printf '%s\n' "$log" | grep -i error; then fail "litestream logged errors in the last 24h"; fi [ -n "$id" ] || fail "litestream is not running"
docker compose -f deploy/compose.yaml logs --since 24h litestream > /tmp/mqcheck/litestream.log || fail "docker compose logs failed"
gs=0
grep -i error /tmp/mqcheck/litestream.log > /tmp/mqcheck/litestream.errors || gs=$?
case "$gs" in
0) cat /tmp/mqcheck/litestream.errors; fail "litestream logged errors in the last 24h";;
1) ;; # no match: no errors logged
*) fail "the litestream log scan failed (status $gs)";;
esac
docker compose -f deploy/compose.yaml exec -T litestream \ docker compose -f deploy/compose.yaml exec -T litestream \
litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
@@ -536,6 +552,7 @@ docker compose -f deploy/compose.yaml run --rm --no-deps litestream \
sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/" sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/"
sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db
sudo chown -R "$USER" /tmp/mqcheck sudo chown -R "$USER" /tmp/mqcheck
sudo ls -ln "$D" # mealquest.db-wal and -shm owned by 10001, not 0: see "Count check"
# c. Count check both: the restored replica must report integrity ok and # c. Count check both: the restored replica must report integrity ok and
# exactly the live snapshot's counts and content-sha256. # exactly the live snapshot's counts and content-sha256.
@@ -546,23 +563,28 @@ grep -qx 'integrity ok' /tmp/mqcheck/restored.txt || fail "the restored replica
diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot" diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot"
# d. The pre-deploy restore point: now, after every check passed, before # d. The pre-deploy restore point: now, after every check passed, before
# the restart. # the restart. It is printed on the way out, once the clean-ups succeed.
point=$(date -u +%Y-%m-%dT%H:%M:%SZ) point=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "pre-deploy restore point: $point"
) )
``` ```
The verification runs in a subshell that stops at the first failed The verification runs in a subshell that stops at the first failed
step and, on the way out, removes `/tmp/mqcheck` and any step. On the way out it always attempts both clean-ups — removing
`predeploy-check.db` left on the volume, keeping the status (a failed `/tmp/mqcheck` and any `predeploy-check.db` left on the volume — and a
clean-up fails it too). Read it like this: failed clean-up fails the verification too. The restore point is
printed last, and only when every check and both clean-ups succeeded.
Read it like this:
- `pre-deploy restore point:` followed by a UTC time, and `echo $?` - `pre-deploy restore point:` followed by a UTC time, and `echo $?`
prints `0` — the backup is verified. Note the time for rollback, with prints `0` — the backup is verified and cleaned up. Note the time for
the previous SHA, and go on to step 3. rollback, with the previous SHA, and go on to step 3.
- `verification FAILED`, any other error, or a nonzero `echo $?` — no - `verification FAILED`, any other error, or a nonzero `echo $?` — no
restore point was printed. If only the counts differed, a write restore point was printed. If only the counts differed, a write
landed between the restore and the snapshot: run the block again. landed between the restore and the snapshot: run the block again. If
a clean-up failed, remove what it names by hand before running the
block again.
- The `ls` in step b shows `mealquest.db-wal` or `-shm` owned by `0`
— stop, as in "Count check", even if the rest passed.
**If any check fails, do not restart.** Production stays on the **If any check fails, do not restart.** Production stays on the
previous SHA (only the new image was built). Restore replication first previous SHA (only the new image was built). Restore replication first
@@ -802,6 +824,17 @@ the database" step 4, the files are closed and checkpointed, and a plain
copy is consistent (a read-only open there would leave a `-wal` and copy is consistent (a read-only open there would leave a `-wal` and
`-shm` beside the restored file). `-shm` beside the restored file).
`mode=ro` stops writes through the snapshot's own connection, but SQLite
can still create a missing `-wal` or `-shm` beside the database, and under
`sudo` it would be owned by root. The live snapshot therefore expects the
app and Litestream to be running and holding the existing
`mealquest.db-wal` and `mealquest.db-shm`, owned by `10001`. If
`sudo ls -ln "$D"` after a live snapshot shows either sidecar owned by
`0` — the snapshot created it — stop before going further: check
`docker compose -f deploy/compose.yaml ps` (mealquest and litestream
running) and, with the cause understood, return the sidecars to
`10001:10001` before proceeding.
`/tmp` is temporary, so write the scripts before every check: `/tmp` is temporary, so write the scripts before every check:
```sh ```sh