Check each deploy verification command's own status and print the restore point last
- Capture the docker compose ps status before testing its output, and scan the Litestream log with grep's status read explicitly: 0 is a logged error, 1 is no match, anything else fails the verification. - Run both clean-ups on exit whatever happens, keep any failure, and print the pre-deploy restore point only after every check and both clean-ups succeeded. - State that the live snapshot expects the existing -wal and -shm, list their ownership after the snapshot, and say to stop and check the service state if a root-owned sidecar appears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+46
-13
@@ -516,15 +516,31 @@ cd /home/overseer/mealquest && (
|
|||||||
set -eu # any failed step ends the verification with a nonzero status
|
set -eu # any failed step ends the verification with a nonzero status
|
||||||
D=/var/lib/docker/volumes/mealquest-data/_data
|
D=/var/lib/docker/volumes/mealquest-data/_data
|
||||||
fail() { echo "$*" >&2; exit 1; }
|
fail() { echo "$*" >&2; exit 1; }
|
||||||
|
point=
|
||||||
rm -rf /tmp/mqcheck
|
rm -rf /tmp/mqcheck
|
||||||
mkdir /tmp/mqcheck
|
mkdir /tmp/mqcheck
|
||||||
trap 'st=$?; rm -rf /tmp/mqcheck; sudo sh -c "rm -f $D/predeploy-check.db*" || st=1; [ "$st" -eq 0 ] || echo "verification FAILED (status $st): do not restart" >&2; exit "$st"' EXIT
|
# On the way out, run both clean-ups whatever happens, keep any failure,
|
||||||
|
# and print the restore point only if every check and clean-up succeeded.
|
||||||
|
trap 'st=$?
|
||||||
|
rm -rf /tmp/mqcheck || st=1
|
||||||
|
sudo sh -c "rm -f $D/predeploy-check.db*" || st=1
|
||||||
|
if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point"
|
||||||
|
else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi
|
||||||
|
exit "$st"' EXIT
|
||||||
|
|
||||||
# a. Replication is running: litestream up, no errors in its log, and the
|
# a. Replication is running: litestream up, no errors in its log, and the
|
||||||
# newest LTX file (listed) is from the latest writes.
|
# newest LTX file (listed) is from the latest writes. Each command's own
|
||||||
[ -n "$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream)" ] || fail "litestream is not running"
|
# status is checked: grep exits 0 on a match, 1 on none, 2+ on an error.
|
||||||
log=$(docker compose -f deploy/compose.yaml logs --since 24h litestream)
|
id=$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream) || fail "docker compose ps failed"
|
||||||
if printf '%s\n' "$log" | grep -i error; then fail "litestream logged errors in the last 24h"; fi
|
[ -n "$id" ] || fail "litestream is not running"
|
||||||
|
docker compose -f deploy/compose.yaml logs --since 24h litestream > /tmp/mqcheck/litestream.log || fail "docker compose logs failed"
|
||||||
|
gs=0
|
||||||
|
grep -i error /tmp/mqcheck/litestream.log > /tmp/mqcheck/litestream.errors || gs=$?
|
||||||
|
case "$gs" in
|
||||||
|
0) cat /tmp/mqcheck/litestream.errors; fail "litestream logged errors in the last 24h";;
|
||||||
|
1) ;; # no match: no errors logged
|
||||||
|
*) fail "the litestream log scan failed (status $gs)";;
|
||||||
|
esac
|
||||||
docker compose -f deploy/compose.yaml exec -T litestream \
|
docker compose -f deploy/compose.yaml exec -T litestream \
|
||||||
litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
|
litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
|
||||||
|
|
||||||
@@ -536,6 +552,7 @@ docker compose -f deploy/compose.yaml run --rm --no-deps litestream \
|
|||||||
sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/"
|
sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/"
|
||||||
sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db
|
sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db
|
||||||
sudo chown -R "$USER" /tmp/mqcheck
|
sudo chown -R "$USER" /tmp/mqcheck
|
||||||
|
sudo ls -ln "$D" # mealquest.db-wal and -shm owned by 10001, not 0: see "Count check"
|
||||||
|
|
||||||
# c. Count check both: the restored replica must report integrity ok and
|
# c. Count check both: the restored replica must report integrity ok and
|
||||||
# exactly the live snapshot's counts and content-sha256.
|
# exactly the live snapshot's counts and content-sha256.
|
||||||
@@ -546,23 +563,28 @@ grep -qx 'integrity ok' /tmp/mqcheck/restored.txt || fail "the restored replica
|
|||||||
diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot"
|
diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot"
|
||||||
|
|
||||||
# d. The pre-deploy restore point: now, after every check passed, before
|
# d. The pre-deploy restore point: now, after every check passed, before
|
||||||
# the restart.
|
# the restart. It is printed on the way out, once the clean-ups succeed.
|
||||||
point=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
point=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||||
echo "pre-deploy restore point: $point"
|
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
The verification runs in a subshell that stops at the first failed
|
The verification runs in a subshell that stops at the first failed
|
||||||
step and, on the way out, removes `/tmp/mqcheck` and any
|
step. On the way out it always attempts both clean-ups — removing
|
||||||
`predeploy-check.db` left on the volume, keeping the status (a failed
|
`/tmp/mqcheck` and any `predeploy-check.db` left on the volume — and a
|
||||||
clean-up fails it too). Read it like this:
|
failed clean-up fails the verification too. The restore point is
|
||||||
|
printed last, and only when every check and both clean-ups succeeded.
|
||||||
|
Read it like this:
|
||||||
|
|
||||||
- `pre-deploy restore point:` followed by a UTC time, and `echo $?`
|
- `pre-deploy restore point:` followed by a UTC time, and `echo $?`
|
||||||
prints `0` — the backup is verified. Note the time for rollback, with
|
prints `0` — the backup is verified and cleaned up. Note the time for
|
||||||
the previous SHA, and go on to step 3.
|
rollback, with the previous SHA, and go on to step 3.
|
||||||
- `verification FAILED`, any other error, or a nonzero `echo $?` — no
|
- `verification FAILED`, any other error, or a nonzero `echo $?` — no
|
||||||
restore point was printed. If only the counts differed, a write
|
restore point was printed. If only the counts differed, a write
|
||||||
landed between the restore and the snapshot: run the block again.
|
landed between the restore and the snapshot: run the block again. If
|
||||||
|
a clean-up failed, remove what it names by hand before running the
|
||||||
|
block again.
|
||||||
|
- The `ls` in step b shows `mealquest.db-wal` or `-shm` owned by `0`
|
||||||
|
— stop, as in "Count check", even if the rest passed.
|
||||||
|
|
||||||
**If any check fails, do not restart.** Production stays on the
|
**If any check fails, do not restart.** Production stays on the
|
||||||
previous SHA (only the new image was built). Restore replication first
|
previous SHA (only the new image was built). Restore replication first
|
||||||
@@ -802,6 +824,17 @@ the database" step 4, the files are closed and checkpointed, and a plain
|
|||||||
copy is consistent (a read-only open there would leave a `-wal` and
|
copy is consistent (a read-only open there would leave a `-wal` and
|
||||||
`-shm` beside the restored file).
|
`-shm` beside the restored file).
|
||||||
|
|
||||||
|
`mode=ro` stops writes through the snapshot's own connection, but SQLite
|
||||||
|
can still create a missing `-wal` or `-shm` beside the database, and under
|
||||||
|
`sudo` it would be owned by root. The live snapshot therefore expects the
|
||||||
|
app and Litestream to be running and holding the existing
|
||||||
|
`mealquest.db-wal` and `mealquest.db-shm`, owned by `10001`. If
|
||||||
|
`sudo ls -ln "$D"` after a live snapshot shows either sidecar owned by
|
||||||
|
`0` — the snapshot created it — stop before going further: check
|
||||||
|
`docker compose -f deploy/compose.yaml ps` (mealquest and litestream
|
||||||
|
running) and, with the cause understood, return the sidecars to
|
||||||
|
`10001:10001` before proceeding.
|
||||||
|
|
||||||
`/tmp` is temporary, so write the scripts before every check:
|
`/tmp` is temporary, so write the scripts before every check:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|||||||
Reference in New Issue
Block a user