From 614c4febfa601bd049b6b81aef94fc0efbed9c4c Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 17:17:27 -0400 Subject: [PATCH 1/9] Require a verified restore point before deploy; mark rotation schedule planned RUNBOOK: the deploy steps now verify, before the restart that applies migrations, that Litestream replication is current and its replica restores to the live data, and record the pre-deploy restore point; if not, do not restart. The rollback section restores to that recorded point. ARCHITECTURE: the rotation_check schedule that calls MarkTried is planned (Feature 20); this release ships MarkTried and the rotation review read and decide APIs only. Task: 008c0a8a-2242-429b-a1fd-69346bc1d414 Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 65 ++++++++++++++++++++++++++++++++++++++++---- docs/ARCHITECTURE.md | 9 ++++-- 2 files changed, 66 insertions(+), 8 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 14adb79..2cf066f 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -491,12 +491,66 @@ Every deploy builds the image on the host from the checked-out commit. ### Deploy +1. Check out and build the release. Nothing restarts yet; production + keeps running the previous SHA. + ```sh cd /home/overseer/mealquest git fetch origin git rev-parse --short HEAD # the SHA being replaced: note it for rollback git checkout --detach origin/main # or a release tag docker compose -f deploy/compose.yaml build +``` + +2. **Verify a restorable pre-deploy point — before the restart.** The + restart applies any new migrations, and an older SHA refuses the + upgraded schema, so the only rollback is restoring the database to a + point before the restart (see "Rollback to a previous SHA"). Check that + Litestream is replicating and that its replica restores to the current + data, with the commands from "Database — Litestream", "Restore the + database" and "Count check" (`/tmp/mqcount.py`): + +```sh +D=/var/lib/docker/volumes/mealquest-data/_data + +# a. Replication is running: litestream up, no errors in its log, and the +# newest LTX file is from the latest writes. +docker compose -f deploy/compose.yaml ps litestream # up +docker compose -f deploy/compose.yaml logs --since 24h litestream | grep -i error # must print nothing +docker compose -f deploy/compose.yaml exec -T litestream \ + litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db + +# b. Restore the replica to a scratch file beside the live database (the +# live files are not touched), move it off the volume and copy the live +# database next to it. +rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck +docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ + restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db +sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/" +sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck +sudo ls -la "$D" # no predeploy-check.db left behind + +# c. Count check both. The restored copy must report integrity ok and the +# same counts and content-sha256 as the live copy. If they differ, a +# write may have landed between b and c: run b and c again. +python3 /tmp/mqcount.py /tmp/mqcheck/predeploy-check.db +python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db +rm -rf /tmp/mqcheck + +# d. The pre-deploy restore point: now, after the check, before the restart. +date -u +%Y-%m-%dT%H:%M:%SZ # note it for rollback, with the previous SHA +``` + + **If any check fails, do not restart.** Production stays on the + previous SHA (only the new image was built). Restore replication first + — `docker compose -f deploy/compose.yaml up -d --force-recreate + litestream`, then read its log for the cause (NAS reachable, key, + host key) — and repeat this step until it passes. Never deploy a + release on an unverified backup. + +3. Restart and check the deployed release. + +```sh sudo systemctl restart mealquest # applies the firewall, then down + up -d docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up, searxng healthy curl -fsS http://10.60.1.14:8090/health @@ -504,8 +558,8 @@ curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/hea git rev-parse HEAD # the deployed SHA ``` -Record the deployed SHA, the previous SHA and the date in the Maestro task -for the release. Migrations run at start-up; the service refuses to start +Record the deployed SHA, the previous SHA, the pre-deploy restore point +and the date in the Maestro task for the release. Migrations run at start-up; the service refuses to start if the database has an applied migration it does not know. If `deploy/.env.example` gained a variable, add it to `deploy/.env` before @@ -524,9 +578,10 @@ curl -fsS http://10.60.1.14:8090/health A rollback across a release that added a migration fails at start-up (the older binary refuses the newer schema). In that case restore the -database to a point before the upgrade — "Restore the database" with -`-timestamp` set just before the deploy — then start the older SHA. -Changes made since that point are lost; say so to the household. +database to the pre-deploy restore point recorded in step 2 of "Deploy" — +"Restore the database" with `-timestamp ` — then +start the older SHA. Changes made since that point are lost; say so to the +household. ## Backups diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ffee938..3d9cd7d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -370,9 +370,12 @@ automatic** (D-021). plan. A date outside the week is 400 `invalid_date`, an unknown dinner or a week with no plan 404, any other dinner 409 `dinner_not_plannable`, and a dinner already on another day of the week 409 `dinner_already_planned`. -- Rotation check (`plan.Service`, D-021). `MarkTried(date)`, which the - `rotation_check` schedule calls for the household-local day just past, - moves every `quest_log` dinner planned on that date, on a plan of any +- Rotation check (`plan.Service`, D-021). This release provides the + `MarkTried` service method and the rotation review read and decide APIs + below. The `rotation_check` schedule that is to call `MarkTried` for the + household-local day just past is planned (Feature 20, the scheduler) and + not wired yet: nothing in production calls `MarkTried`, so no Rotation + check opens automatically. `MarkTried(date)` moves every `quest_log` dinner planned on that date, on a plan of any status, to `tried` and opens a `pending` `rotation_reviews` row for it; a dinner already checked for that date is left as it is, so it is idempotent. `GET /api/v1/rotation-reviews` returns `{reviews}`, the -- 2.55.0 From a7f115f2f0ba11c0fd3db9ea758703aa6eca9248 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 17:35:44 -0400 Subject: [PATCH 2/9] Make the pre-deploy count check runnable and split rollback by migration Write /tmp/mqcount.py from the Count check section with one heredoc command, and give rollbacks that cross a migration an ordered restore-then-start procedure separate from the plain restart. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 2cf066f..eaba571 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -508,7 +508,8 @@ docker compose -f deploy/compose.yaml build point before the restart (see "Rollback to a previous SHA"). Check that Litestream is replicating and that its replica restores to the current data, with the commands from "Database — Litestream", "Restore the - database" and "Count check" (`/tmp/mqcount.py`): + database" and "Count check". First write `/tmp/mqcount.py` with the + `cat > /tmp/mqcount.py` command in "Count check", then: ```sh D=/var/lib/docker/volumes/mealquest-data/_data @@ -567,8 +568,17 @@ the restart; the service fails fast at boot naming any missing required variable ### Rollback to a previous SHA +First check whether the rollback crosses a migration: compare +`migrations/` between the two SHAs. + ```sh cd /home/overseer/mealquest +git diff --stat HEAD -- migrations/ # empty: no migration crossed +``` + +**No migration crossed** — rebuild and restart the older SHA: + +```sh git checkout --detach docker compose -f deploy/compose.yaml build sudo systemctl restart mealquest @@ -576,12 +586,21 @@ docker compose -f deploy/compose.yaml ps curl -fsS http://10.60.1.14:8090/health ``` -A rollback across a release that added a migration fails at start-up -(the older binary refuses the newer schema). In that case restore the -database to the pre-deploy restore point recorded in step 2 of "Deploy" — -"Restore the database" with `-timestamp ` — then -start the older SHA. Changes made since that point are lost; say so to the -household. +**A migration crossed** — the older binary refuses the newer schema, so the +database is restored first and the older SHA started after. Changes made +since the pre-deploy restore point are lost; say so to the household. + +1. Check out and build the older SHA. Nothing restarts yet. + +```sh +git checkout --detach +docker compose -f deploy/compose.yaml build +``` + +2. Run "Restore the database" steps 1–4, adding + `-timestamp ` (recorded in step 2 of + "Deploy") to the `restore` command. Its step 5 then starts the older + SHA and checks health. ## Backups @@ -712,7 +731,10 @@ digits; Litestream's own `_litestream*` tables are excluded) and `PRAGMA integrity_check`. Run it on a **copy** of the database, never on the live file; copy `-wal`/`-shm` along with it when the app is running. -```python +`/tmp` is temporary, so write the script before every check: + +```sh +cat > /tmp/mqcount.py <<'EOF' import sqlite3, sys, hashlib c = sqlite3.connect(sys.argv[1]) tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")] @@ -724,8 +746,11 @@ for t in tabs: h.update(repr(row).encode()) print("tables", len(tabs), "content-sha256", h.hexdigest()[:16]) print("integrity", c.execute("pragma integrity_check").fetchone()[0]) +EOF ``` +Then copy the database and run it: + ```sh rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck -- 2.55.0 From 459ee6ec282f9d69eb4fd7f9cfad271a146e9c3a Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 17:54:42 -0400 Subject: [PATCH 3/9] Decide the rollback path from applied migration versions The rollback check diffed all of migrations/, so a change to the sqlc queries in migrations/queries/ sent the operator down the restore path and discarded household changes made since the pre-deploy point. Compare the versions in the live database's schema_migrations with the top-level *.sql files at the previous SHA instead, the rule internal/store/migrate.go enforces at start-up. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index eaba571..fe8278c 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -568,14 +568,37 @@ the restart; the service fails fast at boot naming any missing required variable ### Rollback to a previous SHA -First check whether the rollback crosses a migration: compare -`migrations/` between the two SHAs. +First check whether the rollback crosses a migration, with the rule the +service applies at start-up (`internal/store/migrate.go`): the older build +refuses a database holding a version in `schema_migrations` that is not one +of its own migration files — the `*.sql` files directly in `migrations/` +(`migrations/queries/` holds sqlc queries, not migrations). Compare the +versions the live database has applied with the files the previous SHA has: ```sh cd /home/overseer/mealquest -git diff --stat HEAD -- migrations/ # empty: no migration crossed +D=/var/lib/docker/volumes/mealquest-data/_data +P= +rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck +sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck +git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only +python3 - /tmp/mqcheck/mealquest.db /tmp/mqcheck/known <<'PY' +import sqlite3, sys +applied = {v for (v,) in sqlite3.connect(sys.argv[1]).execute("select version from schema_migrations")} +known = {n[len("migrations/"):-len(".sql")] for n in open(sys.argv[2]).read().split() if n.endswith(".sql")} +if not applied or not known: + sys.exit("no versions read: check the database copy and the previous SHA, then run again") +unknown = sorted(applied - known) +print("unknown to the previous SHA:", " ".join(unknown) if unknown else "none") +PY +rm -rf /tmp/mqcheck ``` +It prints `none` when no migration crossed; otherwise it lists the versions +the older build does not know, and the migration-crossing path below is +required. If it exits with an error, stop and fix the cause — do not pick a +path. + **No migration crossed** — rebuild and restart the older SHA: ```sh -- 2.55.0 From 08f52d3f1cedd5d8314b409278353866b80af0fd Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 18:12:30 -0400 Subject: [PATCH 4/9] Make the rollback migration check fail fast and keep its status Run the check in a set -eu subshell so a failed cd, copy, git ls-tree or comparison stops it before any verdict is printed, and remove the scratch directory from an EXIT trap that preserves the check's status. Spell out the reading rule: none, listed versions, or stop on error. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index fe8278c..ba95f3f 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -576,11 +576,15 @@ of its own migration files — the `*.sql` files directly in `migrations/` versions the live database has applied with the files the previous SHA has: ```sh -cd /home/overseer/mealquest +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the check with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data P= -rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck -sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck +rm -rf /tmp/mqcheck +mkdir /tmp/mqcheck +trap 'st=$?; rm -rf /tmp/mqcheck; [ "$st" -eq 0 ] || echo "check FAILED (status $st): stop and fix the cause" >&2; exit "$st"' EXIT +sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" +sudo chown -R "$USER" /tmp/mqcheck git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only python3 - /tmp/mqcheck/mealquest.db /tmp/mqcheck/known <<'PY' import sqlite3, sys @@ -591,13 +595,20 @@ if not applied or not known: unknown = sorted(applied - known) print("unknown to the previous SHA:", " ".join(unknown) if unknown else "none") PY -rm -rf /tmp/mqcheck +) ``` -It prints `none` when no migration crossed; otherwise it lists the versions -the older build does not know, and the migration-crossing path below is -required. If it exits with an error, stop and fix the cause — do not pick a -path. +The check runs in a subshell that stops at the first failed step — the +copy, `git ls-tree` or the comparison — and removes `/tmp/mqcheck` on the +way out without changing the status. Read it like this: + +- `unknown to the previous SHA: none` and `echo $?` prints `0` — no + migration crossed: take the plain restart below. +- `unknown to the previous SHA:` followed by versions, and `echo $?` + prints `0` — the older build does not know them: the migration-crossing + path below is required. +- `check FAILED`, any other error, or a nonzero `echo $?` — stop and fix + the cause, then run the check again. Do not pick a path. **No migration crossed** — rebuild and restart the older SHA: -- 2.55.0 From 2b37d8ef261f7eab2a9baa471c69d2089f29e296 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 18:32:17 -0400 Subject: [PATCH 5/9] Take consistent SQLite snapshots in the deploy and rollback checks The pre-deploy verification and the rollback migration check copied the live mealquest.db and its WAL as separate files while the app and Litestream ran, so a checkpoint between the copies could leave an older, readable state: the rollback check could print "none" while the live database held a migration the older SHA does not know. Both now read the live database through /tmp/mqsnap.py, written once in "Count check": SQLite's online backup API from a read-only connection, then PRAGMA integrity_check, exiting nonzero unless it is ok. The pre-deploy verification runs as one fail-fast subshell (set -eu, an EXIT trap that cleans /tmp/mqcheck and the scratch restore while keeping the status), diffs the replica and live count checks, and prints the restore point only after every step has passed. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 104 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 78 insertions(+), 26 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index ba95f3f..7a56dba 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -508,40 +508,62 @@ docker compose -f deploy/compose.yaml build point before the restart (see "Rollback to a previous SHA"). Check that Litestream is replicating and that its replica restores to the current data, with the commands from "Database — Litestream", "Restore the - database" and "Count check". First write `/tmp/mqcount.py` with the - `cat > /tmp/mqcount.py` command in "Count check", then: + database" and "Count check". First write `/tmp/mqcount.py` and + `/tmp/mqsnap.py` with the `cat` command in "Count check", then: ```sh +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the verification with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data +fail() { echo "$*" >&2; exit 1; } +rm -rf /tmp/mqcheck +mkdir /tmp/mqcheck +trap 'st=$?; rm -rf /tmp/mqcheck; sudo sh -c "rm -f $D/predeploy-check.db*" || st=1; [ "$st" -eq 0 ] || echo "verification FAILED (status $st): do not restart" >&2; exit "$st"' EXIT # a. Replication is running: litestream up, no errors in its log, and the -# newest LTX file is from the latest writes. -docker compose -f deploy/compose.yaml ps litestream # up -docker compose -f deploy/compose.yaml logs --since 24h litestream | grep -i error # must print nothing +# newest LTX file (listed) is from the latest writes. +[ -n "$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream)" ] || fail "litestream is not running" +log=$(docker compose -f deploy/compose.yaml logs --since 24h litestream) +if printf '%s\n' "$log" | grep -i error; then fail "litestream logged errors in the last 24h"; fi docker compose -f deploy/compose.yaml exec -T litestream \ litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db # b. Restore the replica to a scratch file beside the live database (the -# live files are not touched), move it off the volume and copy the live -# database next to it. -rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck +# live files are not touched) and move it off the volume; then take a +# consistent snapshot of the live database next to it. docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/" -sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck -sudo ls -la "$D" # no predeploy-check.db left behind +sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db +sudo chown -R "$USER" /tmp/mqcheck -# c. Count check both. The restored copy must report integrity ok and the -# same counts and content-sha256 as the live copy. If they differ, a -# write may have landed between b and c: run b and c again. -python3 /tmp/mqcount.py /tmp/mqcheck/predeploy-check.db -python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db -rm -rf /tmp/mqcheck +# c. Count check both: the restored replica must report integrity ok and +# exactly the live snapshot's counts and content-sha256. +python3 /tmp/mqcount.py /tmp/mqcheck/predeploy-check.db > /tmp/mqcheck/restored.txt +python3 /tmp/mqcount.py /tmp/mqcheck/live.db > /tmp/mqcheck/live.txt +cat /tmp/mqcheck/restored.txt +grep -qx 'integrity ok' /tmp/mqcheck/restored.txt || fail "the restored replica failed integrity_check" +diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot" -# d. The pre-deploy restore point: now, after the check, before the restart. -date -u +%Y-%m-%dT%H:%M:%SZ # note it for rollback, with the previous SHA +# d. The pre-deploy restore point: now, after every check passed, before +# the restart. +point=$(date -u +%Y-%m-%dT%H:%M:%SZ) +echo "pre-deploy restore point: $point" +) ``` + The verification runs in a subshell that stops at the first failed + step and, on the way out, removes `/tmp/mqcheck` and any + `predeploy-check.db` left on the volume, keeping the status (a failed + clean-up fails it too). Read it like this: + + - `pre-deploy restore point:` followed by a UTC time, and `echo $?` + prints `0` — the backup is verified. Note the time for rollback, with + the previous SHA, and go on to step 3. + - `verification FAILED`, any other error, or a nonzero `echo $?` — no + restore point was printed. If only the counts differed, a write + landed between the restore and the snapshot: run the block again. + **If any check fails, do not restart.** Production stays on the previous SHA (only the new image was built). Restore replication first — `docker compose -f deploy/compose.yaml up -d --force-recreate @@ -573,7 +595,9 @@ service applies at start-up (`internal/store/migrate.go`): the older build refuses a database holding a version in `schema_migrations` that is not one of its own migration files — the `*.sql` files directly in `migrations/` (`migrations/queries/` holds sqlc queries, not migrations). Compare the -versions the live database has applied with the files the previous SHA has: +versions a consistent snapshot of the live database has applied with the +files the previous SHA has. First write `/tmp/mqsnap.py` with the `cat` +command in "Count check", then: ```sh cd /home/overseer/mealquest && ( @@ -583,7 +607,7 @@ P= rm -rf /tmp/mqcheck mkdir /tmp/mqcheck trap 'st=$?; rm -rf /tmp/mqcheck; [ "$st" -eq 0 ] || echo "check FAILED (status $st): stop and fix the cause" >&2; exit "$st"' EXIT -sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" +sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/mealquest.db sudo chown -R "$USER" /tmp/mqcheck git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only python3 - /tmp/mqcheck/mealquest.db /tmp/mqcheck/known <<'PY' @@ -591,7 +615,7 @@ import sqlite3, sys applied = {v for (v,) in sqlite3.connect(sys.argv[1]).execute("select version from schema_migrations")} known = {n[len("migrations/"):-len(".sql")] for n in open(sys.argv[2]).read().split() if n.endswith(".sql")} if not applied or not known: - sys.exit("no versions read: check the database copy and the previous SHA, then run again") + sys.exit("no versions read: check the snapshot and the previous SHA, then run again") unknown = sorted(applied - known) print("unknown to the previous SHA:", " ".join(unknown) if unknown else "none") PY @@ -599,8 +623,9 @@ PY ``` The check runs in a subshell that stops at the first failed step — the -copy, `git ls-tree` or the comparison — and removes `/tmp/mqcheck` on the -way out without changing the status. Read it like this: +snapshot or its integrity check, `git ls-tree` or the comparison — and +removes `/tmp/mqcheck` on the way out without changing the status. Read it +like this: - `unknown to the previous SHA: none` and `echo $?` prints `0` — no migration crossed: take the plain restart below. @@ -763,9 +788,21 @@ To restore to a point in time, add `-timestamp 2026-10-05T23:00:00Z` (or application tables, a SHA-256 over every application row (first 16 hex digits; Litestream's own `_litestream*` tables are excluded) and `PRAGMA integrity_check`. Run it on a **copy** of the database, never on the -live file; copy `-wal`/`-shm` along with it when the app is running. +live file. -`/tmp` is temporary, so write the script before every check: +While the app or Litestream has the database open, the copy is a snapshot +from `/tmp/mqsnap.py`: it opens the live database read-only, copies it +with SQLite's online backup API (one consistent read, whatever is still in +the WAL included) and exits nonzero unless the snapshot's +`PRAGMA integrity_check` is `ok`. Never copy `mealquest.db` and its `-wal` +as files while they are open: a checkpoint between the two copies leaves +an older state that still reads cleanly. "Deploy" step 2 and "Rollback to +a previous SHA" use the snapshot. With the stack stopped, as in "Restore +the database" step 4, the files are closed and checkpointed, and a plain +copy is consistent (a read-only open there would leave a `-wal` and +`-shm` beside the restored file). + +`/tmp` is temporary, so write the scripts before every check: ```sh cat > /tmp/mqcount.py <<'EOF' @@ -781,9 +818,24 @@ for t in tabs: print("tables", len(tabs), "content-sha256", h.hexdigest()[:16]) print("integrity", c.execute("pragma integrity_check").fetchone()[0]) EOF +cat > /tmp/mqsnap.py <<'EOF' +import os, sqlite3, sys +src, dst = sys.argv[1], sys.argv[2] +if os.path.exists(dst): + sys.exit(f"{dst} already exists") +s = sqlite3.connect(f"file:{src}?mode=ro", uri=True) +d = sqlite3.connect(dst) +s.backup(d) +s.close() +result = d.execute("pragma integrity_check").fetchall() +d.close() +if result != [("ok",)]: + sys.exit(f"snapshot integrity_check failed: {result}") +print("snapshot", dst, "integrity ok") +EOF ``` -Then copy the database and run it: +With the stack stopped, copy the database and run it: ```sh rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck -- 2.55.0 From dc4b284bd8f6a66c8a0319e993bef7fab848e6fa Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 18:51:47 -0400 Subject: [PATCH 6/9] Check each deploy verification command's own status and print the restore point last - Capture the docker compose ps status before testing its output, and scan the Litestream log with grep's status read explicitly: 0 is a logged error, 1 is no match, anything else fails the verification. - Run both clean-ups on exit whatever happens, keep any failure, and print the pre-deploy restore point only after every check and both clean-ups succeeded. - State that the live snapshot expects the existing -wal and -shm, list their ownership after the snapshot, and say to stop and check the service state if a root-owned sidecar appears. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 59 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 46 insertions(+), 13 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 7a56dba..351d25d 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -516,15 +516,31 @@ cd /home/overseer/mealquest && ( set -eu # any failed step ends the verification with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data fail() { echo "$*" >&2; exit 1; } +point= rm -rf /tmp/mqcheck mkdir /tmp/mqcheck -trap 'st=$?; rm -rf /tmp/mqcheck; sudo sh -c "rm -f $D/predeploy-check.db*" || st=1; [ "$st" -eq 0 ] || echo "verification FAILED (status $st): do not restart" >&2; exit "$st"' EXIT +# On the way out, run both clean-ups whatever happens, keep any failure, +# and print the restore point only if every check and clean-up succeeded. +trap 'st=$? +rm -rf /tmp/mqcheck || st=1 +sudo sh -c "rm -f $D/predeploy-check.db*" || st=1 +if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" +else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi +exit "$st"' EXIT # a. Replication is running: litestream up, no errors in its log, and the -# newest LTX file (listed) is from the latest writes. -[ -n "$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream)" ] || fail "litestream is not running" -log=$(docker compose -f deploy/compose.yaml logs --since 24h litestream) -if printf '%s\n' "$log" | grep -i error; then fail "litestream logged errors in the last 24h"; fi +# newest LTX file (listed) is from the latest writes. Each command's own +# status is checked: grep exits 0 on a match, 1 on none, 2+ on an error. +id=$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream) || fail "docker compose ps failed" +[ -n "$id" ] || fail "litestream is not running" +docker compose -f deploy/compose.yaml logs --since 24h litestream > /tmp/mqcheck/litestream.log || fail "docker compose logs failed" +gs=0 +grep -i error /tmp/mqcheck/litestream.log > /tmp/mqcheck/litestream.errors || gs=$? +case "$gs" in + 0) cat /tmp/mqcheck/litestream.errors; fail "litestream logged errors in the last 24h";; + 1) ;; # no match: no errors logged + *) fail "the litestream log scan failed (status $gs)";; +esac docker compose -f deploy/compose.yaml exec -T litestream \ litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db @@ -536,6 +552,7 @@ docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/" sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db sudo chown -R "$USER" /tmp/mqcheck +sudo ls -ln "$D" # mealquest.db-wal and -shm owned by 10001, not 0: see "Count check" # c. Count check both: the restored replica must report integrity ok and # exactly the live snapshot's counts and content-sha256. @@ -546,23 +563,28 @@ grep -qx 'integrity ok' /tmp/mqcheck/restored.txt || fail "the restored replica diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot" # d. The pre-deploy restore point: now, after every check passed, before -# the restart. +# the restart. It is printed on the way out, once the clean-ups succeed. point=$(date -u +%Y-%m-%dT%H:%M:%SZ) -echo "pre-deploy restore point: $point" ) ``` The verification runs in a subshell that stops at the first failed - step and, on the way out, removes `/tmp/mqcheck` and any - `predeploy-check.db` left on the volume, keeping the status (a failed - clean-up fails it too). Read it like this: + step. On the way out it always attempts both clean-ups — removing + `/tmp/mqcheck` and any `predeploy-check.db` left on the volume — and a + failed clean-up fails the verification too. The restore point is + printed last, and only when every check and both clean-ups succeeded. + Read it like this: - `pre-deploy restore point:` followed by a UTC time, and `echo $?` - prints `0` — the backup is verified. Note the time for rollback, with - the previous SHA, and go on to step 3. + prints `0` — the backup is verified and cleaned up. Note the time for + rollback, with the previous SHA, and go on to step 3. - `verification FAILED`, any other error, or a nonzero `echo $?` — no restore point was printed. If only the counts differed, a write - landed between the restore and the snapshot: run the block again. + landed between the restore and the snapshot: run the block again. If + a clean-up failed, remove what it names by hand before running the + block again. + - The `ls` in step b shows `mealquest.db-wal` or `-shm` owned by `0` + — stop, as in "Count check", even if the rest passed. **If any check fails, do not restart.** Production stays on the previous SHA (only the new image was built). Restore replication first @@ -802,6 +824,17 @@ the database" step 4, the files are closed and checkpointed, and a plain copy is consistent (a read-only open there would leave a `-wal` and `-shm` beside the restored file). +`mode=ro` stops writes through the snapshot's own connection, but SQLite +can still create a missing `-wal` or `-shm` beside the database, and under +`sudo` it would be owned by root. The live snapshot therefore expects the +app and Litestream to be running and holding the existing +`mealquest.db-wal` and `mealquest.db-shm`, owned by `10001`. If +`sudo ls -ln "$D"` after a live snapshot shows either sidecar owned by +`0` — the snapshot created it — stop before going further: check +`docker compose -f deploy/compose.yaml ps` (mealquest and litestream +running) and, with the cause understood, return the sidecars to +`10001:10001` before proceeding. + `/tmp` is temporary, so write the scripts before every check: ```sh -- 2.55.0 From cced89cc2a6a5490993c8c7bafd011443bc21538 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 18:59:17 -0400 Subject: [PATCH 7/9] Report every rollback check failure and stop deploy chains on the first error The rollback check's trap now attempts its clean-up whatever happens, keeps a failed clean-up as a nonzero status and prints "check FAILED" on every nonzero exit. Both checks install their trap before the first command that can fail. Deploy step 1, both rollback paths and the script-writing block now chain with && so a failed checkout, fetch, build or write stops the block with its status, instead of a later command restarting the wrong release or masking the failure. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 55 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 16 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 351d25d..c0cdb15 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -495,13 +495,16 @@ Every deploy builds the image on the host from the checked-out commit. keeps running the previous SHA. ```sh -cd /home/overseer/mealquest -git fetch origin -git rev-parse --short HEAD # the SHA being replaced: note it for rollback -git checkout --detach origin/main # or a release tag +cd /home/overseer/mealquest && +git fetch origin && +git rev-parse --short HEAD && # the SHA being replaced: note it for rollback +git checkout --detach origin/main && # or a release tag docker compose -f deploy/compose.yaml build ``` + Each command runs only if the one before it succeeded. If `echo $?` + then prints anything but `0`, stop: nothing has restarted. + 2. **Verify a restorable pre-deploy point — before the restart.** The restart applies any new migrations, and an older SHA refuses the upgraded schema, so the only rollback is restoring the database to a @@ -517,8 +520,6 @@ set -eu # any failed step ends the verification with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data fail() { echo "$*" >&2; exit 1; } point= -rm -rf /tmp/mqcheck -mkdir /tmp/mqcheck # On the way out, run both clean-ups whatever happens, keep any failure, # and print the restore point only if every check and clean-up succeeded. trap 'st=$? @@ -527,6 +528,8 @@ sudo sh -c "rm -f $D/predeploy-check.db*" || st=1 if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi exit "$st"' EXIT +rm -rf /tmp/mqcheck +mkdir /tmp/mqcheck # a. Replication is running: litestream up, no errors in its log, and the # newest LTX file (listed) is from the latest writes. Each command's own @@ -626,9 +629,14 @@ cd /home/overseer/mealquest && ( set -eu # any failed step ends the check with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data P= +# On the way out, remove /tmp/mqcheck whatever happens, keep any failure, +# and report every nonzero exit. +trap 'st=$? +rm -rf /tmp/mqcheck || st=1 +if [ "$st" -ne 0 ]; then echo "check FAILED (status $st): stop and fix the cause" >&2; fi +exit "$st"' EXIT rm -rf /tmp/mqcheck mkdir /tmp/mqcheck -trap 'st=$?; rm -rf /tmp/mqcheck; [ "$st" -eq 0 ] || echo "check FAILED (status $st): stop and fix the cause" >&2; exit "$st"' EXIT sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/mealquest.db sudo chown -R "$USER" /tmp/mqcheck git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only @@ -645,8 +653,9 @@ PY ``` The check runs in a subshell that stops at the first failed step — the -snapshot or its integrity check, `git ls-tree` or the comparison — and -removes `/tmp/mqcheck` on the way out without changing the status. Read it +snapshot or its integrity check, `git ls-tree` or the comparison. On the +way out it always attempts to remove `/tmp/mqcheck`, and a failed removal +fails the check too; every nonzero exit prints `check FAILED`. Read it like this: - `unknown to the previous SHA: none` and `echo $?` prints `0` — no @@ -655,18 +664,26 @@ like this: prints `0` — the older build does not know them: the migration-crossing path below is required. - `check FAILED`, any other error, or a nonzero `echo $?` — stop and fix - the cause, then run the check again. Do not pick a path. + the cause, then run the check again. Do not pick a path, even if an + `unknown to the previous SHA:` line was printed before it. If the + clean-up failed, remove `/tmp/mqcheck` by hand first. **No migration crossed** — rebuild and restart the older SHA: ```sh -git checkout --detach -docker compose -f deploy/compose.yaml build -sudo systemctl restart mealquest -docker compose -f deploy/compose.yaml ps +git checkout --detach && +docker compose -f deploy/compose.yaml build && +sudo systemctl restart mealquest && +docker compose -f deploy/compose.yaml ps && curl -fsS http://10.60.1.14:8090/health ``` +Each command runs only if the one before it succeeded, so the health +check answers only for the older SHA. If `echo $?` then prints anything +but `0`, the rollback is not done: the first error names the failed step, +and if it came before the restart, production is still on the release +being rolled back from. + **A migration crossed** — the older binary refuses the newer schema, so the database is restored first and the older SHA started after. Changes made since the pre-deploy restore point are lost; say so to the household. @@ -674,10 +691,13 @@ since the pre-deploy restore point are lost; say so to the household. 1. Check out and build the older SHA. Nothing restarts yet. ```sh -git checkout --detach +git checkout --detach && docker compose -f deploy/compose.yaml build ``` + If `echo $?` then prints anything but `0`, stop: the database has not + been touched and nothing has restarted. + 2. Run "Restore the database" steps 1–4, adding `-timestamp ` (recorded in step 2 of "Deploy") to the `restore` command. Its step 5 then starts the older @@ -838,7 +858,7 @@ running) and, with the cause understood, return the sidecars to `/tmp` is temporary, so write the scripts before every check: ```sh -cat > /tmp/mqcount.py <<'EOF' +cat > /tmp/mqcount.py <<'EOF' && import sqlite3, sys, hashlib c = sqlite3.connect(sys.argv[1]) tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")] @@ -868,6 +888,9 @@ print("snapshot", dst, "integrity ok") EOF ``` +The second file is written only if the first was; `echo $?` must then +print `0` before either script is used. + With the stack stopped, copy the database and run it: ```sh -- 2.55.0 From d22167cfd34460a31c01da42c27a545276992e61 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 19:23:21 -0400 Subject: [PATCH 8/9] Make every deploy, rollback and restore block fail fast with a final verdict Deploy step 3 printed `git rev-parse HEAD` as "the deployed SHA" even after a failed restart; the Count check copy block could read a stale /tmp/mqcheck copy and report integrity ok. Apply the established pattern (subshell with set -eu, trap registered first, clean-up that keeps a nonzero status, success line only after everything passed) to Deploy step 3, the no-migration rollback, Restore the database steps 1-3 and 5, the Count check copy block and Restore photos (media). The volume listings, service states, health codes and file counts that the operator previously read by eye are now checked in the block. Task: 008c0a8a-2242-429b-a1fd-69346bc1d414 Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 225 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 181 insertions(+), 44 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index c0cdb15..d33220a 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -599,16 +599,45 @@ point=$(date -u +%Y-%m-%dT%H:%M:%SZ) 3. Restart and check the deployed release. ```sh +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the check with a nonzero status +fail() { echo "$*" >&2; exit 1; } +sha= +# On the way out, print the deployed SHA only if every check passed. +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$sha" ]; then echo "deployed SHA: $sha" +else [ "$st" -ne 0 ] || st=1; echo "deploy FAILED (status $st): no SHA is deployed" >&2; fi +exit "$st"' EXIT sudo systemctl restart mealquest # applies the firewall, then down + up -d -docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up, searxng healthy +docker compose -f deploy/compose.yaml ps +for s in mealquest litestream searxng; do + id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") || fail "docker compose ps failed" + [ -n "$id" ] || fail "$s is not running" +done curl -fsS http://10.60.1.14:8090/health -curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 -git rev-parse HEAD # the deployed SHA +code=$(curl -sS -o /dev/null -w "%{http_code}" https://mealquest.cubecraftlabs.com/health) +[ "$code" = 200 ] || fail "the public /health returned $code, not 200" +sha=$(git rev-parse HEAD) # the checked-out SHA, now running and healthy +) ``` -Record the deployed SHA, the previous SHA, the pre-deploy restore point -and the date in the Maestro task for the release. Migrations run at start-up; the service refuses to start -if the database has an applied migration it does not know. + The check stops at the first failed step and prints the SHA last, only + when the restart, all three services running and both health checks + succeeded. `restart` returns once `mealquest` is healthy (Litestream + waits for it); SearXNG may still show `health: starting` for its first + 15 seconds. + + - `deployed SHA:` followed by the SHA, and `echo $?` prints `0` — + record that SHA, the previous SHA, the pre-deploy restore point and + the date in the Maestro task for the release. + - `deploy FAILED`, any other error, or a nonzero `echo $?` — record + nothing as deployed: the release is not running healthy, and after a + failed restart the stack may be down. Read + `journalctl -u mealquest` and `docker compose -f deploy/compose.yaml + logs`, then fix the cause or roll back ("Rollback to a previous SHA"). + +Migrations run at start-up; the service refuses to start if the database +has an applied migration it does not know. If `deploy/.env.example` gained a variable, add it to `deploy/.env` before the restart; the service fails fast at boot naming any missing required variable. @@ -671,18 +700,35 @@ like this: **No migration crossed** — rebuild and restart the older SHA: ```sh -git checkout --detach && -docker compose -f deploy/compose.yaml build && -sudo systemctl restart mealquest && -docker compose -f deploy/compose.yaml ps && +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the rollback with a nonzero status +P= +fail() { echo "$*" >&2; exit 1; } +sha= +# On the way out, print the running SHA only if every check passed. +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$sha" ]; then echo "rolled back to: $sha" +else [ "$st" -ne 0 ] || st=1; echo "rollback FAILED (status $st)" >&2; fi +exit "$st"' EXIT +git checkout --detach "$P" +docker compose -f deploy/compose.yaml build +sudo systemctl restart mealquest +docker compose -f deploy/compose.yaml ps +for s in mealquest litestream searxng; do + id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") || fail "docker compose ps failed" + [ -n "$id" ] || fail "$s is not running" +done curl -fsS http://10.60.1.14:8090/health +sha=$(git rev-parse HEAD) +) ``` -Each command runs only if the one before it succeeded, so the health -check answers only for the older SHA. If `echo $?` then prints anything -but `0`, the rollback is not done: the first error names the failed step, -and if it came before the restart, production is still on the release -being rolled back from. +Each step runs only if the one before it succeeded, so the health +check answers only for the older SHA. Only `rolled back to:` followed by +the SHA, with `echo $?` printing `0`, means the rollback is done. On +`rollback FAILED` or a nonzero `echo $?` it is not: the first error names +the failed step, and if it came before the restart, production is still +on the release being rolled back from. **A migration crossed** — the older binary refuses the newer schema, so the database is restored first and the older SHA started after. Changes made @@ -788,11 +834,20 @@ database it replaced. So all three files are moved aside together, with Litestream's local state directory `.mealquest.db-litestream`, before the restore — and the restore runs into an empty volume. -Run from `/home/overseer/mealquest`: +Steps 1–3: ```sh +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the restore with a nonzero status D=/var/lib/docker/volumes/mealquest-data/_data A=/var/tmp/mealquest-restore-aside-$(date +%Y%m%d%H%M) +fail() { echo "$*" >&2; exit 1; } +# On the way out, name the aside directory; on any failure the stack +# stays stopped. +trap 'st=$? +if [ "$st" -eq 0 ]; then echo "restored mealquest.db; the old files are in $A: run step 4" +else echo "restore FAILED (status $st): leave the stack stopped; anything moved aside is in $A" >&2; fi +exit "$st"' EXIT # 1. Stop the whole stack (app and Litestream). Nothing may hold the # database open; a clean close checkpoints the WAL into mealquest.db. @@ -805,25 +860,55 @@ sudo mkdir "$A" for f in mealquest.db mealquest.db-wal mealquest.db-shm .mealquest.db-litestream; do if sudo test -e "$D/$f"; then sudo mv "$D/$f" "$A/"; fi done -sudo ls -la "$D" # must list only media/ +left=$(sudo ls -A "$D") +[ "$left" = media ] || fail "the volume must hold only media/, not: $left" # 3. Restore from the NAS. --no-deps keeps the app stopped so it cannot # create a fresh, empty database first. docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db -sudo ls -la "$D" # mealquest.db only: no -wal, no -shm - -# 4. Check the restored file before starting the app (see "Count check"). - -# 5. Start the stack and check health. -sudo systemctl start mealquest -docker compose -f deploy/compose.yaml ps -curl -s -o /dev/null -w "health %{http_code}\n" http://10.60.1.14:8090/health +left=$(sudo ls -A "$D") +[ "$left" = "$(printf 'mealquest.db\nmedia')" ] || fail "the volume must hold only mealquest.db and media/ (no -wal, no -shm), not: $left" +) ``` To restore to a point in time, add `-timestamp 2026-10-05T23:00:00Z` (or `-txid`) to the `restore` command. +The block stops at the first failed step. Only `restored mealquest.db` +with `echo $?` printing `0` lets you go on; on `restore FAILED` leave the +stack stopped, fix the cause and run the block again (a new aside +directory is made; the earlier one keeps what it holds). + +4. Check the restored file before starting the app: run the copy block + in "Count check". + +5. Start the stack and check health, only after step 4 passed: + +```sh +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the check with a nonzero status +fail() { echo "$*" >&2; exit 1; } +sha= +# On the way out, print the running SHA only if every check passed. +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$sha" ]; then echo "stack running and healthy at $sha" +else [ "$st" -ne 0 ] || st=1; echo "start FAILED (status $st)" >&2; fi +exit "$st"' EXIT +sudo systemctl start mealquest +docker compose -f deploy/compose.yaml ps +for s in mealquest litestream searxng; do + id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") || fail "docker compose ps failed" + [ -n "$id" ] || fail "$s is not running" +done +curl -fsS http://10.60.1.14:8090/health +sha=$(git rev-parse HEAD) +) +``` + + Only `stack running and healthy at` with `echo $?` printing `0` ends + the restore; then the aside directory may go. + ### Count check `/tmp/mqcount.py` prints row counts for key tables, the number of @@ -894,12 +979,32 @@ print `0` before either script is used. With the stack stopped, copy the database and run it: ```sh -rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck -sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck -python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the check with a nonzero status +D=/var/lib/docker/volumes/mealquest-data/_data +fail() { echo "$*" >&2; exit 1; } +# On the way out, remove /tmp/mqcheck whatever happens, keep any failure, +# and give the verdict only if every step and the clean-up succeeded. +trap 'st=$? +rm -rf /tmp/mqcheck || st=1 +if [ "$st" -eq 0 ]; then echo "count check: integrity ok" +else echo "count check FAILED (status $st): do not start the stack" >&2; fi +exit "$st"' EXIT +rm -rf /tmp/mqcheck +mkdir /tmp/mqcheck +sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" +sudo chown -R "$USER" /tmp/mqcheck +python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db > /tmp/mqcheck/counts.txt +cat /tmp/mqcheck/counts.txt +grep -qx 'integrity ok' /tmp/mqcheck/counts.txt || fail "the copy failed integrity_check" +) ``` -Before and after must match exactly. +The check stops at the first failed step — the clean-up of an earlier +`/tmp/mqcheck`, the copy, the count or the integrity check — so it never +reads a file left by an earlier run. Only `count check: integrity ok` +with `echo $?` printing `0` passes; then the counts printed above it, +before and after, must match exactly. ## Restore photos (media) @@ -916,20 +1021,43 @@ not start the service until the restore is verified. Run from `/home/overseer/mealquest`: ```sh +cd /home/overseer/mealquest && ( +set -eu # any failed step ends the restore with a nonzero status M=/var/lib/docker/volumes/mealquest-data/_data/media NAS=/volume1/homes/Joshua/mealquest-backups/media SSH="ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" +fail() { echo "$*" >&2; exit 1; } +nas= here= +# On the way out, give the counts only if every step passed; on any +# failure the timer stays stopped. +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$here" ]; then echo "media restored: $here files here, $nas on the NAS" +else [ "$st" -ne 0 ] || st=1; echo "media restore FAILED (status $st): leave the timer stopped and find out why" >&2; fi +exit "$st"' EXIT # 1. Stop the nightly mirror: the timer AND any run in flight. sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service -# 2. Pre-check — all three must hold before media/ is touched. If any -# fails, stop here and find out why. -systemctl is-active mealquest-media-backup.timer # inactive -systemctl is-active mealquest-media-backup.service # inactive (or failed) -pgrep -a -u mealquest rsync || echo "no rsync" # no rsync -sudo $SSH joshua@10.60.1.50 "find $NAS -type f | wc -l" -# ^ note this NAS file count; it must be non-zero and is the target below. +# 2. Pre-check — all must hold before media/ is touched. is-active exits +# nonzero for every state but active, so its printed state is checked; +# pgrep exits 0 on a match, 1 on none, 2+ on an error. +ts=$(systemctl is-active mealquest-media-backup.timer) || : +[ "$ts" = inactive ] || fail "the timer is ${ts:-unknown}, not inactive" +ss=$(systemctl is-active mealquest-media-backup.service) || : +case "$ss" in + inactive|failed) ;; + *) fail "the service is ${ss:-unknown}, not inactive or failed";; +esac +ps=0 +pgrep -a -u mealquest rsync || ps=$? +case "$ps" in + 0) fail "an rsync is still running as mealquest";; + 1) ;; # no match: no rsync + *) fail "pgrep failed (status $ps)";; +esac +l=$(sudo $SSH joshua@10.60.1.50 "find $NAS -type f") || fail "listing the NAS copy failed" +[ -n "$l" ] || fail "the NAS copy holds no files" +nas=$(printf '%s\n' "$l" | wc -l) # the target below # 3. Stop the app (no writes into media/ meanwhile). docker compose -f deploy/compose.yaml stop mealquest @@ -941,21 +1069,30 @@ sudo install -d -o 10001 -g 10001 -m 0755 "$M" # direction: the NAS copy is the source and is only read. sudo rsync -a --stats --chown=10001:10001 -e "$SSH" "joshua@10.60.1.50:$NAS/" "$M/" -# 6. Post-condition — before anything can mirror again: -sudo find "$M" -type f | wc -l # >= the NAS count from step 2 -sudo $SSH joshua@10.60.1.50 "find $NAS -type f | wc -l" # unchanged from step 2 +# 6. Post-condition — before anything can mirror again: at least the NAS +# count here, and the NAS count unchanged. +l=$(sudo find "$M" -type f) || fail "listing $M failed" +[ -n "$l" ] || fail "$M holds no files after the copy" +n=$(printf '%s\n' "$l" | wc -l) +[ "$n" -ge "$nas" ] || fail "only $n of the NAS's $nas files are here: the copy did not complete" +l=$(sudo $SSH joshua@10.60.1.50 "find $NAS -type f") || fail "listing the NAS copy failed" +[ -n "$l" ] && [ "$(printf '%s\n' "$l" | wc -l)" -eq "$nas" ] || fail "the NAS file count changed from $nas" +here=$n # 7. Start the app, then the timer again. Only now may the service run. docker compose -f deploy/compose.yaml up -d curl -fsS http://10.60.1.14:8090/health sudo systemctl start mealquest-media-backup.timer -systemctl is-active mealquest-media-backup.timer # active +systemctl is-active mealquest-media-backup.timer # active; exits nonzero otherwise +) ``` -The local count may exceed the NAS count only if `media/` still held -files the NAS never received (photos added since the last nightly run); -the next nightly run then copies them up. If the local count is lower, -the copy did not complete: leave the timer stopped and re-run step 5. +The block stops at the first failed step and gives the counts last, only +when every step passed. On `media restore FAILED` or a nonzero `echo $?` +leave the timer stopped. The local count may exceed the NAS count only if +`media/` still held files the NAS never received (photos added since the +last nightly run); the next nightly run then copies them up. If the local count is lower, +the copy did not complete: fix the cause and run the block again. When the database is restored too, restore it first ("Restore the database"), then the media, so every photo the database refers to is -- 2.55.0 From a7b079c7714df08664e8883ad517464990e1c303 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Wed, 7 Oct 2026 19:48:14 -0400 Subject: [PATCH 9/9] Verify restored media by content and report the timer's actual state The media restore copied with rsync -a and checked file counts only, so a local photo with the NAS size and mtime but other bytes could survive and still print "media restored". Take a SHA-256 manifest of the NAS before the copy, copy with --checksum, require every manifest entry to be present locally with the same bytes, and require the NAS to still match the manifest. On failure, the message now reads the timer's real state instead of assuming it is stopped. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index d33220a..edfc375 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -1027,13 +1027,17 @@ M=/var/lib/docker/volumes/mealquest-data/_data/media NAS=/volume1/homes/Joshua/mealquest-backups/media SSH="ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" fail() { echo "$*" >&2; exit 1; } -nas= here= +nas= here= w= # On the way out, give the counts only if every step passed; on any -# failure the timer stays stopped. +# failure, report the timer's actual state, not an assumed one. trap 'st=$? -if [ "$st" -eq 0 ] && [ -n "$here" ]; then echo "media restored: $here files here, $nas on the NAS" -else [ "$st" -ne 0 ] || st=1; echo "media restore FAILED (status $st): leave the timer stopped and find out why" >&2; fi +[ -z "$w" ] || rm -rf "$w" +if [ "$st" -eq 0 ] && [ -n "$here" ]; then echo "media restored: $here files here, $nas on the NAS, all $nas match the NAS manifest" +else [ "$st" -ne 0 ] || st=1; t=$(systemctl is-active mealquest-media-backup.timer) || : + if [ "$t" = inactive ]; then echo "media restore FAILED (status $st): the timer is inactive; leave it stopped and find out why" >&2 + else echo "media restore FAILED (status $st): the timer is ${t:-unknown}, not inactive; run sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service and confirm it is inactive before anything else" >&2; fi; fi exit "$st"' EXIT +w=$(mktemp -d) # 1. Stop the nightly mirror: the timer AND any run in flight. sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service @@ -1058,6 +1062,11 @@ esac l=$(sudo $SSH joshua@10.60.1.50 "find $NAS -type f") || fail "listing the NAS copy failed" [ -n "$l" ] || fail "the NAS copy holds no files" nas=$(printf '%s\n' "$l" | wc -l) # the target below +# The NAS manifest: one "sha256 ./path" line per file, taken now, before +# the copy. Every check below compares content against it, not counts. +sudo $SSH joshua@10.60.1.50 "cd $NAS && find . -type f -exec sha256sum {} +" >"$w/raw" || fail "hashing the NAS copy failed" +LC_ALL=C sort "$w/raw" >"$w/nas" +[ "$(wc -l <"$w/nas")" -eq "$nas" ] || fail "the NAS manifest has $(wc -l <"$w/nas") files, the listing $nas" # 3. Stop the app (no writes into media/ meanwhile). docker compose -f deploy/compose.yaml stop mealquest @@ -1066,8 +1075,10 @@ docker compose -f deploy/compose.yaml stop mealquest sudo install -d -o 10001 -g 10001 -m 0755 "$M" # 5. Copy back from the NAS as uid 10001's files. No --delete in this -# direction: the NAS copy is the source and is only read. -sudo rsync -a --stats --chown=10001:10001 -e "$SSH" "joshua@10.60.1.50:$NAS/" "$M/" +# direction: the NAS copy is the source and is only read. --checksum +# compares content, so a local file with the NAS size and mtime but +# other bytes is still replaced. +sudo rsync -a --checksum --stats --chown=10001:10001 -e "$SSH" "joshua@10.60.1.50:$NAS/" "$M/" # 6. Post-condition — before anything can mirror again: at least the NAS # count here, and the NAS count unchanged. @@ -1077,6 +1088,14 @@ n=$(printf '%s\n' "$l" | wc -l) [ "$n" -ge "$nas" ] || fail "only $n of the NAS's $nas files are here: the copy did not complete" l=$(sudo $SSH joshua@10.60.1.50 "find $NAS -type f") || fail "listing the NAS copy failed" [ -n "$l" ] && [ "$(printf '%s\n' "$l" | wc -l)" -eq "$nas" ] || fail "the NAS file count changed from $nas" +# Content: every file in the NAS manifest is here with the same bytes, +# and the NAS still matches the manifest taken before the copy. +sudo sh -c 'cd "$1" && find . -type f -exec sha256sum {} +' sh "$M" >"$w/raw" || fail "hashing $M failed" +LC_ALL=C sort "$w/raw" >"$w/here" +LC_ALL=C comm -23 "$w/nas" "$w/here" >"$w/bad" +[ ! -s "$w/bad" ] || { cat "$w/bad" >&2; fail "$(wc -l <"$w/bad") NAS file(s) above are missing here or differ in content"; } +sudo $SSH joshua@10.60.1.50 "cd $NAS && find . -type f -exec sha256sum {} +" >"$w/raw" || fail "hashing the NAS copy failed" +LC_ALL=C sort "$w/raw" | cmp -s - "$w/nas" || fail "the NAS copy changed during the restore" here=$n # 7. Start the app, then the timer again. Only now may the service run. @@ -1088,8 +1107,11 @@ systemctl is-active mealquest-media-backup.timer # active; exits nonzero other ``` The block stops at the first failed step and gives the counts last, only -when every step passed. On `media restore FAILED` or a nonzero `echo $?` -leave the timer stopped. The local count may exceed the NAS count only if +when every step passed and every file in the NAS manifest (a SHA-256 of +each file, taken before the copy) is present locally with the same bytes. +On `media restore FAILED` or a nonzero `echo $?` the message gives the +timer's actual state: if it is not `inactive`, stop it as the message says +before anything else; then leave it stopped. The local count may exceed the NAS count only if `media/` still held files the NAS never received (photos added since the last nightly run); the next nightly run then copies them up. If the local count is lower, the copy did not complete: fix the cause and run the block again. -- 2.55.0