Automate production deploys of main (Maestro 85cca5ad)

deploy/auto-deploy.sh runs deploy/RUNBOOK.md unattended: no-op when origin/main is the last deployed SHA in the deploy record, wait while its Gitea CI status is pending, refuse (exit 1) on failure or no status, then run every runbook block (rollback SHA from the running /health, clean checkout, make docker, Litestream restore point, boot-required configuration, retag, restart, version and PWA verification) and append a deploy record line. --ref <ref> deploys an explicit ref through the same blocks, bypassing the gates. A failed deploy is recorded and never retried by the timer. mealquest-auto-deploy.service/.timer run it as overseer every 15 minutes (+5 min random delay), skipped while mealquest.service is not active. deploy/RUNBOOK.md is the PR #203 runbook (feature/launch-readiness, not yet on dev) plus an "Automated deploys" section; AGENTS.md describes deploy/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 06:58:51 -04:00
parent 34e84964a3
commit 2b9b0cb399
5 changed files with 640 additions and 1 deletions
@@ -84,7 +84,7 @@ behaviour, exercise it (hit the endpoint, render the page).
 | `internal/*` (planned) | `auth`, `db`, `store`, `secretbox`, `household`, `dinners`, `recipes`, `ai/llm`, `ai/harness`, `ai/skills`, `suggest`, `plan`, `shopping`, `ideas`, `gcal`, `notify`, `push`, `chat`, `jobs`, `scheduler`. |
 | `migrations/` | SQL migrations (planned). |
 | `web/` | React PWA source. |
-| `deploy/` | Deployment files (planned). |
+| `deploy/` | Deployment files: compose stack, systemd units, [deploy runbook](deploy/RUNBOOK.md) and the automated deploy (`auto-deploy.sh` with `systemd/mealquest-auto-deploy.*`). |
 | `docs/` | Project documentation, including [ARCHITECTURE.md](docs/ARCHITECTURE.md). |
 | `design/` | Requirements and UI handoff. **Read-only reference** — do not edit. |
 | `Dockerfile` | Multi-stage image build, at the repository root. |
@@ -0,0 +1,356 @@
+# MealQuest deploy runbook
+
+The one way to deploy MealQuest to production: the docker compose project
+`mealquest` on office-srvr `10.60.1.14` (see
+[ARCHITECTURE.md](../docs/ARCHITECTURE.md#topology)). Production runs `main`
+or a release tag; only `overseer` merges `dev` → `main`. Every deploy builds
+on the host, from the checked-out commit, with `make docker`.
+
+Host setup (clone, firewall, the systemd unit, Traefik, DNS, Litestream) and
+the database restore are in CTXD project `mealquest`, `RUNBOOKS.MD`. This
+file is the build, restart, verification and rollback procedure; where it
+uses a CTXD section, it names it.
+
+All commands run on `10.60.1.14` as `overseer`, in bash, from the checkout
+`/home/overseer/mealquest`. A block that ends with `FAILED` or a non-zero
+`$?` stops the deploy: do not go on to the next step.
+
+## How the image carries its version
+
+`make docker` runs `docker build --build-arg VERSION=<version>` and tags the
+image `mealquest:<version>`, where `<version>` is
+`git describe --tags --always` of the checkout (a release tag such as
+`v1.0.0`, otherwise the short SHA), with `-dirty` added when a tracked file
+is modified. The [Dockerfile](../Dockerfile) links it into the binary, and
+`/health` reports it:
+
+```json
+{"status":"ok","version":"v1.0.0"}
+```
+
+Never pass `--build-arg VERSION` by hand, and never build with
+`docker compose build` or `docker compose up --build`: compose passes no
+version, so that image reports `dev`.
+
+The compose service `mealquest` runs the image named `mealquest-mealquest`
+(compose's name for the image it would build; check with
+`docker compose -f deploy/compose.yaml config --images mealquest`). It
+already exists, so `docker compose up` uses it and never builds. A deploy
+points that name at the `make docker` image with `docker tag`. Every
+`mealquest:<version>` image is kept for rollback; do not prune them.
+
+## Prerequisites
+
+- `make`, `git`, `curl`, `python3` and Docker Engine with Compose on the
+  host (`make` is not in the base install: `sudo apt-get install make`).
+- `deploy/.env` (git-ignored, owner `overseer`, mode 0600) sets every
+  variable [`deploy/.env.example`](.env.example) sets, with
+  `MEALQUEST_BIND_ADDRESS=10.60.1.14`. It must not set the variables
+  `deploy/compose.yaml` fixes (`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR`,
+  `MEALQUEST_SEARXNG_URL` and the `*_FILE` variables). Creating it: CTXD
+  `RUNBOOKS.MD` "Stack and first deploy" > "3. Environment file".
+- `deploy/secrets/` (git-ignored, mode 0700) holds `secret_key`,
+  `oidc_client_secret`, `google_client_secret`, `vapid_private_key` and
+  `litestream_key`, each owned by `10001:10001` with mode 0600, and
+  `searxng.env`, owned by `overseer` with mode 0600. Creating them: CTXD
+  `RUNBOOKS.MD` "Stack and first deploy" > "4. Secrets".
+- The compose file is the checkout's [`deploy/compose.yaml`](compose.yaml);
+  the systemd unit [`mealquest.service`](systemd/mealquest.service) starts
+  and stops it and is the only way the stack starts.
+- DNS and Traefik already route `https://mealquest.cubecraftlabs.com` to
+  `10.60.1.14:8090` (CTXD `RUNBOOKS.MD` "Reverse proxy (Traefik)", "DNS").
+
+Check `deploy/.env` and the secrets before every deploy (a new release may
+add a variable; the service refuses to boot without it). Both commands must
+print nothing:
+
+```sh
+cd /home/overseer/mealquest
+comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \
+         <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u)   # variables missing from deploy/.env
+sudo find deploy/secrets -type f ! -perm 600                       # secrets not mode 0600
+```
+
+## Deploy
+
+### 1. Build
+
+Record the release production is running now, check out the new release
+and build its image. Nothing restarts.
+
+The rollback target is the version the running service reports on
+`/health`, read before the checkout changes, never the checkout's `HEAD`:
+after a failed or deferred deploy the checkout can be ahead of production.
+The block resolves that version to a commit and prints it as the rollback
+SHA. A running version of `dev` or `*-dirty` names no commit: the block
+fails, and the rollback target must be established by hand before going on.
+
+The checkout must be clean (the same test as the Makefile's `GIT_DIRTY`:
+no modified tracked file outside the generated `internal/frontend/dist`).
+The block computes the image tag exactly as `make docker` does, `-dirty`
+included, and checks that image exists after the build.
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+built=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$built" ]; then echo "built: $built"
+else [ "$st" -ne 0 ] || st=1; echo "build FAILED (status $st)" >&2; fi
+exit "$st"' EXIT
+bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-)
+[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
+running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+  || fail "http://$bind:8090/health did not return a version: no running release to roll back to"
+case "$running" in
+  dev|*-dirty) fail "the running release reports $running, which names no commit" ;;
+esac
+git fetch --tags origin
+rollback=$(git rev-parse --verify --quiet "$running^{commit}") \
+  || fail "running version $running is not a commit in this repository"
+echo "rollback SHA: $rollback (running version $running)"   # RECORD it for rollback
+git checkout --detach origin/main            # or the release tag, e.g. v1.0.0
+dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist')
+[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first:
+$dirty"
+v=$(git describe --tags --always)${dirty:+-dirty}   # the Makefile's VERSION
+make docker
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed"
+built="mealquest:$v"
+)
+```
+
+### 2. Ship and restart
+
+1. **Pre-restart gates.** Run CTXD `RUNBOOKS.MD` "Routine deploy and
+   rollback" > "Deploy" step 2 (a verified pre-deploy restore point; record
+   it with the rollback SHA) and step 3 (boot-required configuration). The
+   restart applies migrations the previous release refuses, so without that
+   restore point there is no rollback. If either fails, do not restart:
+   production stays on the previous release.
+2. **Point the stack at the new image and restart.** `docker compose pull`
+   fetches the pinned `litestream` and `searxng` images before anything
+   stops; `mealquest` is never pulled. The restart applies the firewall,
+   then runs `down` and `up -d`, and returns once `mealquest` passes its
+   healthcheck (`/health` inside the container).
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+up=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$up" ]; then echo "restarted on: $up"
+else [ "$st" -ne 0 ] || st=1; echo "restart FAILED (status $st)" >&2; fi
+exit "$st"' EXIT
+v=$(git describe --tags --always)
+docker compose -f deploy/compose.yaml pull litestream searxng
+docker tag "mealquest:$v" mealquest-mealquest
+sudo systemctl restart mealquest
+for s in mealquest litestream searxng; do
+  id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s")
+  [ -n "$id" ] || fail "$s is not running"
+done
+id=$(docker compose -f deploy/compose.yaml ps --quiet mealquest)
+health=$(docker inspect -f '{{.State.Health.Status}}' "$id")
+[ "$health" = healthy ] || fail "mealquest is $health, not healthy"
+up="mealquest:$v"
+)
+```
+
+On failure read `journalctl -u mealquest` and
+`docker compose -f deploy/compose.yaml logs mealquest`: a configuration
+error names the variable. Fix it or roll back.
+
+### 3. Verify the deployed version and a real screen
+
+A `/health` 200 alone proves only that some build is up. This step checks,
+on the LAN address and through Traefik, that:
+
+- `/health` reports exactly the checkout's `git describe --tags --always`;
+- `/` and the Plan screen's route `/plan` serve the built PWA shell (the
+  React root and a hashed `/assets/` script, not the placeholder page an
+  unbuilt embed serves);
+- that script is served as JavaScript.
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+want=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$want" ]; then echo "deployed and verified: $want ($(git rev-parse HEAD))"
+else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): the deploy is not verified" >&2; fi
+exit "$st"' EXIT
+v=$(git describe --tags --always)
+for base in http://10.60.1.14:8090 https://mealquest.cubecraftlabs.com; do
+  got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+    || fail "$base/health did not return a version"
+  [ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v"
+  for path in / /plan; do
+    page=$(curl -fsS "$base$path") || fail "$base$path did not return 200"
+    grep -q '<div id="root"></div>' <<< "$page" || fail "$base$path is not the PWA shell"
+    js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true
+    [ -n "$js" ] || fail "$base$path references no built /assets/ script"
+    type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200"
+    case "$type" in
+      text/javascript*) ;;
+      *) fail "$base$js is $type, not JavaScript" ;;
+    esac
+  done
+  echo "$base: version $got, PWA shell and $js served"
+done
+want=$v
+)
+```
+
+- `deployed and verified: <version> (<SHA>)` and `$?` = `0` → open
+  `https://mealquest.cubecraftlabs.com` in a browser: sign in and the Home
+  screen renders with the household's data. Then record the deployed
+  version and SHA, the rollback SHA, the pre-deploy restore point and the
+  date in the release's Maestro task.
+- `verification FAILED`, any error or a non-zero `$?` → the deploy is not
+  done. A version mismatch means the stack runs a different image than the
+  checkout: repeat step 2, or roll back.
+
+## Rollback to a previous release
+
+Rolling back points `mealquest-mealquest` at the previous release's
+`mealquest:<version>` image again; that release's checkout keeps the
+compose file and the version check consistent with it.
+
+1. **Check whether the rollback crosses a migration** with CTXD
+   `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a previous
+   SHA" (its first block), run against the rollback SHA that deploy step 1
+   recorded. If it prints versions unknown to the rollback SHA, restore the
+   database to the pre-deploy restore point first (CTXD `RUNBOOKS.MD`
+   "Restore the database", with `-timestamp <restore point>`); changes
+   since then are lost, so tell the household.
+2. **Check out the rollback SHA and retag its image.** An image built before
+   this runbook was not tagged `mealquest:<version>`; `make docker` builds
+   it from the same commit.
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+P=<rollback SHA>   # recorded by step 1, from the running release's /health
+back=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$back" ]; then echo "rolled back to: $back"
+else [ "$st" -ne 0 ] || st=1; echo "rollback FAILED (status $st)" >&2; fi
+exit "$st"' EXIT
+git checkout --detach "$P"
+v=$(git describe --tags --always)
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || make docker
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v"
+docker tag "mealquest:$v" mealquest-mealquest
+sudo systemctl restart mealquest
+back="mealquest:$v"
+)
+```
+
+3. **Run step 3 above** unchanged. The rollback is done only when it prints
+   `deployed and verified:` with the previous release's version.
+
+## Automated deploys
+
+[`auto-deploy.sh`](auto-deploy.sh), installed as
+`/usr/local/sbin/mealquest-auto-deploy`, runs this runbook's deploy unattended:
+step 1 (build), the prerequisite `deploy/.env` and secrets check, CTXD
+`RUNBOOKS.MD` "Deploy" steps 2 (pre-deploy restore point) and 3
+(boot-required configuration), step 2 (ship and restart) and step 3
+(verify). No check is skipped or weakened, and it stops at the first
+failure. Every failure before the restart leaves production on the previous
+release.
+
+### What runs, and when
+
+[`mealquest-auto-deploy.timer`](systemd/mealquest-auto-deploy.timer) starts
+[`mealquest-auto-deploy.service`](systemd/mealquest-auto-deploy.service)
+2 minutes after the timer starts, then 15 minutes after each run, plus up to
+5 minutes' random delay. The service runs the script as `overseer` from the
+checkout, and only while `mealquest.service` is active: a stack stopped on
+purpose (a database or media restore) is skipped, not redeployed. Output goes
+to the journal (`journalctl -u mealquest-auto-deploy`).
+
+Each run fetches `origin` and compares `origin/main` with the last
+`deployed=` SHA in the deploy record `/home/overseer/mealquest-auto-deploy.log`:
+
+| `origin/main` | Result |
+| --- | --- |
+| is the last deployed SHA | No-op: exit 0, no output. |
+| CI (Gitea commit status) `pending` | Exit 0, no output; the next run checks again. |
+| CI `failure`, `error`, `warning`, or no status yet | Error in the journal, exit 1, nothing deployed. |
+| the last record line is `FAILED` for this SHA | Error in the journal, exit 1, nothing deployed: a failed deploy is never retried by the timer. |
+| CI `success` | Deploy. |
+
+The CI status is read from
+`https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits/<SHA>/status`
+(public; no credential). A commit merged moments before a run can briefly
+report no status, which fails that run; the next run reads it again.
+
+A deploy appends one line to the record:
+
+```text
+2026-10-12T15:04:05Z previous=<rollback SHA> deployed=<SHA> restore_point=2026-10-12T15:01:59Z
+```
+
+`previous` is the rollback SHA (from the running release's `/health`,
+never the checkout) and `restore_point` the verified pre-deploy restore
+point: record both in the release's Maestro task. A failed deploy appends
+`<date> FAILED target=<SHA> status=<exit status>` and logs
+`auto-deploy FAILED (status <n>)` to the journal, adding whether it failed
+before the restart (production untouched) or after it (read
+`journalctl -u mealquest`, then fix or roll back). Either way the checkout is
+left at the target commit; the running release on `/health` stays the
+rollback target.
+
+### Install on the host
+
+The script and units are installed copies: repeat this whenever a release
+changes `deploy/auto-deploy.sh` or the units. From the checkout, at a release
+that contains them:
+
+```sh
+cd /home/overseer/mealquest
+sudo install -m 0755 deploy/auto-deploy.sh /usr/local/sbin/mealquest-auto-deploy
+sudo install -m 0644 deploy/systemd/mealquest-auto-deploy.service \
+  deploy/systemd/mealquest-auto-deploy.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+```
+
+The timer refuses to deploy until the record holds a deploy, so seed it with
+one manual deploy of the release production should run (below), then start
+the timer:
+
+```sh
+sudo /usr/local/sbin/mealquest-auto-deploy --ref origin/main
+sudo systemctl enable --now mealquest-auto-deploy.timer
+systemctl list-timers mealquest-auto-deploy.timer
+```
+
+To stop automated deploys: `sudo systemctl disable --now mealquest-auto-deploy.timer`.
+
+### Manual override
+
+```sh
+sudo /usr/local/sbin/mealquest-auto-deploy --ref <ref>
+```
+
+deploys an explicit ref (a SHA, tag or `origin/<branch>`) through the same
+blocks, bypassing the CI and no-op gates: for a roll-forward fix, a re-run
+after a `FAILED` record, or seeding the record. Run as root, the script
+re-executes itself as `overseer`; it holds a lock, so it never runs alongside
+the timer's run. It prints `deployed and verified:` and exits 0 only when
+step 3 passed.
+
+### Rollback
+
+Rollback is not automated: follow "Rollback to a previous release" above
+and CTXD `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a
+previous SHA", using the `previous` SHA and `restore_point` from the record.
+The timer does not undo a manual rollback: `origin/main` is still the last
+recorded deploy, so its runs are no-ops until `main` moves, and a new `main`
+then deploys as usual. Disable the timer if the rollback must hold.
@@ -0,0 +1,243 @@
+#!/usr/bin/env bash
+# MealQuest automated deploy: deploy/RUNBOOK.md, run unattended.
+#
+#   mealquest-auto-deploy              deploy origin/main if it is new and CI passed
+#   mealquest-auto-deploy --ref <ref>  deploy <ref> now, bypassing the CI and no-op gates
+#
+# Installed as /usr/local/sbin/mealquest-auto-deploy and run as overseer by
+# mealquest-auto-deploy.service (deploy/systemd); install, semantics and the
+# manual override: deploy/RUNBOOK.md "Automated deploys". Run as root (sudo)
+# it re-executes itself as overseer.
+#
+# Without --ref: exit 0 with no output when origin/main is the last deployed
+# SHA in the deploy record or its CI is pending; exit 1, deploying nothing,
+# when its CI failed or reports no status, or when the last attempt to deploy
+# that SHA failed. Every deploy runs the runbook's blocks in order and stops
+# at the first failure; nothing before the restart touches production. A
+# verified deploy appends one line to the deploy record:
+#   <UTC date> previous=<SHA> deployed=<SHA> restore_point=<UTC date>
+# a failed one appends:
+#   <UTC date> FAILED target=<SHA> status=<exit status>
+set -euo pipefail
+
+CHECKOUT=/home/overseer/mealquest
+RECORD=/home/overseer/mealquest-auto-deploy.log
+LOCK=/home/overseer/mealquest-auto-deploy.lock
+STATUS_API=https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits
+PUBLIC=https://mealquest.cubecraftlabs.com
+D=/var/lib/docker/volumes/mealquest-data/_data
+COMPOSE=(docker compose -f deploy/compose.yaml)
+
+fail() { echo "$*" >&2; exit 1; }
+
+if [ "$(id -u)" -eq 0 ]; then
+  exec sudo -u overseer -H -- "$0" "$@"
+fi
+[ "$(id -un)" = overseer ] || fail "run as overseer (or root), not $(id -un)"
+
+ref=
+case "$#" in
+  0) ;;
+  2) [ "$1" = --ref ] && [ -n "$2" ] || fail "usage: $0 [--ref <ref>]"; ref=$2 ;;
+  *) fail "usage: $0 [--ref <ref>]" ;;
+esac
+
+exec 9>> "$LOCK"
+flock -n 9 || fail "another mealquest-auto-deploy run holds $LOCK"
+
+cd "$CHECKOUT"
+git fetch --quiet --tags origin || fail "git fetch origin failed"
+
+if [ -n "$ref" ]; then
+  target=$(git rev-parse --verify --quiet "$ref^{commit}") || fail "$ref is not a commit in $CHECKOUT"
+else
+  target=$(git rev-parse --verify origin/main)
+  [ -f "$RECORD" ] || fail "no deploy record at $RECORD: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")"
+  last=$(grep -oE 'deployed=[0-9a-f]{40}' "$RECORD" | tail -n 1 | cut -d = -f 2) || true
+  [ -n "$last" ] || fail "$RECORD records no deploy: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")"
+  [ "$target" != "$last" ] || exit 0
+  if tail -n 1 "$RECORD" | grep -q " FAILED target=$target status="; then
+    fail "auto-deploy of $target already failed (see $RECORD and the journal): fix it, then deploy with --ref"
+  fi
+  state=$(curl -fsS --max-time 30 "$STATUS_API/$target/status" \
+    | python3 -c 'import json, sys; d = json.load(sys.stdin); print(d["state"] if d["total_count"] else "absent")') \
+    || fail "could not read the CI status of $target from $STATUS_API"
+  case "$state" in
+    success) ;;
+    pending) exit 0 ;;
+    *) fail "CI for origin/main $target is $state, not success: not deploying" ;;
+  esac
+fi
+
+echo "auto-deploy: deploying $target${ref:+ (--ref $ref)}"
+restarted=
+on_exit() {
+  local st=$?
+  if [ "$st" -ne 0 ]; then
+    echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) FAILED target=$target status=$st" >> "$RECORD"
+    if [ -n "$restarted" ]; then
+      echo "auto-deploy FAILED (status $st) after the restart: production may be down or unverified; read journalctl -u mealquest, then fix or roll back (deploy/RUNBOOK.md)" >&2
+    else
+      echo "auto-deploy FAILED (status $st): production untouched" >&2
+    fi
+  fi
+  exit "$st"
+}
+trap on_exit EXIT
+
+# deploy/RUNBOOK.md "Deploy" > "1. Build": record the rollback SHA from the
+# RUNNING release, check out the target, reject a dirty checkout, build.
+bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-) || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
+[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
+running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+  || fail "http://$bind:8090/health did not return a version: no running release to roll back to"
+case "$running" in
+  dev|*-dirty) fail "the running release reports $running, which names no commit" ;;
+esac
+rollback=$(git rev-parse --verify --quiet "$running^{commit}") \
+  || fail "running version $running is not a commit in this repository"
+echo "rollback SHA: $rollback (running version $running)"
+git checkout --quiet --detach "$target"
+dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist')
+[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first:
+$dirty"
+v=$(git describe --tags --always)${dirty:+-dirty}   # the Makefile's VERSION
+make docker
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed"
+echo "built: mealquest:$v"
+
+# deploy/RUNBOOK.md "Prerequisites": deploy/.env sets every variable of the
+# target's .env.example, and every secret is mode 0600.
+missing=$(comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \
+                   <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u))
+[ -z "$missing" ] || fail "deploy/.env is missing: $missing"
+loose=$(sudo find deploy/secrets -type f ! -perm 600)
+[ -z "$loose" ] || fail "secrets not mode 0600: $loose"
+
+# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 2: a verified
+# pre-deploy restore point. The scripts are CTXD "Count check"; /tmp is
+# temporary, so they are written before every check.
+cat > /tmp/mqcount.py <<'EOF'
+import sqlite3, sys, hashlib
+c = sqlite3.connect(sys.argv[1])
+tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
+for t in ("dinners", "week_plans", "schema_migrations"):
+    print(t, c.execute(f"select count(*) from {t}").fetchone()[0])
+h = hashlib.sha256()
+for t in tabs:
+    for row in c.execute(f"select * from \"{t}\" order by rowid"):
+        h.update(repr(row).encode())
+print("tables", len(tabs), "content-sha256", h.hexdigest()[:16])
+print("integrity", c.execute("pragma integrity_check").fetchone()[0])
+EOF
+cat > /tmp/mqsnap.py <<'EOF'
+import os, sqlite3, sys
+src, dst = sys.argv[1], sys.argv[2]
+if os.path.exists(dst):
+    sys.exit(f"{dst} already exists")
+s = sqlite3.connect(f"file:{src}?mode=ro", uri=True)
+d = sqlite3.connect(dst)
+s.backup(d)
+s.close()
+result = d.execute("pragma integrity_check").fetchall()
+d.close()
+if result != [("ok",)]:
+    sys.exit(f"snapshot integrity_check failed: {result}")
+print("snapshot", dst, "integrity ok")
+EOF
+point=$(
+set -eu
+point=
+trap 'st=$?
+rm -rf /tmp/mqcheck || st=1
+sudo sh -c "rm -f $D/predeploy-check.db*" || st=1
+if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" >&2; echo "$point"
+else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi
+exit "$st"' EXIT
+exec 3>&1 1>&2   # block output to the journal; only the restore point to $point
+rm -rf /tmp/mqcheck
+mkdir /tmp/mqcheck
+# a. Replication running, no log errors (grep: 0 match, 1 none, 2+ error), newest LTX listed
+id=$("${COMPOSE[@]}" ps --status running --quiet litestream) || fail "docker compose ps failed"
+[ -n "$id" ] || fail "litestream is not running"
+"${COMPOSE[@]}" logs --since 24h litestream > /tmp/mqcheck/litestream.log || fail "docker compose logs failed"
+gs=0
+grep -i error /tmp/mqcheck/litestream.log > /tmp/mqcheck/litestream.errors || gs=$?
+case "$gs" in
+  0) cat /tmp/mqcheck/litestream.errors; fail "litestream logged errors in the last 24h";;
+  1) ;;
+  *) fail "the litestream log scan failed (status $gs)";;
+esac
+"${COMPOSE[@]}" exec -T litestream \
+  litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
+# b. Restore replica to a scratch file (live files untouched), move off the volume, snapshot live
+"${COMPOSE[@]}" run --rm --no-deps litestream \
+  restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db
+sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/"
+sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db
+sudo chown -R "$(id -un)" /tmp/mqcheck
+sudo ls -ln "$D"
+# mealquest.db-wal and -shm owned by 10001, never 0 (CTXD "Count check")
+root=$(sudo find "$D" -maxdepth 1 \( -name mealquest.db-wal -o -name mealquest.db-shm \) -uid 0)
+[ -z "$root" ] || fail "owned by root, not 10001 (CTXD RUNBOOKS.MD \"Count check\"): $root"
+# c. Restored replica: integrity ok and identical counts/content-sha256 to the live snapshot
+python3 /tmp/mqcount.py /tmp/mqcheck/predeploy-check.db > /tmp/mqcheck/restored.txt
+python3 /tmp/mqcount.py /tmp/mqcheck/live.db > /tmp/mqcheck/live.txt
+cat /tmp/mqcheck/restored.txt
+grep -qx 'integrity ok' /tmp/mqcheck/restored.txt || fail "the restored replica failed integrity_check"
+diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot"
+# d. Restore point: after every check, before the restart
+point=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+exec 1>&3
+)
+
+# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 3:
+# boot-required configuration, secrets read the way the service reads them.
+[ -f deploy/.env ] || fail "missing deploy/.env"
+cid=$(grep -m 1 -E '^MEALQUEST_GOOGLE_CLIENT_ID=' deploy/.env | cut -d = -f 2- | tr -d '"' | tr -d "'") \
+  || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env"
+[ -n "$cid" ] || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env"
+[ -e deploy/secrets/google_client_secret ] || fail "missing deploy/secrets/google_client_secret"
+[ -s deploy/secrets/google_client_secret ] || fail "deploy/secrets/google_client_secret is empty"
+id=$("${COMPOSE[@]}" ps --status running --quiet mealquest) || fail "docker compose ps failed"
+[ -n "$id" ] || fail "the mealquest container is not running: start it (the check reads secrets the way the service does)"
+docker exec "$id" sh -c 'test -s /run/secrets/google_client_secret && test -r /run/secrets/google_client_secret' \
+  || fail "google_client_secret is not readable by the service user inside the container"
+echo "boot-required configuration present"
+
+# deploy/RUNBOOK.md "2. Ship and restart".
+"${COMPOSE[@]}" pull litestream searxng
+docker tag "mealquest:$v" mealquest-mealquest
+restarted=yes
+sudo systemctl restart mealquest
+for s in mealquest litestream searxng; do
+  id=$("${COMPOSE[@]}" ps --status running --quiet "$s")
+  [ -n "$id" ] || fail "$s is not running"
+done
+id=$("${COMPOSE[@]}" ps --quiet mealquest)
+health=$(docker inspect -f '{{.State.Health.Status}}' "$id")
+[ "$health" = healthy ] || fail "mealquest is $health, not healthy"
+echo "restarted on: mealquest:$v"
+
+# deploy/RUNBOOK.md "3. Verify the deployed version and a real screen".
+for base in "http://$bind:8090" "$PUBLIC"; do
+  got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+    || fail "$base/health did not return a version"
+  [ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v"
+  for path in / /plan; do
+    page=$(curl -fsS "$base$path") || fail "$base$path did not return 200"
+    grep -q '<div id="root"></div>' <<< "$page" || fail "$base$path is not the PWA shell"
+    js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true
+    [ -n "$js" ] || fail "$base$path references no built /assets/ script"
+    type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200"
+    case "$type" in
+      text/javascript*) ;;
+      *) fail "$base$js is $type, not JavaScript" ;;
+    esac
+  done
+  echo "$base: version $got, PWA shell and $js served"
+done
+
+deployed=$(git rev-parse HEAD)
+echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) previous=$rollback deployed=$deployed restore_point=$point" >> "$RECORD"
+echo "deployed and verified: $v ($deployed), previous $rollback, restore point $point"
@@ -0,0 +1,23 @@
+# Deploys origin/main to production when it is new and its CI passed
+# (/usr/local/sbin/mealquest-auto-deploy, from deploy/auto-deploy.sh).
+# Triggered by mealquest-auto-deploy.timer; output goes to the journal.
+# Installing it and the manual override: deploy/RUNBOOK.md "Automated deploys".
+#
+# ExecCondition= skips the run while mealquest.service is not active, so a
+# stack stopped on purpose (a database or media restore) is never redeployed
+# under the operator. It is deliberately not Requires=/Requisite= on
+# mealquest.service: those would start the stack, or stop this run when the
+# deploy itself restarts mealquest.service. Runs as overseer, who owns the
+# checkout and uses sudo for the runbook's privileged steps.
+[Unit]
+Description=MealQuest automated deploy of main
+Wants=network-online.target
+After=network-online.target docker.service mealquest.service
+
+[Service]
+Type=oneshot
+User=overseer
+Group=overseer
+WorkingDirectory=/home/overseer/mealquest
+ExecCondition=/usr/bin/systemctl is-active --quiet mealquest.service
+ExecStart=/usr/local/sbin/mealquest-auto-deploy
@@ -0,0 +1,17 @@
+# Checks for a new main every 15 minutes (plus up to 5 minutes' random
+# delay), starting 2 minutes after the timer starts. OnUnitActiveSec= alone
+# never fires before the service's first run, so OnActiveSec= starts the
+# cycle at boot and at `enable --now`. Persistent= is set as the task
+# specified; systemd applies it only to OnCalendar= triggers, so with these
+# monotonic triggers it has no effect.
+[Unit]
+Description=Periodic MealQuest automated deploy
+
+[Timer]
+OnActiveSec=2min
+OnUnitActiveSec=15min
+RandomizedDelaySec=5min
+Persistent=true
+
+[Install]
+WantedBy=timers.target