diff --git a/AGENTS.md b/AGENTS.md index d306043..afee245 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,7 +84,7 @@ behaviour, exercise it (hit the endpoint, render the page). | `internal/*` (planned) | `auth`, `db`, `store`, `secretbox`, `household`, `dinners`, `recipes`, `ai/llm`, `ai/harness`, `ai/skills`, `suggest`, `plan`, `shopping`, `ideas`, `gcal`, `notify`, `push`, `chat`, `jobs`, `scheduler`. | | `migrations/` | SQL migrations (planned). | | `web/` | React PWA source. | -| `deploy/` | Deployment files (planned). | +| `deploy/` | Deployment files: compose stack, systemd units, [deploy runbook](deploy/RUNBOOK.md) and the automated deploy (`auto-deploy.sh` with `systemd/mealquest-auto-deploy.*`). | | `docs/` | Project documentation, including [ARCHITECTURE.md](docs/ARCHITECTURE.md). | | `design/` | Requirements and UI handoff. **Read-only reference** — do not edit. | | `Dockerfile` | Multi-stage image build, at the repository root. | diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md new file mode 100644 index 0000000..f79b41d --- /dev/null +++ b/deploy/RUNBOOK.md @@ -0,0 +1,357 @@ +# MealQuest deploy runbook + +The one way to deploy MealQuest to production: the docker compose project +`mealquest` on office-srvr `10.60.1.14` (see +[ARCHITECTURE.md](../docs/ARCHITECTURE.md#topology)). Production runs `main` +or a release tag; only `overseer` merges `dev` → `main`. Every deploy builds +on the host, from the checked-out commit, with `make docker`. + +Host setup (clone, firewall, the systemd unit, Traefik, DNS, Litestream) and +the database restore are in CTXD project `mealquest`, `RUNBOOKS.MD`. This +file is the build, restart, verification and rollback procedure; where it +uses a CTXD section, it names it. + +All commands run on `10.60.1.14` as `overseer`, in bash, from the checkout +`/home/overseer/mealquest`. A block that ends with `FAILED` or a non-zero +`$?` stops the deploy: do not go on to the next step. + +## How the image carries its version + +`make docker` runs `docker build --build-arg VERSION=` and tags the +image `mealquest:`, where `` is +`git describe --tags --always` of the checkout (a release tag such as +`v1.0.0`, otherwise the short SHA), with `-dirty` added when a tracked file +is modified. The [Dockerfile](../Dockerfile) links it into the binary, and +`/health` reports it: + +```json +{"status":"ok","version":"v1.0.0"} +``` + +Never pass `--build-arg VERSION` by hand, and never build with +`docker compose build` or `docker compose up --build`: compose passes no +version, so that image reports `dev`. + +The compose service `mealquest` runs the image named `mealquest-mealquest` +(compose's name for the image it would build; check with +`docker compose -f deploy/compose.yaml config --images mealquest`). It +already exists, so `docker compose up` uses it and never builds. A deploy +points that name at the `make docker` image with `docker tag`. Every +`mealquest:` image is kept for rollback; do not prune them. + +## Prerequisites + +- `make`, `git`, `curl`, `python3` and Docker Engine with Compose on the + host (`make` is not in the base install: `sudo apt-get install make`). +- `deploy/.env` (git-ignored, owner `overseer`, mode 0600) sets every + variable [`deploy/.env.example`](.env.example) sets, with + `MEALQUEST_BIND_ADDRESS=10.60.1.14`. It must not set the variables + `deploy/compose.yaml` fixes (`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR`, + `MEALQUEST_SEARXNG_URL` and the `*_FILE` variables). Creating it: CTXD + `RUNBOOKS.MD` "Stack and first deploy" > "3. Environment file". +- `deploy/secrets/` (git-ignored, mode 0700) holds `secret_key`, + `oidc_client_secret`, `google_client_secret`, `vapid_private_key` and + `litestream_key`, each owned by `10001:10001` with mode 0600, and + `searxng.env`, owned by `overseer` with mode 0600. Creating them: CTXD + `RUNBOOKS.MD` "Stack and first deploy" > "4. Secrets". +- The compose file is the checkout's [`deploy/compose.yaml`](compose.yaml); + the systemd unit [`mealquest.service`](systemd/mealquest.service) starts + and stops it and is the only way the stack starts. +- DNS and Traefik already route `https://mealquest.cubecraftlabs.com` to + `10.60.1.14:8090` (CTXD `RUNBOOKS.MD` "Reverse proxy (Traefik)", "DNS"). + +Check `deploy/.env` and the secrets before every deploy (a new release may +add a variable; the service refuses to boot without it). Both commands must +print nothing: + +```sh +cd /home/overseer/mealquest +comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \ + <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u) # variables missing from deploy/.env +sudo find deploy/secrets -type f ! -perm 600 # secrets not mode 0600 +``` + +## Deploy + +### 1. Build + +Record the release production is running now, check out the new release +and build its image. Nothing restarts. + +The rollback target is the version the running service reports on +`/health`, read before the checkout changes, never the checkout's `HEAD`: +after a failed or deferred deploy the checkout can be ahead of production. +The block resolves that version to a commit and prints it as the rollback +SHA. A running version of `dev` or `*-dirty` names no commit: the block +fails, and the rollback target must be established by hand before going on. + +The checkout must be clean (the same test as the Makefile's `GIT_DIRTY`: +no modified tracked file outside the generated `internal/frontend/dist`). +The block computes the image tag exactly as `make docker` does, `-dirty` +included, and checks that image exists after the build. + +```sh +cd /home/overseer/mealquest && ( +set -euo pipefail +fail() { echo "$*" >&2; exit 1; } +built= +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$built" ]; then echo "built: $built" +else [ "$st" -ne 0 ] || st=1; echo "build FAILED (status $st)" >&2; fi +exit "$st"' EXIT +bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-) +[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS" +running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \ + || fail "http://$bind:8090/health did not return a version: no running release to roll back to" +case "$running" in + dev|*-dirty) fail "the running release reports $running, which names no commit" ;; +esac +git fetch --tags origin +rollback=$(git rev-parse --verify --quiet "$running^{commit}") \ + || fail "running version $running is not a commit in this repository" +echo "rollback SHA: $rollback (running version $running)" # RECORD it for rollback +git checkout --detach "refs/remotes/origin/main^{commit}" # or the release tag, e.g. v1.0.0 +dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist') +[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first: +$dirty" +v=$(git describe --tags --always)${dirty:+-dirty} # the Makefile's VERSION +make docker +docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed" +built="mealquest:$v" +) +``` + +### 2. Ship and restart + +1. **Pre-restart gates.** Run CTXD `RUNBOOKS.MD` "Routine deploy and + rollback" > "Deploy" step 2 (a verified pre-deploy restore point; record + it with the rollback SHA) and step 3 (boot-required configuration). The + restart applies migrations the previous release refuses, so without that + restore point there is no rollback. If either fails, do not restart: + production stays on the previous release. +2. **Point the stack at the new image and restart.** `docker compose pull` + fetches the pinned `litestream` and `searxng` images before anything + stops; `mealquest` is never pulled. The restart applies the firewall, + then runs `down` and `up -d`, and returns once `mealquest` passes its + healthcheck (`/health` inside the container). + +```sh +cd /home/overseer/mealquest && ( +set -euo pipefail +fail() { echo "$*" >&2; exit 1; } +up= +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$up" ]; then echo "restarted on: $up" +else [ "$st" -ne 0 ] || st=1; echo "restart FAILED (status $st)" >&2; fi +exit "$st"' EXIT +v=$(git describe --tags --always) +docker compose -f deploy/compose.yaml pull litestream searxng +docker tag "mealquest:$v" mealquest-mealquest +sudo systemctl restart mealquest +for s in mealquest litestream searxng; do + id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") + [ -n "$id" ] || fail "$s is not running" +done +id=$(docker compose -f deploy/compose.yaml ps --quiet mealquest) +health=$(docker inspect -f '{{.State.Health.Status}}' "$id") +[ "$health" = healthy ] || fail "mealquest is $health, not healthy" +up="mealquest:$v" +) +``` + +On failure read `journalctl -u mealquest` and +`docker compose -f deploy/compose.yaml logs mealquest`: a configuration +error names the variable. Fix it or roll back. + +### 3. Verify the deployed version and a real screen + +A `/health` 200 alone proves only that some build is up. This step checks, +on the LAN address and through Traefik, that: + +- `/health` reports exactly the checkout's `git describe --tags --always`; +- `/` and the Plan screen's route `/plan` serve the built PWA shell (the + React root and a hashed `/assets/` script, not the placeholder page an + unbuilt embed serves); +- that script is served as JavaScript. + +```sh +cd /home/overseer/mealquest && ( +set -euo pipefail +fail() { echo "$*" >&2; exit 1; } +want= +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$want" ]; then echo "deployed and verified: $want ($(git rev-parse HEAD))" +else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): the deploy is not verified" >&2; fi +exit "$st"' EXIT +v=$(git describe --tags --always) +for base in http://10.60.1.14:8090 https://mealquest.cubecraftlabs.com; do + got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \ + || fail "$base/health did not return a version" + [ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v" + for path in / /plan; do + page=$(curl -fsS "$base$path") || fail "$base$path did not return 200" + grep -q '
' <<< "$page" || fail "$base$path is not the PWA shell" + js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true + [ -n "$js" ] || fail "$base$path references no built /assets/ script" + type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200" + case "$type" in + text/javascript*) ;; + *) fail "$base$js is $type, not JavaScript" ;; + esac + done + echo "$base: version $got, PWA shell and $js served" +done +want=$v +) +``` + +- `deployed and verified: ()` and `$?` = `0` → open + `https://mealquest.cubecraftlabs.com` in a browser: sign in and the Home + screen renders with the household's data. Then record the deployed + version and SHA, the rollback SHA, the pre-deploy restore point and the + date in the release's Maestro task. +- `verification FAILED`, any error or a non-zero `$?` → the deploy is not + done. A version mismatch means the stack runs a different image than the + checkout: repeat step 2, or roll back. + +## Rollback to a previous release + +Rolling back points `mealquest-mealquest` at the previous release's +`mealquest:` image again; that release's checkout keeps the +compose file and the version check consistent with it. + +1. **Check whether the rollback crosses a migration** with CTXD + `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a previous + SHA" (its first block), run against the rollback SHA that deploy step 1 + recorded. If it prints versions unknown to the rollback SHA, restore the + database to the pre-deploy restore point first (CTXD `RUNBOOKS.MD` + "Restore the database", with `-timestamp `); changes + since then are lost, so tell the household. +2. **Check out the rollback SHA and retag its image.** An image built before + this runbook was not tagged `mealquest:`; `make docker` builds + it from the same commit. + +```sh +cd /home/overseer/mealquest && ( +set -euo pipefail +fail() { echo "$*" >&2; exit 1; } +P= # recorded by step 1, from the running release's /health +back= +trap 'st=$? +if [ "$st" -eq 0 ] && [ -n "$back" ]; then echo "rolled back to: $back" +else [ "$st" -ne 0 ] || st=1; echo "rollback FAILED (status $st)" >&2; fi +exit "$st"' EXIT +git checkout --detach "$P" +v=$(git describe --tags --always) +docker image inspect "mealquest:$v" > /dev/null 2>&1 || make docker +docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v" +docker tag "mealquest:$v" mealquest-mealquest +sudo systemctl restart mealquest +back="mealquest:$v" +) +``` + +3. **Run step 3 above** unchanged. The rollback is done only when it prints + `deployed and verified:` with the previous release's version. + +## Automated deploys + +[`auto-deploy.sh`](auto-deploy.sh), installed as +`/usr/local/sbin/mealquest-auto-deploy`, runs this runbook's deploy unattended: +step 1 (build), the prerequisite `deploy/.env` and secrets check, CTXD +`RUNBOOKS.MD` "Deploy" steps 2 (pre-deploy restore point) and 3 +(boot-required configuration), step 2 (ship and restart) and step 3 +(verify). No check is skipped or weakened, and it stops at the first +failure. Every failure before the restart leaves production on the previous +release. + +### What runs, and when + +[`mealquest-auto-deploy.timer`](systemd/mealquest-auto-deploy.timer) starts +[`mealquest-auto-deploy.service`](systemd/mealquest-auto-deploy.service) +at every quarter hour (`OnCalendar=*-*-* *:0/15:00`), plus up to 5 minutes' +random delay; a check missed while the host was down runs as soon as it is +back (`Persistent=true`). The service runs the script as `overseer` from the +checkout, and only while `mealquest.service` is active: a stack stopped on +purpose (a database or media restore) is skipped, not redeployed. Output goes +to the journal (`journalctl -u mealquest-auto-deploy`). + +Each run fetches `origin` and compares `origin/main` with the last +`deployed=` SHA in the deploy record `/home/overseer/mealquest-auto-deploy.log`: + +| `origin/main` | Result | +| --- | --- | +| is the last deployed SHA | No-op: exit 0, no output. | +| CI (Gitea commit status) `pending` | Exit 0, no output; the next run checks again. | +| CI `failure`, `error`, `warning`, or no status yet | Error in the journal, exit 1, nothing deployed. | +| the last record line is `FAILED` for this SHA | Error in the journal, exit 1, nothing deployed: a failed deploy is never retried by the timer. | +| CI `success` | Deploy. | + +The CI status is read from +`https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits//status` +(public; no credential). A commit merged moments before a run can briefly +report no status, which fails that run; the next run reads it again. + +A deploy appends one line to the record: + +```text +2026-10-12T15:04:05Z previous= deployed= restore_point=2026-10-12T15:01:59Z +``` + +`previous` is the rollback SHA (from the running release's `/health`, +never the checkout) and `restore_point` the verified pre-deploy restore +point: record both in the release's Maestro task. A failed deploy appends +` FAILED target= status=` and logs +`auto-deploy FAILED (status )` to the journal, adding whether it failed +before the restart (production untouched) or after it (read +`journalctl -u mealquest`, then fix or roll back). Either way the checkout is +left at the target commit; the running release on `/health` stays the +rollback target. + +### Install on the host + +The script and units are installed copies: repeat this whenever a release +changes `deploy/auto-deploy.sh` or the units. From the checkout, at a release +that contains them: + +```sh +cd /home/overseer/mealquest +sudo install -m 0755 deploy/auto-deploy.sh /usr/local/sbin/mealquest-auto-deploy +sudo install -m 0644 deploy/systemd/mealquest-auto-deploy.service \ + deploy/systemd/mealquest-auto-deploy.timer /etc/systemd/system/ +sudo systemctl daemon-reload +``` + +The timer refuses to deploy until the record holds a deploy, so seed it with +one manual deploy of the release production should run (below), then start +the timer: + +```sh +sudo /usr/local/sbin/mealquest-auto-deploy --ref refs/remotes/origin/main +sudo systemctl enable --now mealquest-auto-deploy.timer +systemctl list-timers mealquest-auto-deploy.timer +``` + +To stop automated deploys: `sudo systemctl disable --now mealquest-auto-deploy.timer`. + +### Manual override + +```sh +sudo /usr/local/sbin/mealquest-auto-deploy --ref +``` + +deploys an explicit ref (a SHA, tag or `origin/`) through the same +blocks, bypassing the CI and no-op gates: for a roll-forward fix, a re-run +after a `FAILED` record, or seeding the record. Run as root, the script +re-executes itself as `overseer`; it holds a lock, so it never runs alongside +the timer's run. It prints `deployed and verified:` and exits 0 only when +step 3 passed. + +### Rollback + +Rollback is not automated: follow "Rollback to a previous release" above +and CTXD `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a +previous SHA", using the `previous` SHA and `restore_point` from the record. +The timer does not undo a manual rollback: `origin/main` is still the last +recorded deploy, so its runs are no-ops until `main` moves, and a new `main` +then deploys as usual. Disable the timer if the rollback must hold. diff --git a/deploy/auto-deploy.sh b/deploy/auto-deploy.sh new file mode 100755 index 0000000..fa4876f --- /dev/null +++ b/deploy/auto-deploy.sh @@ -0,0 +1,251 @@ +#!/usr/bin/env bash +# MealQuest automated deploy: deploy/RUNBOOK.md, run unattended. +# +# mealquest-auto-deploy deploy origin/main if it is new and CI passed +# mealquest-auto-deploy --ref deploy now, bypassing the CI and no-op gates +# +# Installed as /usr/local/sbin/mealquest-auto-deploy and run as overseer by +# mealquest-auto-deploy.service (deploy/systemd); install, semantics and the +# manual override: deploy/RUNBOOK.md "Automated deploys". Run as root (sudo) +# it re-executes itself as overseer. +# +# Without --ref: exit 0 with no output when origin/main is the last deployed +# SHA in the deploy record or its CI is pending; exit 1, deploying nothing, +# when its CI failed or reports no status, or when the last attempt to deploy +# that SHA failed. Every deploy runs the runbook's blocks in order and stops +# at the first failure; nothing before the restart touches production. A +# verified deploy appends one line to the deploy record: +# previous= deployed= restore_point= +# a failed one appends: +# FAILED target= status= +set -euo pipefail + +CHECKOUT=/home/overseer/mealquest +RECORD=/home/overseer/mealquest-auto-deploy.log +LOCK=/home/overseer/mealquest-auto-deploy.lock +STATUS_API=https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits +PUBLIC=https://mealquest.cubecraftlabs.com +D=/var/lib/docker/volumes/mealquest-data/_data +COMPOSE=(docker compose -f deploy/compose.yaml) + +fail() { echo "$*" >&2; exit 1; } + +if [ "$(id -u)" -eq 0 ]; then + exec sudo -u overseer -H -- "$0" "$@" +fi +[ "$(id -un)" = overseer ] || fail "run as overseer (or root), not $(id -un)" + +ref= +case "$#" in + 0) ;; + 2) [ "$1" = --ref ] && [ -n "$2" ] || fail "usage: $0 [--ref ]"; ref=$2 ;; + *) fail "usage: $0 [--ref ]" ;; +esac + +exec 9>> "$LOCK" +flock -n 9 || fail "another mealquest-auto-deploy run holds $LOCK" + +cd "$CHECKOUT" +git fetch --quiet --tags origin || fail "git fetch origin failed" + +if [ -n "$ref" ]; then + target=$(git rev-parse --verify --quiet "$ref^{commit}") || fail "$ref is not a commit in $CHECKOUT" +else + # The full remote-tracking ref: a bare origin/main resolves a tag or local + # branch of that name first, which could deploy a commit not on main. + target=$(git rev-parse --verify --quiet "refs/remotes/origin/main^{commit}") \ + || fail "refs/remotes/origin/main is not a commit in $CHECKOUT" + [ -f "$RECORD" ] || fail "no deploy record at $RECORD: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")" + last=$(grep -oE 'deployed=[0-9a-f]{40}' "$RECORD" | tail -n 1 | cut -d = -f 2) || true + [ -n "$last" ] || fail "$RECORD records no deploy: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")" + [ "$target" != "$last" ] || exit 0 + if tail -n 1 "$RECORD" | grep -q " FAILED target=$target status="; then + fail "auto-deploy of $target already failed (see $RECORD and the journal): fix it, then deploy with --ref" + fi + state=$(curl -fsS --max-time 30 "$STATUS_API/$target/status" \ + | python3 -c 'import json, sys; d = json.load(sys.stdin); print(d["state"] if d["total_count"] else "absent")') \ + || fail "could not read the CI status of $target from $STATUS_API" + case "$state" in + success) ;; + pending) exit 0 ;; + *) fail "CI for origin/main $target is $state, not success: not deploying" ;; + esac +fi + +echo "auto-deploy: deploying $target${ref:+ (--ref $ref)}" +restarted= +W= +on_exit() { + local st=$? + [ -z "$W" ] || rm -rf "$W" || st=1 + if [ "$st" -ne 0 ]; then + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) FAILED target=$target status=$st" >> "$RECORD" + if [ -n "$restarted" ]; then + echo "auto-deploy FAILED (status $st) after the restart: production may be down or unverified; read journalctl -u mealquest, then fix or roll back (deploy/RUNBOOK.md)" >&2 + else + echo "auto-deploy FAILED (status $st): production untouched" >&2 + fi + fi + exit "$st" +} +trap on_exit EXIT + +# deploy/RUNBOOK.md "Deploy" > "1. Build": record the rollback SHA from the +# RUNNING release, check out the target, reject a dirty checkout, build. +bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-) || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS" +[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS" +running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \ + || fail "http://$bind:8090/health did not return a version: no running release to roll back to" +case "$running" in + dev|*-dirty) fail "the running release reports $running, which names no commit" ;; +esac +rollback=$(git rev-parse --verify --quiet "$running^{commit}") \ + || fail "running version $running is not a commit in this repository" +echo "rollback SHA: $rollback (running version $running)" +git checkout --quiet --detach "$target" +dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist') +[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first: +$dirty" +v=$(git describe --tags --always)${dirty:+-dirty} # the Makefile's VERSION +make docker +docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed" +echo "built: mealquest:$v" + +# deploy/RUNBOOK.md "Prerequisites": deploy/.env sets every variable of the +# target's .env.example, and every secret is mode 0600. +missing=$(comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \ + <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u)) +[ -z "$missing" ] || fail "deploy/.env is missing: $missing" +loose=$(sudo find deploy/secrets -type f ! -perm 600) +[ -z "$loose" ] || fail "secrets not mode 0600: $loose" + +# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 2: a verified +# pre-deploy restore point. The scripts are CTXD "Count check". They and the +# check files live in a private mktemp directory (mode 0700), not at shared +# /tmp paths, and python runs with -I: the snapshot runs as root, and a +# shared directory on its import path would let any local user run code as +# root. +W=$(mktemp -d) +cat > "$W/mqcount.py" <<'EOF' +import sqlite3, sys, hashlib +c = sqlite3.connect(sys.argv[1]) +tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")] +for t in ("dinners", "week_plans", "schema_migrations"): + print(t, c.execute(f"select count(*) from {t}").fetchone()[0]) +h = hashlib.sha256() +for t in tabs: + for row in c.execute(f"select * from \"{t}\" order by rowid"): + h.update(repr(row).encode()) +print("tables", len(tabs), "content-sha256", h.hexdigest()[:16]) +print("integrity", c.execute("pragma integrity_check").fetchone()[0]) +EOF +cat > "$W/mqsnap.py" <<'EOF' +import os, sqlite3, sys +src, dst = sys.argv[1], sys.argv[2] +if os.path.exists(dst): + sys.exit(f"{dst} already exists") +s = sqlite3.connect(f"file:{src}?mode=ro", uri=True) +d = sqlite3.connect(dst) +s.backup(d) +s.close() +result = d.execute("pragma integrity_check").fetchall() +d.close() +if result != [("ok",)]: + sys.exit(f"snapshot integrity_check failed: {result}") +print("snapshot", dst, "integrity ok") +EOF +point=$( +set -eu +point= +trap 'st=$? +rm -rf "$W" || st=1 +sudo sh -c "rm -f $D/predeploy-check.db*" || st=1 +if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" >&2; echo "$point" +else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi +exit "$st"' EXIT +exec 3>&1 1>&2 # block output to the journal; only the restore point to $point +mkdir "$W/mqcheck" +# a. Replication running, no log errors (grep: 0 match, 1 none, 2+ error), newest LTX listed +id=$("${COMPOSE[@]}" ps --status running --quiet litestream) || fail "docker compose ps failed" +[ -n "$id" ] || fail "litestream is not running" +"${COMPOSE[@]}" logs --since 24h litestream > "$W/mqcheck/litestream.log" || fail "docker compose logs failed" +gs=0 +grep -i error "$W/mqcheck/litestream.log" > "$W/mqcheck/litestream.errors" || gs=$? +case "$gs" in + 0) cat "$W/mqcheck/litestream.errors"; fail "litestream logged errors in the last 24h";; + 1) ;; + *) fail "the litestream log scan failed (status $gs)";; +esac +"${COMPOSE[@]}" exec -T litestream \ + litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db +# b. Restore replica to a scratch file (live files untouched), move off the volume, snapshot live +"${COMPOSE[@]}" run --rm --no-deps litestream \ + restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db +sudo sh -c "mv $D/predeploy-check.db* $W/mqcheck/" +sudo python3 -I "$W/mqsnap.py" "$D/mealquest.db" "$W/mqcheck/live.db" +sudo chown -R "$(id -un)" "$W/mqcheck" +sudo ls -ln "$D" +# mealquest.db-wal and -shm owned by 10001, never 0 (CTXD "Count check") +root=$(sudo find "$D" -maxdepth 1 \( -name mealquest.db-wal -o -name mealquest.db-shm \) -uid 0) +[ -z "$root" ] || fail "owned by root, not 10001 (CTXD RUNBOOKS.MD \"Count check\"): $root" +# c. Restored replica: integrity ok and identical counts/content-sha256 to the live snapshot +python3 -I "$W/mqcount.py" "$W/mqcheck/predeploy-check.db" > "$W/mqcheck/restored.txt" +python3 -I "$W/mqcount.py" "$W/mqcheck/live.db" > "$W/mqcheck/live.txt" +cat "$W/mqcheck/restored.txt" +grep -qx 'integrity ok' "$W/mqcheck/restored.txt" || fail "the restored replica failed integrity_check" +diff "$W/mqcheck/restored.txt" "$W/mqcheck/live.txt" || fail "the restored replica differs from the live snapshot" +# d. Restore point: after every check, before the restart +point=$(date -u +%Y-%m-%dT%H:%M:%SZ) +exec 1>&3 +) + +# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 3: +# boot-required configuration, secrets read the way the service reads them. +[ -f deploy/.env ] || fail "missing deploy/.env" +cid=$(grep -m 1 -E '^MEALQUEST_GOOGLE_CLIENT_ID=' deploy/.env | cut -d = -f 2- | tr -d '"' | tr -d "'") \ + || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env" +[ -n "$cid" ] || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env" +[ -e deploy/secrets/google_client_secret ] || fail "missing deploy/secrets/google_client_secret" +[ -s deploy/secrets/google_client_secret ] || fail "deploy/secrets/google_client_secret is empty" +id=$("${COMPOSE[@]}" ps --status running --quiet mealquest) || fail "docker compose ps failed" +[ -n "$id" ] || fail "the mealquest container is not running: start it (the check reads secrets the way the service does)" +docker exec "$id" sh -c 'test -s /run/secrets/google_client_secret && test -r /run/secrets/google_client_secret' \ + || fail "google_client_secret is not readable by the service user inside the container" +echo "boot-required configuration present" + +# deploy/RUNBOOK.md "2. Ship and restart". +"${COMPOSE[@]}" pull litestream searxng +docker tag "mealquest:$v" mealquest-mealquest +restarted=yes +sudo systemctl restart mealquest +for s in mealquest litestream searxng; do + id=$("${COMPOSE[@]}" ps --status running --quiet "$s") + [ -n "$id" ] || fail "$s is not running" +done +id=$("${COMPOSE[@]}" ps --quiet mealquest) +health=$(docker inspect -f '{{.State.Health.Status}}' "$id") +[ "$health" = healthy ] || fail "mealquest is $health, not healthy" +echo "restarted on: mealquest:$v" + +# deploy/RUNBOOK.md "3. Verify the deployed version and a real screen". +for base in "http://$bind:8090" "$PUBLIC"; do + got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \ + || fail "$base/health did not return a version" + [ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v" + for path in / /plan; do + page=$(curl -fsS "$base$path") || fail "$base$path did not return 200" + grep -q '
' <<< "$page" || fail "$base$path is not the PWA shell" + js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true + [ -n "$js" ] || fail "$base$path references no built /assets/ script" + type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200" + case "$type" in + text/javascript*) ;; + *) fail "$base$js is $type, not JavaScript" ;; + esac + done + echo "$base: version $got, PWA shell and $js served" +done + +deployed=$(git rev-parse HEAD) +echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) previous=$rollback deployed=$deployed restore_point=$point" >> "$RECORD" +echo "deployed and verified: $v ($deployed), previous $rollback, restore point $point" diff --git a/deploy/systemd/mealquest-auto-deploy.service b/deploy/systemd/mealquest-auto-deploy.service new file mode 100644 index 0000000..671f5ce --- /dev/null +++ b/deploy/systemd/mealquest-auto-deploy.service @@ -0,0 +1,23 @@ +# Deploys origin/main to production when it is new and its CI passed +# (/usr/local/sbin/mealquest-auto-deploy, from deploy/auto-deploy.sh). +# Triggered by mealquest-auto-deploy.timer; output goes to the journal. +# Installing it and the manual override: deploy/RUNBOOK.md "Automated deploys". +# +# ExecCondition= skips the run while mealquest.service is not active, so a +# stack stopped on purpose (a database or media restore) is never redeployed +# under the operator. It is deliberately not Requires=/Requisite= on +# mealquest.service: those would start the stack, or stop this run when the +# deploy itself restarts mealquest.service. Runs as overseer, who owns the +# checkout and uses sudo for the runbook's privileged steps. +[Unit] +Description=MealQuest automated deploy of main +Wants=network-online.target +After=network-online.target docker.service mealquest.service + +[Service] +Type=oneshot +User=overseer +Group=overseer +WorkingDirectory=/home/overseer/mealquest +ExecCondition=/usr/bin/systemctl is-active --quiet mealquest.service +ExecStart=/usr/local/sbin/mealquest-auto-deploy diff --git a/deploy/systemd/mealquest-auto-deploy.timer b/deploy/systemd/mealquest-auto-deploy.timer new file mode 100644 index 0000000..864ea43 --- /dev/null +++ b/deploy/systemd/mealquest-auto-deploy.timer @@ -0,0 +1,13 @@ +# Checks for a new main at every quarter hour (plus up to 5 minutes' random +# delay). Persistent= runs a check at once when the host comes back if a +# scheduled one was missed while it was down. +[Unit] +Description=Periodic MealQuest automated deploy + +[Timer] +OnCalendar=*-*-* *:0/15:00 +RandomizedDelaySec=5min +Persistent=true + +[Install] +WantedBy=timers.target