diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md new file mode 100644 index 0000000..89534f5 --- /dev/null +++ b/deploy/RUNBOOK.md @@ -0,0 +1,318 @@ +# MealQuest deployment runbook + +How MealQuest is deployed on office-srvr (10.60.1.14) and how its data is +backed up and restored. Sections marked *task 876134bc* are completed by +that task (Feature 07 deployment runbook); this file currently records the +reverse-proxy route, the backups and the restore procedure. + +Paths used below: + +| Name | Path | +| --- | --- | +| Checkout on the host | `/home/overseer/mealquest` | +| Data volume (`mealquest-data`) on the host | `/var/lib/docker/volumes/mealquest-data/_data` | +| Same volume inside the containers | `/data` | +| Database | `mealquest.db` (+ `mealquest.db-wal`, `mealquest.db-shm` while open) | +| Media | `media/` in the data volume | +| Backup target (NAS, over SSH) | `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups` | +| Same target over SFTP (Synology chroot) | `/homes/Joshua/mealquest-backups` | + +## Stack and first deploy + +*Completed by task 876134bc.* The compose stack is `deploy/compose.yaml`; +start it with `docker compose -f deploy/compose.yaml up -d --build`. + +## Reverse proxy (Traefik) + +MealQuest is published at by the +shared Traefik instance on 10.60.1.33 (Feature 07, task 0c1dc172). The +route is a dynamic-configuration file that lives on the Traefik host only, +at `/home/overseer/Code/traefik/config/dynamic/mealquest-ccl.yml`. Its exact +contents: + +```yaml +# MealQuest - https://mealquest.cubecraftlabs.com +# Feature 07 task 0c1dc172. Single Go binary (API + embedded PWA) on +# office-srvr 10.60.1.14:8090. HTTP -> HTTPS is handled globally by +# http-conf.yml, so only a websecure router is defined here. +# This file lives on the Traefik host only; the repo copy is recorded in +# deploy/RUNBOOK.md by task 876134bc. +http: + routers: + mealquest: + entryPoints: + - websecure + rule: "Host(`mealquest.cubecraftlabs.com`)" + priority: 100 + middlewares: + - mealquest-headers + service: mealquest + tls: + certResolver: myresolver + domains: + - main: mealquest.cubecraftlabs.com + + services: + mealquest: + loadBalancer: + servers: + - url: "http://10.60.1.14:8090" + + middlewares: + mealquest-headers: + headers: + stsSeconds: 31536000 + stsIncludeSubdomains: true + contentTypeNosniff: true + frameDeny: true +``` + +## Backups + +### Database — Litestream + +The `litestream` compose service streams every committed change to +`mealquest.db` to the NAS over SFTP (`deploy/litestream.yml`). It takes a +full snapshot daily (compaction level 9) and keeps a week of snapshots +plus the incremental LTX files between them. + +One-time setup (already done on 10.60.1.14): + +1. Generate the dedicated keypair, git-ignored, owned by the container + user: + + ```sh + cd /home/overseer/mealquest + ssh-keygen -t ed25519 -N '' -C mealquest-litestream -f deploy/secrets/litestream_key + # append deploy/secrets/litestream_key.pub to joshua@10.60.1.50:~/.ssh/authorized_keys + rm deploy/secrets/litestream_key.pub + sudo chown 10001:10001 deploy/secrets/litestream_key + sudo chmod 0600 deploy/secrets/litestream_key + ``` + +2. The NAS host key is pinned in `deploy/litestream.yml` (`host-key`). + +Check replication — list every LTX file at every level. Level 9 holds the +full snapshots; the default listing (`-level 0`) shows only L0 files: + +```sh +docker compose -f deploy/compose.yaml exec -T litestream \ + litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db +``` + +### Media — nightly rsync + +`mealquest-media-backup.timer` runs `mealquest-media-backup.service` at +03:30 every night. The service runs `/usr/local/sbin/mealquest-media-backup` +(`deploy/systemd/mealquest-media-backup.sh`) as the unprivileged host user +`mealquest` (uid/gid 10001, the app's own user) with no capabilities. +systemd bind-mounts the data volume read-only at `/run/mealquest-data` and +passes the Litestream key in as the credential `nas_key`; uid 10001 cannot +otherwise reach `/var/lib/docker` or `deploy/secrets/`. The script refuses a +`media` path that is a symlink and fails (exit 1) if `media` is missing. + +One-time setup (already done on 10.60.1.14): + +```sh +cd /home/overseer/mealquest +# Host user for the backup; ssh needs a passwd entry for its uid. +sudo groupadd --system --gid 10001 mealquest +sudo useradd --system --uid 10001 --gid 10001 --no-create-home \ + --home-dir /nonexistent --shell /usr/sbin/nologin mealquest +# Pinned NAS host keys, root-owned and world-readable. +sudo install -d -o root -g root -m 0755 /etc/mealquest +sudo install -o root -g root -m 0644 deploy/known_hosts /etc/mealquest/known_hosts +# The media directory; the backup fails if it is ever missing. +sudo install -d -o 10001 -g 10001 -m 0755 /var/lib/docker/volumes/mealquest-data/_data/media +# Script and units. +sudo install -o root -g root -m 0755 deploy/systemd/mealquest-media-backup.sh /usr/local/sbin/mealquest-media-backup +sudo install -o root -g root -m 0644 deploy/systemd/mealquest-media-backup.service \ + deploy/systemd/mealquest-media-backup.timer /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now mealquest-media-backup.timer +``` + +Run it by hand and read the result: + +```sh +sudo systemctl start mealquest-media-backup.service +journalctl -u mealquest-media-backup --no-pager -n 30 +``` + +Restore media by reversing the copy into the (empty or stale) media +directory as uid 10001: + +```sh +sudo rsync -a --chown=10001:10001 \ + -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \ + joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \ + /var/lib/docker/volumes/mealquest-data/_data/media/ +``` + +## Restore the database + +SQLite in WAL mode keeps recent commits in `mealquest.db-wal`, indexed by +`mealquest.db-shm`. **A restored database must never sit beside an old WAL +or shared-memory file**: on open, SQLite would replay the old WAL's pages +onto the restored database and corrupt it or mix in state from the +database it replaced. So all three files are moved aside together, with +Litestream's local state directory `.mealquest.db-litestream`, before the +restore — and the restore runs into an empty volume. + +Run from `/home/overseer/mealquest`: + +```sh +D=/var/lib/docker/volumes/mealquest-data/_data +A=/var/tmp/mealquest-restore-aside-$(date +%Y%m%d%H%M) + +# 1. Stop the whole stack (app and Litestream). Nothing may hold the +# database open; a clean close checkpoints the WAL into mealquest.db. +docker compose -f deploy/compose.yaml down + +# 2. Move the database, its -wal and -shm files and Litestream's state +# aside — all of them, every time. Keep the directory until the +# restored stack is verified. +sudo mkdir "$A" +for f in mealquest.db mealquest.db-wal mealquest.db-shm .mealquest.db-litestream; do + if sudo test -e "$D/$f"; then sudo mv "$D/$f" "$A/"; fi +done +sudo ls -la "$D" # must list only media/ + +# 3. Restore from the NAS. --no-deps keeps the app stopped so it cannot +# create a fresh, empty database first. +docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ + restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db +sudo ls -la "$D" # mealquest.db only: no -wal, no -shm + +# 4. Check the restored file before starting the app (see "Count check"). + +# 5. Start the stack and check health. +docker compose -f deploy/compose.yaml up -d +docker compose -f deploy/compose.yaml ps +curl -s -o /dev/null -w "health %{http_code}\n" http://10.60.1.14:8090/health +``` + +To restore to a point in time, add `-timestamp 2026-10-05T23:00:00Z` (or +`-txid`) to the `restore` command. + +### Count check + +`/tmp/mqcount.py` prints row counts for key tables, the number of +application tables, a SHA-256 over every application row (first 16 hex +digits; Litestream's own `_litestream*` tables are excluded) and +`PRAGMA integrity_check`. Run it on a **copy** of the database, never on the +live file; copy `-wal`/`-shm` along with it when the app is running. + +```python +import sqlite3, sys, hashlib +c = sqlite3.connect(sys.argv[1]) +tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")] +for t in ("dinners", "week_plans", "schema_migrations"): + print(t, c.execute(f"select count(*) from {t}").fetchone()[0]) +h = hashlib.sha256() +for t in tabs: + for row in c.execute(f"select * from \"{t}\" order by rowid"): + h.update(repr(row).encode()) +print("tables", len(tabs), "content-sha256", h.hexdigest()[:16]) +print("integrity", c.execute("pragma integrity_check").fetchone()[0]) +``` + +```sh +rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck +sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck +python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db +``` + +Before and after must match exactly. + +## Restore drill record — 2026-10-05 + +Drill on 10.60.1.14 against the live (pre-launch, empty-household) +database, Litestream 0.5.17. Commands as in "Restore the database"; +output trimmed only of compose progress lines. + +Replica before the drill (`ltx -level all`; level 9 is the full snapshot): + +``` +level min_txid max_txid size created +0 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z +1 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z +1 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z +2 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z +2 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z +3 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z +9 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z +``` + +1. `docker compose -f deploy/compose.yaml down` — both containers stopped + and removed. The volume then held `mealquest.db`, + `.mealquest.db-litestream/` and `media/`; the clean close had + checkpointed and removed `-wal` and `-shm`. +2. Before (copy of the stopped database): + + ``` + dinners 0 + week_plans 0 + schema_migrations 7 + tables 26 content-sha256 836e230e16c58dc7 + integrity ok + ``` + +3. Moved aside to `/var/tmp/mealquest-restore-aside-20261005`: + `mealquest.db` and `.mealquest.db-litestream` (no `-wal`/`-shm` existed + after the clean stop; the loop moves them whenever they do). Volume + afterwards: `media/` only. +4. `docker compose -f deploy/compose.yaml run --rm --no-deps litestream restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db` + — volume afterwards: `mealquest.db` (258048 bytes) and `media/`; no + `-wal`, no `-shm`. +5. After (copy of the restored file, before the app started): + + ``` + dinners 0 + week_plans 0 + schema_migrations 7 + tables 26 content-sha256 836e230e16c58dc7 + integrity ok + ``` + +6. `docker compose -f deploy/compose.yaml up -d` — `mealquest-mealquest-1` + healthy, `mealquest-litestream-1` up; `GET /health` → `200`. Counts on a + copy of the running database (`mealquest.db`, `-wal`, `-shm`) were + identical to the above. +7. Litestream resumed against the restored database: + + ``` + level=INFO msg="detected database behind replica" db=mealquest.db db_txid=0000000000000000 replica_txid=0000000000000003 + level=INFO msg="fetched latest L0 file from replica" db=mealquest.db min_txid=0000000000000003 max_txid=0000000000000003 + level=INFO msg="compaction complete" db=mealquest.db level=1 txid.min=0000000000000004 txid.max=0000000000000004 size=14110 + ``` + + Replica after the drill (`ltx -level all`): + + ``` + level min_txid max_txid size created + 0 0000000000000004 0000000000000004 14110 2026-10-05T23:27:21Z + 1 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z + 1 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z + 1 0000000000000004 0000000000000004 14110 2026-10-05T23:27:21Z + 2 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z + 2 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z + 3 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z + 9 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z + ``` + +The first drill earlier the same day (before `-level all` was recorded) +gave the same before/after result: checksum `836e230e16c58dc7`, 26 tables, +integrity ok. + +### Media backup checks — 2026-10-05 + +`sudo systemctl start mealquest-media-backup.service` on 10.60.1.14: + +| Case | Result | +| --- | --- | +| Normal run | `Result=success`; journal: `synced /run/mealquest-data/media to joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ as uid 10001` | +| A test file added | `Number of regular files transferred: 1`; removing it then gave `Number of deleted files: 1 (reg: 1)` | +| `media` replaced by a symlink to `/etc` | `Result=exit-code`, status 1; `/run/mealquest-data/media is a symlink to /etc; refusing to back it up` | +| `media` removed | `Result=exit-code`, status 1; `media directory /run/mealquest-data/media is missing` | +| `sudo -u mealquest` writing into the volume via `/var/lib/docker/...` | `Permission denied` — uid 10001 cannot traverse `/var/lib/docker` (mode 0710 root) | diff --git a/deploy/compose.yaml b/deploy/compose.yaml index 6ec07bb..3c4251e 100644 --- a/deploy/compose.yaml +++ b/deploy/compose.yaml @@ -56,10 +56,8 @@ services: # the app's own user (owner of the volume) with a read-only root # filesystem, no privilege escalation and no Linux capabilities. # - # Restore (app stopped; --no-deps so the app does not start and create a - # fresh database first): - # docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ - # restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db + # Restore: follow "Restore the database" in deploy/RUNBOOK.md (stop the + # stack, move the database with its -wal and -shm aside, restore, check). litestream: image: litestream/litestream:0.5.17 user: "10001:10001" diff --git a/deploy/known_hosts b/deploy/known_hosts new file mode 100644 index 0000000..1b17352 --- /dev/null +++ b/deploy/known_hosts @@ -0,0 +1,6 @@ +# Host keys of the Synology NAS (10.60.1.50), the backup target. Installed +# root-owned with mode 0644 as /etc/mealquest/known_hosts and pinned by the +# media backup (UserKnownHostsFile, StrictHostKeyChecking=yes). The ECDSA +# key is also pinned in deploy/litestream.yml (host-key). +10.60.1.50 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKHxIfgjavBABMRE5GW/EFY4C4gO6mo0sKbhkUW2NcAo +10.60.1.50 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNujay9nYEiXuRQf4wDRDU1gp+CDvpeMdhuhC2XypIihzKGEtB654kwrUGfA2kDWAoSPGlmojeS3OIRtHPkj6GY= diff --git a/deploy/systemd/mealquest-media-backup.service b/deploy/systemd/mealquest-media-backup.service index 2f5d005..9f5a61c 100644 --- a/deploy/systemd/mealquest-media-backup.service +++ b/deploy/systemd/mealquest-media-backup.service @@ -3,6 +3,11 @@ # systemctl start mealquest-media-backup # and read the result with # journalctl -u mealquest-media-backup +# +# The script runs as the app's user (10001, host user "mealquest") with no +# capabilities. systemd, as root, only bind-mounts the data volume's root +# (a root-owned path the app cannot replace) read-only at +# /run/mealquest-data and copies the NAS key into the credentials directory. [Unit] Description=Back up MealQuest media to the Synology NAS After=network-online.target @@ -11,3 +16,14 @@ Wants=network-online.target [Service] Type=oneshot ExecStart=/usr/local/sbin/mealquest-media-backup +User=mealquest +Group=mealquest +LoadCredential=nas_key:/home/overseer/mealquest/deploy/secrets/litestream_key +BindReadOnlyPaths=/var/lib/docker/volumes/mealquest-data/_data:/run/mealquest-data +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +NoNewPrivileges=yes +CapabilityBoundingSet= +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX diff --git a/deploy/systemd/mealquest-media-backup.sh b/deploy/systemd/mealquest-media-backup.sh index be9376a..f4ee14d 100755 --- a/deploy/systemd/mealquest-media-backup.sh +++ b/deploy/systemd/mealquest-media-backup.sh @@ -3,38 +3,42 @@ # to the Synology NAS. The database itself is replicated continuously by # Litestream; this covers the files beside it. # -# rsync mirrors the media directory onto .../mealquest-backups/media/ only: -# both paths end in a slash, so --delete removes files from that target -# directory and nothing outside it. A missing source directory (no media -# uploaded yet) is reported and skipped, never synced, so an empty source -# can never wipe the remote copy. Re-runs transfer only what changed. +# Runs as the unprivileged app user (10001, host user "mealquest"), never +# root: mealquest-media-backup.service bind-mounts the data volume +# read-only at /run/mealquest-data and hands the NAS key in as a systemd +# credential. Anything the app could plant in the volume is therefore read +# with the app's own permissions only. # -# The volume is writable by the app container, so a media path that is a -# symlink is refused: root's rsync would otherwise follow it and copy -# whatever it points at. Symlinks inside the directory are copied as links -# (-a), never followed. +# The script changes into the media directory and checks that the physical +# path is the expected one, so a media path that is a symlink is refused. +# rsync then copies "./" — the directory the shell is already in — so +# swapping the path afterwards cannot redirect it. Symlinks inside the +# directory are copied as links (-a), never followed. # -# Connects with the dedicated MealQuest key; the NAS host key must already -# be in root's known_hosts (StrictHostKeyChecking=yes fails otherwise). -# Installed as /usr/local/sbin/mealquest-media-backup; output goes to the +# The media directory is created at deploy time (deploy/RUNBOOK.md). If it +# is missing the job fails; it is never treated as "nothing to back up". +# +# rsync mirrors into .../mealquest-backups/media/ only: the target ends in +# a slash, so --delete removes files from that directory and nothing +# outside it. Re-runs transfer only what changed. Output goes to the # journal via mealquest-media-backup.service. set -eu -src=/var/lib/docker/volumes/mealquest-data/_data/media/ +src=/run/mealquest-data/media dest=joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ -key=/home/overseer/mealquest/deploy/secrets/litestream_key +key=$CREDENTIALS_DIRECTORY/nas_key +known_hosts=/etc/mealquest/known_hosts -if [ -L "${src%/}" ]; then - echo "mealquest-media-backup: ${src%/} is a symlink; refusing to back it up" >&2 +if ! cd "$src"; then + echo "mealquest-media-backup: media directory $src is missing" >&2 + exit 1 +fi +if [ "$(pwd -P)" != "$src" ]; then + echo "mealquest-media-backup: $src is a symlink to $(pwd -P); refusing to back it up" >&2 exit 1 fi -if [ ! -d "$src" ]; then - echo "mealquest-media-backup: no media directory at $src yet; nothing to back up" - exit 0 -fi - rsync -a --delete --stats \ - -e "ssh -i $key -o IdentitiesOnly=yes -o BatchMode=yes -o StrictHostKeyChecking=yes" \ - "$src" "$dest" -echo "mealquest-media-backup: synced $src to $dest" + -e "ssh -F none -i $key -o IdentitiesOnly=yes -o BatchMode=yes -o UserKnownHostsFile=$known_hosts -o GlobalKnownHostsFile=/dev/null -o StrictHostKeyChecking=yes" \ + ./ "$dest" +echo "mealquest-media-backup: synced $src to $dest as uid $(id -u)"