MealQuest/deploy/RUNBOOK.md

MealQuest deployment runbook

How MealQuest is deployed on office-srvr (10.60.1.14) and how its data is backed up and restored. Sections marked task 876134bc are completed by that task (Feature 07 deployment runbook); this file currently records the reverse-proxy route, the backups and the restore procedure.

Paths used below:

Name	Path
Checkout on the host	/home/overseer/mealquest
Data volume (mealquest-data) on the host	/var/lib/docker/volumes/mealquest-data/_data
Same volume inside the containers	/data
Database	mealquest.db (+ mealquest.db-wal, mealquest.db-shm while open)
Media	media/ in the data volume
Backup target (NAS, over SSH)	joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups
Same target over SFTP (Synology chroot)	/homes/Joshua/mealquest-backups

Stack and first deploy

Completed by task 876134bc. The compose stack is deploy/compose.yaml; start it with docker compose -f deploy/compose.yaml up -d --build.

Reverse proxy (Traefik)

MealQuest is published at https://mealquest.cubecraftlabs.com by the shared Traefik instance on 10.60.1.33 (Feature 07, task 0c1dc172). The route is a dynamic-configuration file that lives on the Traefik host only, at /home/overseer/Code/traefik/config/dynamic/mealquest-ccl.yml. Its exact contents:

# MealQuest - https://mealquest.cubecraftlabs.com
#   Feature 07 task 0c1dc172. Single Go binary (API + embedded PWA) on
#   office-srvr 10.60.1.14:8090. HTTP -> HTTPS is handled globally by
#   http-conf.yml, so only a websecure router is defined here.
#   This file lives on the Traefik host only; the repo copy is recorded in
#   deploy/RUNBOOK.md by task 876134bc.
http:
  routers:
    mealquest:
      entryPoints:
        - websecure
      rule: "Host(`mealquest.cubecraftlabs.com`)"
      priority: 100
      middlewares:
        - mealquest-headers
      service: mealquest
      tls:
        certResolver: myresolver
        domains:
          - main: mealquest.cubecraftlabs.com

  services:
    mealquest:
      loadBalancer:
        servers:
          - url: "http://10.60.1.14:8090"

  middlewares:
    mealquest-headers:
      headers:
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        contentTypeNosniff: true
        frameDeny: true

Backups

Database — Litestream

The litestream compose service streams every committed change to mealquest.db to the NAS over SFTP (deploy/litestream.yml). It takes a full snapshot daily (compaction level 9) and keeps a week of snapshots plus the incremental LTX files between them.

One-time setup (already done on 10.60.1.14):

1. Generate the dedicated keypair, git-ignored, owned by the container user:

   cd /home/overseer/mealquest
   ssh-keygen -t ed25519 -N '' -C mealquest-litestream -f deploy/secrets/litestream_key
   # append deploy/secrets/litestream_key.pub to joshua@10.60.1.50:~/.ssh/authorized_keys
   rm deploy/secrets/litestream_key.pub
   sudo chown 10001:10001 deploy/secrets/litestream_key
   sudo chmod 0600 deploy/secrets/litestream_key
   

2. The NAS host key is pinned in deploy/litestream.yml (host-key).

Check replication — list every LTX file at every level. Level 9 holds the full snapshots; the default listing (-level 0) shows only L0 files:

docker compose -f deploy/compose.yaml exec -T litestream \
  litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db

Media — nightly rsync

mealquest-media-backup.timer runs mealquest-media-backup.service at 03:30 every night. The service runs /usr/local/sbin/mealquest-media-backup (deploy/systemd/mealquest-media-backup.sh) as the unprivileged host user mealquest (uid/gid 10001, the app's own user) with no capabilities. systemd bind-mounts the data volume read-only at /run/mealquest-data and passes the Litestream key in as the credential nas_key; uid 10001 cannot otherwise reach /var/lib/docker or deploy/secrets/. The script refuses a media path that is a symlink and fails (exit 1) if media is missing.

One-time setup (already done on 10.60.1.14):

cd /home/overseer/mealquest
# Host user for the backup; ssh needs a passwd entry for its uid.
sudo groupadd --system --gid 10001 mealquest
sudo useradd --system --uid 10001 --gid 10001 --no-create-home \
  --home-dir /nonexistent --shell /usr/sbin/nologin mealquest
# Pinned NAS host keys, root-owned and world-readable.
sudo install -d -o root -g root -m 0755 /etc/mealquest
sudo install -o root -g root -m 0644 deploy/known_hosts /etc/mealquest/known_hosts
# The media directory; the backup fails if it is ever missing.
sudo install -d -o 10001 -g 10001 -m 0755 /var/lib/docker/volumes/mealquest-data/_data/media
# Script and units.
sudo install -o root -g root -m 0755 deploy/systemd/mealquest-media-backup.sh /usr/local/sbin/mealquest-media-backup
sudo install -o root -g root -m 0644 deploy/systemd/mealquest-media-backup.service \
  deploy/systemd/mealquest-media-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now mealquest-media-backup.timer

Run it by hand and read the result:

sudo systemctl start mealquest-media-backup.service
journalctl -u mealquest-media-backup --no-pager -n 30

Restore media by reversing the copy into the (empty or stale) media directory as uid 10001:

sudo rsync -a --chown=10001:10001 \
  -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \
  joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \
  /var/lib/docker/volumes/mealquest-data/_data/media/

Restore the database

SQLite in WAL mode keeps recent commits in mealquest.db-wal, indexed by mealquest.db-shm. A restored database must never sit beside an old WAL or shared-memory file: on open, SQLite would replay the old WAL's pages onto the restored database and corrupt it or mix in state from the database it replaced. So all three files are moved aside together, with Litestream's local state directory .mealquest.db-litestream, before the restore — and the restore runs into an empty volume.

Run from /home/overseer/mealquest:

D=/var/lib/docker/volumes/mealquest-data/_data
A=/var/tmp/mealquest-restore-aside-$(date +%Y%m%d%H%M)

# 1. Stop the whole stack (app and Litestream). Nothing may hold the
#    database open; a clean close checkpoints the WAL into mealquest.db.
docker compose -f deploy/compose.yaml down

# 2. Move the database, its -wal and -shm files and Litestream's state
#    aside — all of them, every time. Keep the directory until the
#    restored stack is verified.
sudo mkdir "$A"
for f in mealquest.db mealquest.db-wal mealquest.db-shm .mealquest.db-litestream; do
  if sudo test -e "$D/$f"; then sudo mv "$D/$f" "$A/"; fi
done
sudo ls -la "$D"   # must list only media/

# 3. Restore from the NAS. --no-deps keeps the app stopped so it cannot
#    create a fresh, empty database first.
docker compose -f deploy/compose.yaml run --rm --no-deps litestream \
  restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db
sudo ls -la "$D"   # mealquest.db only: no -wal, no -shm

# 4. Check the restored file before starting the app (see "Count check").

# 5. Start the stack and check health.
docker compose -f deploy/compose.yaml up -d
docker compose -f deploy/compose.yaml ps
curl -s -o /dev/null -w "health %{http_code}\n" http://10.60.1.14:8090/health

To restore to a point in time, add -timestamp 2026-10-05T23:00:00Z (or -txid) to the restore command.

Count check

/tmp/mqcount.py prints row counts for key tables, the number of application tables, a SHA-256 over every application row (first 16 hex digits; Litestream's own _litestream* tables are excluded) and PRAGMA integrity_check. Run it on a copy of the database, never on the live file; copy -wal/-shm along with it when the app is running.

import sqlite3, sys, hashlib
c = sqlite3.connect(sys.argv[1])
tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
for t in ("dinners", "week_plans", "schema_migrations"):
    print(t, c.execute(f"select count(*) from {t}").fetchone()[0])
h = hashlib.sha256()
for t in tabs:
    for row in c.execute(f"select * from \"{t}\" order by rowid"):
        h.update(repr(row).encode())
print("tables", len(tabs), "content-sha256", h.hexdigest()[:16])
print("integrity", c.execute("pragma integrity_check").fetchone()[0])

rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck
sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck
python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db

Before and after must match exactly.

Restore drill record — 2026-10-05

Drill on 10.60.1.14 against the live (pre-launch, empty-household) database, Litestream 0.5.17. Commands as in "Restore the database"; output trimmed only of compose progress lines.

Replica before the drill (ltx -level all; level 9 is the full snapshot):

level  min_txid          max_txid          size   created
0      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
1      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
1      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
2      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
2      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
3      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
9      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z

1. docker compose -f deploy/compose.yaml down — both containers stopped and removed. The volume then held mealquest.db, .mealquest.db-litestream/ and media/; the clean close had checkpointed and removed -wal and -shm.

2. Before (copy of the stopped database):

   dinners 0
   week_plans 0
   schema_migrations 7
   tables 26 content-sha256 836e230e16c58dc7
   integrity ok
   

3. Moved aside to /var/tmp/mealquest-restore-aside-20261005: mealquest.db and .mealquest.db-litestream (no -wal/-shm existed after the clean stop; the loop moves them whenever they do). Volume afterwards: media/ only.

4. docker compose -f deploy/compose.yaml run --rm --no-deps litestream restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db — volume afterwards: mealquest.db (258048 bytes) and media/; no -wal, no -shm.

5. After (copy of the restored file, before the app started):

   dinners 0
   week_plans 0
   schema_migrations 7
   tables 26 content-sha256 836e230e16c58dc7
   integrity ok
   

6. docker compose -f deploy/compose.yaml up -d — mealquest-mealquest-1 healthy, mealquest-litestream-1 up; GET /health → 200. Counts on a copy of the running database (mealquest.db, -wal, -shm) were identical to the above.

7. Litestream resumed against the restored database:

   level=INFO msg="detected database behind replica" db=mealquest.db db_txid=0000000000000000 replica_txid=0000000000000003
   level=INFO msg="fetched latest L0 file from replica" db=mealquest.db min_txid=0000000000000003 max_txid=0000000000000003
   level=INFO msg="compaction complete" db=mealquest.db level=1 txid.min=0000000000000004 txid.max=0000000000000004 size=14110
   

   Replica after the drill (ltx -level all):

   level  min_txid          max_txid          size   created
   0      0000000000000004  0000000000000004  14110  2026-10-05T23:27:21Z
   1      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
   1      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
   1      0000000000000004  0000000000000004  14110  2026-10-05T23:27:21Z
   2      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
   2      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
   3      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
   9      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
   

The first drill earlier the same day (before -level all was recorded) gave the same before/after result: checksum 836e230e16c58dc7, 26 tables, integrity ok.

Media backup checks — 2026-10-05

sudo systemctl start mealquest-media-backup.service on 10.60.1.14:

Case	Result
Normal run	Result=success; journal: synced /run/mealquest-data/media to joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ as uid 10001
A test file added	Number of regular files transferred: 1; removing it then gave Number of deleted files: 1 (reg: 1)
media replaced by a symlink to /etc	Result=exit-code, status 1; /run/mealquest-data/media is a symlink to /etc; refusing to back it up
media removed	Result=exit-code, status 1; media directory /run/mealquest-data/media is missing
sudo -u mealquest writing into the volume via /var/lib/docker/...	Permission denied — uid 10001 cannot traverse /var/lib/docker (mode 0710 root)