Litestream replication and proven restore (#73)

Co-authored-by: dev_agent <dev_agent@cubecraftlabs.com>
2026-10-05 19:41:11 -04:00
parent 9d0e943fc3
commit ceb68cc26f
7 changed files with 473 additions and 0 deletions
@@ -0,0 +1,318 @@
+# MealQuest deployment runbook
+
+How MealQuest is deployed on office-srvr (10.60.1.14) and how its data is
+backed up and restored. Sections marked *task 876134bc* are completed by
+that task (Feature 07 deployment runbook); this file currently records the
+reverse-proxy route, the backups and the restore procedure.
+
+Paths used below:
+
+| Name | Path |
+| --- | --- |
+| Checkout on the host | `/home/overseer/mealquest` |
+| Data volume (`mealquest-data`) on the host | `/var/lib/docker/volumes/mealquest-data/_data` |
+| Same volume inside the containers | `/data` |
+| Database | `mealquest.db` (+ `mealquest.db-wal`, `mealquest.db-shm` while open) |
+| Media | `media/` in the data volume |
+| Backup target (NAS, over SSH) | `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups` |
+| Same target over SFTP (Synology chroot) | `/homes/Joshua/mealquest-backups` |
+
+## Stack and first deploy
+
+*Completed by task 876134bc.* The compose stack is `deploy/compose.yaml`;
+start it with `docker compose -f deploy/compose.yaml up -d --build`.
+
+## Reverse proxy (Traefik)
+
+MealQuest is published at <https://mealquest.cubecraftlabs.com> by the
+shared Traefik instance on 10.60.1.33 (Feature 07, task 0c1dc172). The
+route is a dynamic-configuration file that lives on the Traefik host only,
+at `/home/overseer/Code/traefik/config/dynamic/mealquest-ccl.yml`. Its exact
+contents:
+
+```yaml
+# MealQuest - https://mealquest.cubecraftlabs.com
+#   Feature 07 task 0c1dc172. Single Go binary (API + embedded PWA) on
+#   office-srvr 10.60.1.14:8090. HTTP -> HTTPS is handled globally by
+#   http-conf.yml, so only a websecure router is defined here.
+#   This file lives on the Traefik host only; the repo copy is recorded in
+#   deploy/RUNBOOK.md by task 876134bc.
+http:
+  routers:
+    mealquest:
+      entryPoints:
+        - websecure
+      rule: "Host(`mealquest.cubecraftlabs.com`)"
+      priority: 100
+      middlewares:
+        - mealquest-headers
+      service: mealquest
+      tls:
+        certResolver: myresolver
+        domains:
+          - main: mealquest.cubecraftlabs.com
+
+  services:
+    mealquest:
+      loadBalancer:
+        servers:
+          - url: "http://10.60.1.14:8090"
+
+  middlewares:
+    mealquest-headers:
+      headers:
+        stsSeconds: 31536000
+        stsIncludeSubdomains: true
+        contentTypeNosniff: true
+        frameDeny: true
+```
+
+## Backups
+
+### Database — Litestream
+
+The `litestream` compose service streams every committed change to
+`mealquest.db` to the NAS over SFTP (`deploy/litestream.yml`). It takes a
+full snapshot daily (compaction level 9) and keeps a week of snapshots
+plus the incremental LTX files between them.
+
+One-time setup (already done on 10.60.1.14):
+
+1. Generate the dedicated keypair, git-ignored, owned by the container
+   user:
+
+   ```sh
+   cd /home/overseer/mealquest
+   ssh-keygen -t ed25519 -N '' -C mealquest-litestream -f deploy/secrets/litestream_key
+   # append deploy/secrets/litestream_key.pub to joshua@10.60.1.50:~/.ssh/authorized_keys
+   rm deploy/secrets/litestream_key.pub
+   sudo chown 10001:10001 deploy/secrets/litestream_key
+   sudo chmod 0600 deploy/secrets/litestream_key
+   ```
+
+2. The NAS host key is pinned in `deploy/litestream.yml` (`host-key`).
+
+Check replication — list every LTX file at every level. Level 9 holds the
+full snapshots; the default listing (`-level 0`) shows only L0 files:
+
+```sh
+docker compose -f deploy/compose.yaml exec -T litestream \
+  litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
+```
+
+### Media — nightly rsync
+
+`mealquest-media-backup.timer` runs `mealquest-media-backup.service` at
+03:30 every night. The service runs `/usr/local/sbin/mealquest-media-backup`
+(`deploy/systemd/mealquest-media-backup.sh`) as the unprivileged host user
+`mealquest` (uid/gid 10001, the app's own user) with no capabilities.
+systemd bind-mounts the data volume read-only at `/run/mealquest-data` and
+passes the Litestream key in as the credential `nas_key`; uid 10001 cannot
+otherwise reach `/var/lib/docker` or `deploy/secrets/`. The script refuses a
+`media` path that is a symlink and fails (exit 1) if `media` is missing.
+
+One-time setup (already done on 10.60.1.14):
+
+```sh
+cd /home/overseer/mealquest
+# Host user for the backup; ssh needs a passwd entry for its uid.
+sudo groupadd --system --gid 10001 mealquest
+sudo useradd --system --uid 10001 --gid 10001 --no-create-home \
+  --home-dir /nonexistent --shell /usr/sbin/nologin mealquest
+# Pinned NAS host keys, root-owned and world-readable.
+sudo install -d -o root -g root -m 0755 /etc/mealquest
+sudo install -o root -g root -m 0644 deploy/known_hosts /etc/mealquest/known_hosts
+# The media directory; the backup fails if it is ever missing.
+sudo install -d -o 10001 -g 10001 -m 0755 /var/lib/docker/volumes/mealquest-data/_data/media
+# Script and units.
+sudo install -o root -g root -m 0755 deploy/systemd/mealquest-media-backup.sh /usr/local/sbin/mealquest-media-backup
+sudo install -o root -g root -m 0644 deploy/systemd/mealquest-media-backup.service \
+  deploy/systemd/mealquest-media-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now mealquest-media-backup.timer
+```
+
+Run it by hand and read the result:
+
+```sh
+sudo systemctl start mealquest-media-backup.service
+journalctl -u mealquest-media-backup --no-pager -n 30
+```
+
+Restore media by reversing the copy into the (empty or stale) media
+directory as uid 10001:
+
+```sh
+sudo rsync -a --chown=10001:10001 \
+  -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \
+  joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \
+  /var/lib/docker/volumes/mealquest-data/_data/media/
+```
+
+## Restore the database
+
+SQLite in WAL mode keeps recent commits in `mealquest.db-wal`, indexed by
+`mealquest.db-shm`. **A restored database must never sit beside an old WAL
+or shared-memory file**: on open, SQLite would replay the old WAL's pages
+onto the restored database and corrupt it or mix in state from the
+database it replaced. So all three files are moved aside together, with
+Litestream's local state directory `.mealquest.db-litestream`, before the
+restore — and the restore runs into an empty volume.
+
+Run from `/home/overseer/mealquest`:
+
+```sh
+D=/var/lib/docker/volumes/mealquest-data/_data
+A=/var/tmp/mealquest-restore-aside-$(date +%Y%m%d%H%M)
+
+# 1. Stop the whole stack (app and Litestream). Nothing may hold the
+#    database open; a clean close checkpoints the WAL into mealquest.db.
+docker compose -f deploy/compose.yaml down
+
+# 2. Move the database, its -wal and -shm files and Litestream's state
+#    aside — all of them, every time. Keep the directory until the
+#    restored stack is verified.
+sudo mkdir "$A"
+for f in mealquest.db mealquest.db-wal mealquest.db-shm .mealquest.db-litestream; do
+  if sudo test -e "$D/$f"; then sudo mv "$D/$f" "$A/"; fi
+done
+sudo ls -la "$D"   # must list only media/
+
+# 3. Restore from the NAS. --no-deps keeps the app stopped so it cannot
+#    create a fresh, empty database first.
+docker compose -f deploy/compose.yaml run --rm --no-deps litestream \
+  restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db
+sudo ls -la "$D"   # mealquest.db only: no -wal, no -shm
+
+# 4. Check the restored file before starting the app (see "Count check").
+
+# 5. Start the stack and check health.
+docker compose -f deploy/compose.yaml up -d
+docker compose -f deploy/compose.yaml ps
+curl -s -o /dev/null -w "health %{http_code}\n" http://10.60.1.14:8090/health
+```
+
+To restore to a point in time, add `-timestamp 2026-10-05T23:00:00Z` (or
+`-txid`) to the `restore` command.
+
+### Count check
+
+`/tmp/mqcount.py` prints row counts for key tables, the number of
+application tables, a SHA-256 over every application row (first 16 hex
+digits; Litestream's own `_litestream*` tables are excluded) and
+`PRAGMA integrity_check`. Run it on a **copy** of the database, never on the
+live file; copy `-wal`/`-shm` along with it when the app is running.
+
+```python
+import sqlite3, sys, hashlib
+c = sqlite3.connect(sys.argv[1])
+tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
+for t in ("dinners", "week_plans", "schema_migrations"):
+    print(t, c.execute(f"select count(*) from {t}").fetchone()[0])
+h = hashlib.sha256()
+for t in tabs:
+    for row in c.execute(f"select * from \"{t}\" order by rowid"):
+        h.update(repr(row).encode())
+print("tables", len(tabs), "content-sha256", h.hexdigest()[:16])
+print("integrity", c.execute("pragma integrity_check").fetchone()[0])
+```
+
+```sh
+rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck
+sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck
+python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db
+```
+
+Before and after must match exactly.
+
+## Restore drill record — 2026-10-05
+
+Drill on 10.60.1.14 against the live (pre-launch, empty-household)
+database, Litestream 0.5.17. Commands as in "Restore the database";
+output trimmed only of compose progress lines.
+
+Replica before the drill (`ltx -level all`; level 9 is the full snapshot):
+
+```
+level  min_txid          max_txid          size   created
+0      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
+1      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+1      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
+2      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+2      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
+3      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+9      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+```
+
+1. `docker compose -f deploy/compose.yaml down` — both containers stopped
+   and removed. The volume then held `mealquest.db`,
+   `.mealquest.db-litestream/` and `media/`; the clean close had
+   checkpointed and removed `-wal` and `-shm`.
+2. Before (copy of the stopped database):
+
+   ```
+   dinners 0
+   week_plans 0
+   schema_migrations 7
+   tables 26 content-sha256 836e230e16c58dc7
+   integrity ok
+   ```
+
+3. Moved aside to `/var/tmp/mealquest-restore-aside-20261005`:
+   `mealquest.db` and `.mealquest.db-litestream` (no `-wal`/`-shm` existed
+   after the clean stop; the loop moves them whenever they do). Volume
+   afterwards: `media/` only.
+4. `docker compose -f deploy/compose.yaml run --rm --no-deps litestream restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db`
+   — volume afterwards: `mealquest.db` (258048 bytes) and `media/`; no
+   `-wal`, no `-shm`.
+5. After (copy of the restored file, before the app started):
+
+   ```
+   dinners 0
+   week_plans 0
+   schema_migrations 7
+   tables 26 content-sha256 836e230e16c58dc7
+   integrity ok
+   ```
+
+6. `docker compose -f deploy/compose.yaml up -d` — `mealquest-mealquest-1`
+   healthy, `mealquest-litestream-1` up; `GET /health` → `200`. Counts on a
+   copy of the running database (`mealquest.db`, `-wal`, `-shm`) were
+   identical to the above.
+7. Litestream resumed against the restored database:
+
+   ```
+   level=INFO msg="detected database behind replica" db=mealquest.db db_txid=0000000000000000 replica_txid=0000000000000003
+   level=INFO msg="fetched latest L0 file from replica" db=mealquest.db min_txid=0000000000000003 max_txid=0000000000000003
+   level=INFO msg="compaction complete" db=mealquest.db level=1 txid.min=0000000000000004 txid.max=0000000000000004 size=14110
+   ```
+
+   Replica after the drill (`ltx -level all`):
+
+   ```
+   level  min_txid          max_txid          size   created
+   0      0000000000000004  0000000000000004  14110  2026-10-05T23:27:21Z
+   1      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+   1      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
+   1      0000000000000004  0000000000000004  14110  2026-10-05T23:27:21Z
+   2      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+   2      0000000000000003  0000000000000003  14110  2026-10-05T23:13:30Z
+   3      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+   9      0000000000000001  0000000000000002  14110  2026-10-05T23:12:38Z
+   ```
+
+The first drill earlier the same day (before `-level all` was recorded)
+gave the same before/after result: checksum `836e230e16c58dc7`, 26 tables,
+integrity ok.
+
+### Media backup checks — 2026-10-05
+
+`sudo systemctl start mealquest-media-backup.service` on 10.60.1.14:
+
+| Case | Result |
+| --- | --- |
+| Normal run | `Result=success`; journal: `synced /run/mealquest-data/media to joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ as uid 10001` |
+| A test file added | `Number of regular files transferred: 1`; removing it then gave `Number of deleted files: 1 (reg: 1)` |
+| `media` replaced by a symlink to `/etc` | `Result=exit-code`, status 1; `/run/mealquest-data/media is a symlink to /etc; refusing to back it up` |
+| `media` removed | `Result=exit-code`, status 1; `media directory /run/mealquest-data/media is missing` |
+| `sudo -u mealquest` writing into the volume via `/var/lib/docker/...` | `Permission denied` — uid 10001 cannot traverse `/var/lib/docker` (mode 0710 root) |
@@ -46,6 +46,40 @@ services:
    cap_drop:
      - ALL

+  # Litestream streams the database to the Synology NAS over SFTP
+  # (config: deploy/litestream.yml). The SFTP private key
+  # deploy/secrets/litestream_key is a dedicated keypair, owned by
+  # 10001:10001 with mode 0600 like the other secrets.
+  #
+  # The data volume is mounted read-write on purpose: Litestream writes its
+  # tracking tables into the database and checkpoints the WAL. It runs as
+  # the app's own user (owner of the volume) with a read-only root
+  # filesystem, no privilege escalation and no Linux capabilities.
+  #
+  # Restore: follow "Restore the database" in deploy/RUNBOOK.md (stop the
+  # stack, move the database with its -wal and -shm aside, restore, check).
+  litestream:
+    image: litestream/litestream:0.5.17
+    user: "10001:10001"
+    restart: unless-stopped
+    command: ["replicate", "-config", "/etc/litestream.yml"]
+    depends_on:
+      mealquest:
+        condition: service_healthy
+    volumes:
+      - mealquest-data:/data
+      - ./litestream.yml:/etc/litestream.yml:ro
+      - ./secrets/litestream_key:/etc/litestream/sftp_key:ro
+    networks:
+      - mealquest
+    read_only: true
+    tmpfs:
+      - /tmp
+    security_opt:
+      - no-new-privileges:true
+    cap_drop:
+      - ALL
+
 secrets:
  secret_key:
    file: ./secrets/secret_key
@@ -0,0 +1,6 @@
+# Host keys of the Synology NAS (10.60.1.50), the backup target. Installed
+# root-owned with mode 0644 as /etc/mealquest/known_hosts and pinned by the
+# media backup (UserKnownHostsFile, StrictHostKeyChecking=yes). The ECDSA
+# key is also pinned in deploy/litestream.yml (host-key).
+10.60.1.50 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKHxIfgjavBABMRE5GW/EFY4C4gO6mo0sKbhkUW2NcAo
+10.60.1.50 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNujay9nYEiXuRQf4wDRDU1gp+CDvpeMdhuhC2XypIihzKGEtB654kwrUGfA2kDWAoSPGlmojeS3OIRtHPkj6GY=
@@ -0,0 +1,31 @@
+# Litestream continuous backup of the MealQuest SQLite database (Feature
+# 07). The ``litestream`` compose service mounts the same named volume as
+# the app and streams every committed change to the Synology NAS over
+# SFTP, so the backup lives outside the Docker host.
+#
+# Nothing here is secret. The SFTP private key is a dedicated, git-ignored
+# keypair (deploy/secrets/litestream_key) mounted read-only by compose at
+# /etc/litestream/sftp_key; its public half is in
+# joshua@10.60.1.50:~/.ssh/authorized_keys. ``host-key`` pins the NAS's
+# ECDSA host key, the algorithm Litestream's SSH client negotiates with
+# this NAS.
+#
+# The Synology SFTP server is chrooted to the shared folders, so the
+# directory seen over SSH as /volume1/homes/Joshua/mealquest-backups is
+# /homes/Joshua/mealquest-backups over SFTP.
+
+# Take a full snapshot daily and keep a week of snapshots plus the
+# incremental changes between them (point-in-time restore window).
+snapshot:
+  interval: 24h
+  retention: 168h
+
+dbs:
+  - path: /data/mealquest.db
+    replicas:
+      - type: sftp
+        host: 10.60.1.50
+        user: joshua
+        path: /homes/Joshua/mealquest-backups
+        key-path: /etc/litestream/sftp_key
+        host-key: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNujay9nYEiXuRQf4wDRDU1gp+CDvpeMdhuhC2XypIihzKGEtB654kwrUGfA2kDWAoSPGlmojeS3OIRtHPkj6GY=
@@ -0,0 +1,29 @@
+# Runs the nightly MealQuest media backup (/usr/local/sbin/mealquest-media-backup).
+# Triggered by mealquest-media-backup.timer; run by hand with
+#   systemctl start mealquest-media-backup
+# and read the result with
+#   journalctl -u mealquest-media-backup
+#
+# The script runs as the app's user (10001, host user "mealquest") with no
+# capabilities. systemd, as root, only bind-mounts the data volume's root
+# (a root-owned path the app cannot replace) read-only at
+# /run/mealquest-data and copies the NAS key into the credentials directory.
+[Unit]
+Description=Back up MealQuest media to the Synology NAS
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/mealquest-media-backup
+User=mealquest
+Group=mealquest
+LoadCredential=nas_key:/home/overseer/mealquest/deploy/secrets/litestream_key
+BindReadOnlyPaths=/var/lib/docker/volumes/mealquest-data/_data:/run/mealquest-data
+ProtectSystem=strict
+ProtectHome=yes
+PrivateTmp=yes
+PrivateDevices=yes
+NoNewPrivileges=yes
+CapabilityBoundingSet=
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
@@ -0,0 +1,44 @@
+#!/bin/sh
+# Nightly copy of the MealQuest media directory (recipe and dinner photos)
+# to the Synology NAS. The database itself is replicated continuously by
+# Litestream; this covers the files beside it.
+#
+# Runs as the unprivileged app user (10001, host user "mealquest"), never
+# root: mealquest-media-backup.service bind-mounts the data volume
+# read-only at /run/mealquest-data and hands the NAS key in as a systemd
+# credential. Anything the app could plant in the volume is therefore read
+# with the app's own permissions only.
+#
+# The script changes into the media directory and checks that the physical
+# path is the expected one, so a media path that is a symlink is refused.
+# rsync then copies "./" — the directory the shell is already in — so
+# swapping the path afterwards cannot redirect it. Symlinks inside the
+# directory are copied as links (-a), never followed.
+#
+# The media directory is created at deploy time (deploy/RUNBOOK.md). If it
+# is missing the job fails; it is never treated as "nothing to back up".
+#
+# rsync mirrors into .../mealquest-backups/media/ only: the target ends in
+# a slash, so --delete removes files from that directory and nothing
+# outside it. Re-runs transfer only what changed. Output goes to the
+# journal via mealquest-media-backup.service.
+set -eu
+
+src=/run/mealquest-data/media
+dest=joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/
+key=$CREDENTIALS_DIRECTORY/nas_key
+known_hosts=/etc/mealquest/known_hosts
+
+if ! cd "$src"; then
+	echo "mealquest-media-backup: media directory $src is missing" >&2
+	exit 1
+fi
+if [ "$(pwd -P)" != "$src" ]; then
+	echo "mealquest-media-backup: $src is a symlink to $(pwd -P); refusing to back it up" >&2
+	exit 1
+fi
+
+rsync -a --delete --stats \
+	-e "ssh -F none -i $key -o IdentitiesOnly=yes -o BatchMode=yes -o UserKnownHostsFile=$known_hosts -o GlobalKnownHostsFile=/dev/null -o StrictHostKeyChecking=yes" \
+	./ "$dest"
+echo "mealquest-media-backup: synced $src to $dest as uid $(id -u)"
@@ -0,0 +1,11 @@
+# Nightly trigger for mealquest-media-backup.service. Persistent= runs a
+# missed backup at the next boot.
+[Unit]
+Description=Nightly MealQuest media backup
+
+[Timer]
+OnCalendar=*-*-* 03:30:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target