319 lines
13 KiB
Markdown
319 lines
13 KiB
Markdown
|
# MealQuest deployment runbook
|
||
|
|
|
||
|
|
How MealQuest is deployed on office-srvr (10.60.1.14) and how its data is
|
||
|
|
backed up and restored. Sections marked *task 876134bc* are completed by
|
||
|
|
that task (Feature 07 deployment runbook); this file currently records the
|
||
|
|
reverse-proxy route, the backups and the restore procedure.
|
||
|
|
|
||
|
|
Paths used below:
|
||
|
|
|
||
|
|
| Name | Path |
|
||
|
|
| --- | --- |
|
||
|
|
| Checkout on the host | `/home/overseer/mealquest` |
|
||
|
|
| Data volume (`mealquest-data`) on the host | `/var/lib/docker/volumes/mealquest-data/_data` |
|
||
|
|
| Same volume inside the containers | `/data` |
|
||
|
|
| Database | `mealquest.db` (+ `mealquest.db-wal`, `mealquest.db-shm` while open) |
|
||
|
|
| Media | `media/` in the data volume |
|
||
|
|
| Backup target (NAS, over SSH) | `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups` |
|
||
|
|
| Same target over SFTP (Synology chroot) | `/homes/Joshua/mealquest-backups` |
|
||
|
|
|
||
|
|
## Stack and first deploy
|
||
|
|
|
||
|
|
*Completed by task 876134bc.* The compose stack is `deploy/compose.yaml`;
|
||
|
|
start it with `docker compose -f deploy/compose.yaml up -d --build`.
|
||
|
|
|
||
|
|
## Reverse proxy (Traefik)
|
||
|
|
|
||
|
|
MealQuest is published at <https://mealquest.cubecraftlabs.com> by the
|
||
|
|
shared Traefik instance on 10.60.1.33 (Feature 07, task 0c1dc172). The
|
||
|
|
route is a dynamic-configuration file that lives on the Traefik host only,
|
||
|
|
at `/home/overseer/Code/traefik/config/dynamic/mealquest-ccl.yml`. Its exact
|
||
|
|
contents:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
# MealQuest - https://mealquest.cubecraftlabs.com
|
||
|
|
# Feature 07 task 0c1dc172. Single Go binary (API + embedded PWA) on
|
||
|
|
# office-srvr 10.60.1.14:8090. HTTP -> HTTPS is handled globally by
|
||
|
|
# http-conf.yml, so only a websecure router is defined here.
|
||
|
|
# This file lives on the Traefik host only; the repo copy is recorded in
|
||
|
|
# deploy/RUNBOOK.md by task 876134bc.
|
||
|
|
http:
|
||
|
|
routers:
|
||
|
|
mealquest:
|
||
|
|
entryPoints:
|
||
|
|
- websecure
|
||
|
|
rule: "Host(`mealquest.cubecraftlabs.com`)"
|
||
|
|
priority: 100
|
||
|
|
middlewares:
|
||
|
|
- mealquest-headers
|
||
|
|
service: mealquest
|
||
|
|
tls:
|
||
|
|
certResolver: myresolver
|
||
|
|
domains:
|
||
|
|
- main: mealquest.cubecraftlabs.com
|
||
|
|
|
||
|
|
services:
|
||
|
|
mealquest:
|
||
|
|
loadBalancer:
|
||
|
|
servers:
|
||
|
|
- url: "http://10.60.1.14:8090"
|
||
|
|
|
||
|
|
middlewares:
|
||
|
|
mealquest-headers:
|
||
|
|
headers:
|
||
|
|
stsSeconds: 31536000
|
||
|
|
stsIncludeSubdomains: true
|
||
|
|
contentTypeNosniff: true
|
||
|
|
frameDeny: true
|
||
|
|
```
|
||
|
|
|
||
|
|
## Backups
|
||
|
|
|
||
|
|
### Database — Litestream
|
||
|
|
|
||
|
|
The `litestream` compose service streams every committed change to
|
||
|
|
`mealquest.db` to the NAS over SFTP (`deploy/litestream.yml`). It takes a
|
||
|
|
full snapshot daily (compaction level 9) and keeps a week of snapshots
|
||
|
|
plus the incremental LTX files between them.
|
||
|
|
|
||
|
|
One-time setup (already done on 10.60.1.14):
|
||
|
|
|
||
|
|
1. Generate the dedicated keypair, git-ignored, owned by the container
|
||
|
|
user:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
cd /home/overseer/mealquest
|
||
|
|
ssh-keygen -t ed25519 -N '' -C mealquest-litestream -f deploy/secrets/litestream_key
|
||
|
|
# append deploy/secrets/litestream_key.pub to joshua@10.60.1.50:~/.ssh/authorized_keys
|
||
|
|
rm deploy/secrets/litestream_key.pub
|
||
|
|
sudo chown 10001:10001 deploy/secrets/litestream_key
|
||
|
|
sudo chmod 0600 deploy/secrets/litestream_key
|
||
|
|
```
|
||
|
|
|
||
|
|
2. The NAS host key is pinned in `deploy/litestream.yml` (`host-key`).
|
||
|
|
|
||
|
|
Check replication — list every LTX file at every level. Level 9 holds the
|
||
|
|
full snapshots; the default listing (`-level 0`) shows only L0 files:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
docker compose -f deploy/compose.yaml exec -T litestream \
|
||
|
|
litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
|
||
|
|
```
|
||
|
|
|
||
|
|
### Media — nightly rsync
|
||
|
|
|
||
|
|
`mealquest-media-backup.timer` runs `mealquest-media-backup.service` at
|
||
|
|
03:30 every night. The service runs `/usr/local/sbin/mealquest-media-backup`
|
||
|
|
(`deploy/systemd/mealquest-media-backup.sh`) as the unprivileged host user
|
||
|
|
`mealquest` (uid/gid 10001, the app's own user) with no capabilities.
|
||
|
|
systemd bind-mounts the data volume read-only at `/run/mealquest-data` and
|
||
|
|
passes the Litestream key in as the credential `nas_key`; uid 10001 cannot
|
||
|
|
otherwise reach `/var/lib/docker` or `deploy/secrets/`. The script refuses a
|
||
|
|
`media` path that is a symlink and fails (exit 1) if `media` is missing.
|
||
|
|
|
||
|
|
One-time setup (already done on 10.60.1.14):
|
||
|
|
|
||
|
|
```sh
|
||
|
|
cd /home/overseer/mealquest
|
||
|
|
# Host user for the backup; ssh needs a passwd entry for its uid.
|
||
|
|
sudo groupadd --system --gid 10001 mealquest
|
||
|
|
sudo useradd --system --uid 10001 --gid 10001 --no-create-home \
|
||
|
|
--home-dir /nonexistent --shell /usr/sbin/nologin mealquest
|
||
|
|
# Pinned NAS host keys, root-owned and world-readable.
|
||
|
|
sudo install -d -o root -g root -m 0755 /etc/mealquest
|
||
|
|
sudo install -o root -g root -m 0644 deploy/known_hosts /etc/mealquest/known_hosts
|
||
|
|
# The media directory; the backup fails if it is ever missing.
|
||
|
|
sudo install -d -o 10001 -g 10001 -m 0755 /var/lib/docker/volumes/mealquest-data/_data/media
|
||
|
|
# Script and units.
|
||
|
|
sudo install -o root -g root -m 0755 deploy/systemd/mealquest-media-backup.sh /usr/local/sbin/mealquest-media-backup
|
||
|
|
sudo install -o root -g root -m 0644 deploy/systemd/mealquest-media-backup.service \
|
||
|
|
deploy/systemd/mealquest-media-backup.timer /etc/systemd/system/
|
||
|
|
sudo systemctl daemon-reload
|
||
|
|
sudo systemctl enable --now mealquest-media-backup.timer
|
||
|
|
```
|
||
|
|
|
||
|
|
Run it by hand and read the result:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
sudo systemctl start mealquest-media-backup.service
|
||
|
|
journalctl -u mealquest-media-backup --no-pager -n 30
|
||
|
|
```
|
||
|
|
|
||
|
|
Restore media by reversing the copy into the (empty or stale) media
|
||
|
|
directory as uid 10001:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
sudo rsync -a --chown=10001:10001 \
|
||
|
|
-e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \
|
||
|
|
joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \
|
||
|
|
/var/lib/docker/volumes/mealquest-data/_data/media/
|
||
|
|
```
|
||
|
|
|
||
|
|
## Restore the database
|
||
|
|
|
||
|
|
SQLite in WAL mode keeps recent commits in `mealquest.db-wal`, indexed by
|
||
|
|
`mealquest.db-shm`. **A restored database must never sit beside an old WAL
|
||
|
|
or shared-memory file**: on open, SQLite would replay the old WAL's pages
|
||
|
|
onto the restored database and corrupt it or mix in state from the
|
||
|
|
database it replaced. So all three files are moved aside together, with
|
||
|
|
Litestream's local state directory `.mealquest.db-litestream`, before the
|
||
|
|
restore — and the restore runs into an empty volume.
|
||
|
|
|
||
|
|
Run from `/home/overseer/mealquest`:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
D=/var/lib/docker/volumes/mealquest-data/_data
|
||
|
|
A=/var/tmp/mealquest-restore-aside-$(date +%Y%m%d%H%M)
|
||
|
|
|
||
|
|
# 1. Stop the whole stack (app and Litestream). Nothing may hold the
|
||
|
|
# database open; a clean close checkpoints the WAL into mealquest.db.
|
||
|
|
docker compose -f deploy/compose.yaml down
|
||
|
|
|
||
|
|
# 2. Move the database, its -wal and -shm files and Litestream's state
|
||
|
|
# aside — all of them, every time. Keep the directory until the
|
||
|
|
# restored stack is verified.
|
||
|
|
sudo mkdir "$A"
|
||
|
|
for f in mealquest.db mealquest.db-wal mealquest.db-shm .mealquest.db-litestream; do
|
||
|
|
if sudo test -e "$D/$f"; then sudo mv "$D/$f" "$A/"; fi
|
||
|
|
done
|
||
|
|
sudo ls -la "$D" # must list only media/
|
||
|
|
|
||
|
|
# 3. Restore from the NAS. --no-deps keeps the app stopped so it cannot
|
||
|
|
# create a fresh, empty database first.
|
||
|
|
docker compose -f deploy/compose.yaml run --rm --no-deps litestream \
|
||
|
|
restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db
|
||
|
|
sudo ls -la "$D" # mealquest.db only: no -wal, no -shm
|
||
|
|
|
||
|
|
# 4. Check the restored file before starting the app (see "Count check").
|
||
|
|
|
||
|
|
# 5. Start the stack and check health.
|
||
|
|
docker compose -f deploy/compose.yaml up -d
|
||
|
|
docker compose -f deploy/compose.yaml ps
|
||
|
|
curl -s -o /dev/null -w "health %{http_code}\n" http://10.60.1.14:8090/health
|
||
|
|
```
|
||
|
|
|
||
|
|
To restore to a point in time, add `-timestamp 2026-10-05T23:00:00Z` (or
|
||
|
|
`-txid`) to the `restore` command.
|
||
|
|
|
||
|
|
### Count check
|
||
|
|
|
||
|
|
`/tmp/mqcount.py` prints row counts for key tables, the number of
|
||
|
|
application tables, a SHA-256 over every application row (first 16 hex
|
||
|
|
digits; Litestream's own `_litestream*` tables are excluded) and
|
||
|
|
`PRAGMA integrity_check`. Run it on a **copy** of the database, never on the
|
||
|
|
live file; copy `-wal`/`-shm` along with it when the app is running.
|
||
|
|
|
||
|
|
```python
|
||
|
|
import sqlite3, sys, hashlib
|
||
|
|
c = sqlite3.connect(sys.argv[1])
|
||
|
|
tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
|
||
|
|
for t in ("dinners", "week_plans", "schema_migrations"):
|
||
|
|
print(t, c.execute(f"select count(*) from {t}").fetchone()[0])
|
||
|
|
h = hashlib.sha256()
|
||
|
|
for t in tabs:
|
||
|
|
for row in c.execute(f"select * from \"{t}\" order by rowid"):
|
||
|
|
h.update(repr(row).encode())
|
||
|
|
print("tables", len(tabs), "content-sha256", h.hexdigest()[:16])
|
||
|
|
print("integrity", c.execute("pragma integrity_check").fetchone()[0])
|
||
|
|
```
|
||
|
|
|
||
|
|
```sh
|
||
|
|
rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck
|
||
|
|
sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck
|
||
|
|
python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db
|
||
|
|
```
|
||
|
|
|
||
|
|
Before and after must match exactly.
|
||
|
|
|
||
|
|
## Restore drill record — 2026-10-05
|
||
|
|
|
||
|
|
Drill on 10.60.1.14 against the live (pre-launch, empty-household)
|
||
|
|
database, Litestream 0.5.17. Commands as in "Restore the database";
|
||
|
|
output trimmed only of compose progress lines.
|
||
|
|
|
||
|
|
Replica before the drill (`ltx -level all`; level 9 is the full snapshot):
|
||
|
|
|
||
|
|
```
|
||
|
|
level min_txid max_txid size created
|
||
|
|
0 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z
|
||
|
|
1 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
1 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z
|
||
|
|
2 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
2 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z
|
||
|
|
3 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
9 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
```
|
||
|
|
|
||
|
|
1. `docker compose -f deploy/compose.yaml down` — both containers stopped
|
||
|
|
and removed. The volume then held `mealquest.db`,
|
||
|
|
`.mealquest.db-litestream/` and `media/`; the clean close had
|
||
|
|
checkpointed and removed `-wal` and `-shm`.
|
||
|
|
2. Before (copy of the stopped database):
|
||
|
|
|
||
|
|
```
|
||
|
|
dinners 0
|
||
|
|
week_plans 0
|
||
|
|
schema_migrations 7
|
||
|
|
tables 26 content-sha256 836e230e16c58dc7
|
||
|
|
integrity ok
|
||
|
|
```
|
||
|
|
|
||
|
|
3. Moved aside to `/var/tmp/mealquest-restore-aside-20261005`:
|
||
|
|
`mealquest.db` and `.mealquest.db-litestream` (no `-wal`/`-shm` existed
|
||
|
|
after the clean stop; the loop moves them whenever they do). Volume
|
||
|
|
afterwards: `media/` only.
|
||
|
|
4. `docker compose -f deploy/compose.yaml run --rm --no-deps litestream restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db`
|
||
|
|
— volume afterwards: `mealquest.db` (258048 bytes) and `media/`; no
|
||
|
|
`-wal`, no `-shm`.
|
||
|
|
5. After (copy of the restored file, before the app started):
|
||
|
|
|
||
|
|
```
|
||
|
|
dinners 0
|
||
|
|
week_plans 0
|
||
|
|
schema_migrations 7
|
||
|
|
tables 26 content-sha256 836e230e16c58dc7
|
||
|
|
integrity ok
|
||
|
|
```
|
||
|
|
|
||
|
|
6. `docker compose -f deploy/compose.yaml up -d` — `mealquest-mealquest-1`
|
||
|
|
healthy, `mealquest-litestream-1` up; `GET /health` → `200`. Counts on a
|
||
|
|
copy of the running database (`mealquest.db`, `-wal`, `-shm`) were
|
||
|
|
identical to the above.
|
||
|
|
7. Litestream resumed against the restored database:
|
||
|
|
|
||
|
|
```
|
||
|
|
level=INFO msg="detected database behind replica" db=mealquest.db db_txid=0000000000000000 replica_txid=0000000000000003
|
||
|
|
level=INFO msg="fetched latest L0 file from replica" db=mealquest.db min_txid=0000000000000003 max_txid=0000000000000003
|
||
|
|
level=INFO msg="compaction complete" db=mealquest.db level=1 txid.min=0000000000000004 txid.max=0000000000000004 size=14110
|
||
|
|
```
|
||
|
|
|
||
|
|
Replica after the drill (`ltx -level all`):
|
||
|
|
|
||
|
|
```
|
||
|
|
level min_txid max_txid size created
|
||
|
|
0 0000000000000004 0000000000000004 14110 2026-10-05T23:27:21Z
|
||
|
|
1 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
1 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z
|
||
|
|
1 0000000000000004 0000000000000004 14110 2026-10-05T23:27:21Z
|
||
|
|
2 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
2 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z
|
||
|
|
3 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
9 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z
|
||
|
|
```
|
||
|
|
|
||
|
|
The first drill earlier the same day (before `-level all` was recorded)
|
||
|
|
gave the same before/after result: checksum `836e230e16c58dc7`, 26 tables,
|
||
|
|
integrity ok.
|
||
|
|
|
||
|
|
### Media backup checks — 2026-10-05
|
||
|
|
|
||
|
|
`sudo systemctl start mealquest-media-backup.service` on 10.60.1.14:
|
||
|
|
|
||
|
|
| Case | Result |
|
||
|
|
| --- | --- |
|
||
|
|
| Normal run | `Result=success`; journal: `synced /run/mealquest-data/media to joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ as uid 10001` |
|
||
|
|
| A test file added | `Number of regular files transferred: 1`; removing it then gave `Number of deleted files: 1 (reg: 1)` |
|
||
|
|
| `media` replaced by a symlink to `/etc` | `Result=exit-code`, status 1; `/run/mealquest-data/media is a symlink to /etc; refusing to back it up` |
|
||
|
|
| `media` removed | `Result=exit-code`, status 1; `media directory /run/mealquest-data/media is missing` |
|
||
|
|
| `sudo -u mealquest` writing into the volume via `/var/lib/docker/...` | `Permission denied` — uid 10001 cannot traverse `/var/lib/docker` (mode 0710 root) |
|