From 25c1701ddca1118efac628d10528a39eb073cb3b Mon Sep 17 00:00:00 2001 From: dev_agent Date: Mon, 5 Oct 2026 19:51:34 -0400 Subject: [PATCH 1/4] Complete the deployment runbook Add prerequisites, the first deploy on 10.60.1.14 (clone path, env file, secrets layout and permissions, port 8090 check), Authentik provider, application and group settings, DNS records with the AAAA caveat, the DOCKER-USER firewall unit, routine deploy and rollback to a previous SHA, photo restore, and secret rotation for the app key, OIDC secret, LLM key, VAPID keys and the NAS key. Record the host checkout state left by the pre-release first deploy. Task: 876134bc-9f1c-46b3-bc3c-0d889c2fd5f6 Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 448 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 442 insertions(+), 6 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 89534f5..93433a9 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -1,9 +1,10 @@ # MealQuest deployment runbook -How MealQuest is deployed on office-srvr (10.60.1.14) and how its data is -backed up and restored. Sections marked *task 876134bc* are completed by -that task (Feature 07 deployment runbook); this file currently records the -reverse-proxy route, the backups and the restore procedure. +How MealQuest is deployed on office-srvr (10.60.1.14), how it is reached +through Authentik, Traefik and DNS, how it is updated and rolled back, how +its data is backed up and restored, and how its secrets are rotated. Values +are the ones in use on 2026-10-05. Secrets themselves are never written +here. Paths used below: @@ -17,10 +18,194 @@ Paths used below: | Backup target (NAS, over SSH) | `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups` | | Same target over SFTP (Synology chroot) | `/homes/Joshua/mealquest-backups` | +## Prerequisites + +On office-srvr (10.60.1.14), as recorded on 2026-10-05: + +| Item | Value | +| --- | --- | +| OS | Ubuntu 26.04.1 LTS | +| Docker Engine / Compose plugin | 29.8.2 / 5.6.0 | +| git, rsync, iptables | 2.53.0, 3.4.1, 1.8.11 (nf_tables) | +| Account | `overseer`, with sudo | +| Network | LAN reachability to Traefik (10.60.1.33), Authentik (`auth.cubecraftlabs.com`) and the NAS (10.60.1.50, SSH) | + +Elsewhere: + +- Admin access to Authentik at . +- SSH access to the Traefik host 10.60.1.33. +- Cloudflare access for the `cubecraftlabs.com` zone, and admin access to + both Pi-holes (10.60.1.101 `dns.ccl`, 10.60.1.85 `dns2.ccl`). +- A NAS account `joshua` on 10.60.1.50 with the backup directory + `/volume1/homes/Joshua/mealquest-backups` (see "Backups"). +- A Gitea token able to read `Maniacs_Mansion/MealQuest`, for the clone. + ## Stack and first deploy -*Completed by task 876134bc.* The compose stack is `deploy/compose.yaml`; -start it with `docker compose -f deploy/compose.yaml up -d --build`. +The compose stack is `deploy/compose.yaml`: the `mealquest` service (built +from the repository `Dockerfile`, runs as uid/gid 10001, listens on 8080 in +the container) and the `litestream` service. Data lives in the named volume +`mealquest-data`. + +### 1. Choose and check the port + +The container's 8080 is published on `10.60.1.14:8090`. Ports 8080 and 8081 +on this host belong to NestQuest, so 8090 was chosen. Before the first +deploy it was checked free with: + +```sh +ss -ltnp | grep 8090 # printed nothing +``` + +After the deploy the same host shows NestQuest on 8080/8081 and MealQuest +on 8090: + +``` +LISTEN 0 4096 10.60.1.14:8080 0.0.0.0:* +LISTEN 0 4096 10.60.1.14:8081 0.0.0.0:* +LISTEN 0 4096 10.60.1.14:8090 0.0.0.0:* +``` + +The published host port is fixed in `deploy/compose.yaml`, in +`deploy/systemd/mealquest-firewall.sh` and in the Traefik route; change all +three together. + +### 2. Clone + +The checkout lives at `/home/overseer/mealquest`, next to NestQuest's +`~/nestquest` (not `/opt/mealquest`). Clone over HTTPS with a Gitea token, +then reset the remote to the clean URL so the token is not stored in +`.git/config`: + +```sh +cd /home/overseer +git clone https://:@code.cubecraftlabs.com/Maniacs_Mansion/MealQuest.git mealquest +cd mealquest +git remote set-url origin https://code.cubecraftlabs.com/Maniacs_Mansion/MealQuest.git +git remote -v # must show no token +git checkout main +``` + +### 3. Environment file + +`deploy/.env` (git-ignored) is copied from `deploy/.env.example` and is +mode 0600, owned by `overseer`: + +```sh +cp deploy/.env.example deploy/.env +chmod 600 deploy/.env +``` + +Values in use (the client id is not a secret but is read from the +Authentik provider, not repeated here): + +| Variable | Value | +| --- | --- | +| `MEALQUEST_BIND_ADDRESS` | `10.60.1.14` | +| `MEALQUEST_PUBLIC_URL` | `https://mealquest.cubecraftlabs.com` | +| `MEALQUEST_SHUTDOWN_TIMEOUT` | `10s` | +| `MEALQUEST_TRUSTED_PROXY` | `10.60.1.33` | +| `MEALQUEST_DEFAULT_TZ` | `America/New_York` | +| `MEALQUEST_OIDC_ISSUER` | `https://auth.cubecraftlabs.com/application/o/mealquest/` | +| `MEALQUEST_OIDC_CLIENT_ID` | the Client ID of the Authentik provider `mealquest` | +| `MEALQUEST_OIDC_ALLOWED_GROUPS` | `mealquest-admin` | +| `MEALQUEST_SESSION_IDLE` / `MEALQUEST_SESSION_MAX` | `720h` / `4320h` | + +`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR` and the `*_FILE` variables are +set by `deploy/compose.yaml` and must not appear in `deploy/.env`. + +### 4. Secrets + +`deploy/secrets/` (git-ignored) is mode 0700, owned by `overseer`. Each +file holds one value, is owned by `10001:10001` (host user `mealquest`) and +is mode 0600. Compose bind-mounts file secrets with their host owner and +mode, so any other owner or mode stops the container reading them and the +service refuses to start. + +| File | Content | Mounted at | +| --- | --- | --- | +| `secret_key` | base64 of 32 random bytes; encrypts stored secrets (AES-256-GCM) | `/run/secrets/secret_key` | +| `oidc_client_secret` | Client Secret of the Authentik provider `mealquest` | `/run/secrets/oidc_client_secret` | +| `litestream_key` | dedicated SSH private key for the NAS (see "Backups") | `/etc/litestream/sftp_key` | + +```sh +mkdir -p deploy/secrets && chmod 700 deploy/secrets +head -c 32 /dev/urandom | base64 > deploy/secrets/secret_key +printf '%s\n' '' > deploy/secrets/oidc_client_secret +# litestream_key: see "Backups" > "Database — Litestream" +sudo chown 10001:10001 deploy/secrets/* +sudo chmod 600 deploy/secrets/* +sudo ls -la deploy/secrets # three files, mealquest mealquest, -rw------- +``` + +The `mealquest` host user (uid/gid 10001) is created in "Media — nightly +rsync"; until it exists `ls` shows the numeric ids. + +### 5. Firewall, then the stack + +Install the firewall unit first (see "Host firewall"), so port 8090 is +never open to the LAN. Create the media directory (see "Media — nightly +rsync"), then build and start: + +```sh +sudo systemctl start mealquest-firewall +docker compose -f deploy/compose.yaml up -d --build +docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up +curl -fsS http://10.60.1.14:8090/health +curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 +``` + +### Host checkout state on 2026-10-05 + +The first deploy ran before Feature 07 reached `main`. The checkout is at +`main` `9cddbbb` and the Feature 07 files under `deploy/` were copied in +from the feature branch: `deploy/compose.yaml` and `deploy/.env.example` +are listed in `.git/info/exclude`, the rest show as untracked. Before the +first routine deploy of a `main` that contains `deploy/`, remove those +`.git/info/exclude` lines and the copied files (keep `deploy/.env` and +`deploy/secrets/`), or `git checkout` refuses to overwrite them: + +```sh +cd /home/overseer/mealquest +git status --porcelain --untracked-files=all deploy # the copied files +sed -i '/^deploy\/\(compose.yaml\|.env.example\)$/d' .git/info/exclude +git clean -n deploy # dry run: must not list .env or secrets/ (both git-ignored) +git clean -f deploy +``` + +Then follow "Routine deploy"; the checkout brings the tracked copies back. + +## Authentik + +Sign-in is OpenID Connect against Authentik at +. + +| Object | Setting | +| --- | --- | +| Provider | OAuth2/OpenID provider `mealquest`, client type **Confidential** | +| Redirect URI | `https://mealquest.cubecraftlabs.com/auth/callback` (strict) | +| Signing | RS256, with the instance's signing key | +| Scopes | MealQuest requests `openid profile email groups offline_access`; the provider's scope mappings include openid, profile, email and offline_access, and the `groups` claim is delivered by a property mapping (the default profile mapping carries `groups`) | +| Application | name MealQuest, slug `mealquest`, provider `mealquest` | +| Group | `mealquest-admin`, bound to the application; household members are added to it | +| Client ID | in `deploy/.env` as `MEALQUEST_OIDC_CLIENT_ID` | +| Client Secret | in `deploy/secrets/oidc_client_secret` only | + +MealQuest refuses a sign-in unless the user's `groups` claim (from the ID +token, or from userinfo when the token has none) contains a group in +`MEALQUEST_OIDC_ALLOWED_GROUPS`. + +Check discovery from the Docker host — the service discovers the issuer +at boot and will not start if it cannot: + +```sh +curl -s -o /dev/null -w "%{http_code}\n" \ + https://auth.cubecraftlabs.com/application/o/mealquest/.well-known/openid-configuration # 200 +``` + +On 2026-10-05 discovery returned 200 with issuer +`https://auth.cubecraftlabs.com/application/o/mealquest/` and +`id_token_signing_alg_values_supported: ["RS256"]`. ## Reverse proxy (Traefik) @@ -67,6 +252,133 @@ http: frameDeny: true ``` +Traefik watches the dynamic directory, so saving the file loads the route; +the certificate comes from `certResolver: myresolver` (Let's Encrypt, +DNS-01 through Cloudflare). Check from the LAN: + +```sh +curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 +``` + +## DNS + +| Where | Record | +| --- | --- | +| Cloudflare, zone `cubecraftlabs.com` | `A mealquest` → the public address, **proxied** (public resolvers answer with Cloudflare addresses) | +| Pi-hole primary 10.60.1.101 (`dns.ccl`), Local DNS records | `mealquest.cubecraftlabs.com` → `10.60.1.33` | +| Pi-hole secondary 10.60.1.85 (`dns2.ccl`), Local DNS records | `mealquest.cubecraftlabs.com` → `10.60.1.33` | + +The local record must be on **both** Pi-holes, or clients using the +secondary go out through Cloudflare. Check: + +```sh +for s in 10.60.1.101 10.60.1.85; do dig +short @$s mealquest.cubecraftlabs.com A; done +# 10.60.1.33 from each +``` + +**AAAA caveat.** The Pi-hole local record is A only, so both Pi-holes still +forward AAAA queries and return Cloudflare's IPv6 addresses +(`2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75` on 2026-10-05). An +IPv6-capable LAN client may prefer those and reach MealQuest through +Cloudflare instead of directly through Traefik. It still works (Cloudflare +proxies to the public route), but it is not the LAN path. If that matters, +add a local AAAA override or block AAAA for this name on both Pi-holes. + +## Host firewall + +Docker publishes `10.60.1.14:8090` past the host's INPUT rules, so the +port is restricted in the `DOCKER-USER` chain instead: only Traefik +(10.60.1.33) and the host itself (10.60.1.14) may connect; every other +source is dropped. + +Docker DNATs the published port to the container address and port 8080 +**before** `DOCKER-USER`, so a plain `-p tcp --dport 8090` rule never +matches. The rules match the original destination `10.60.1.14:8090` with +conntrack (`--ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL`) +and are tagged with the comment `mealquest-firewall`. The script deletes +its own tagged rules and re-inserts them at positions 1–3, so re-runs are +idempotent. NestQuest's 8080/8081 rules below them are left untouched. + +| Repository file | Installed as | +| --- | --- | +| `deploy/systemd/mealquest-firewall.sh` | `/usr/local/sbin/mealquest-firewall` (root, 0755) | +| `deploy/systemd/mealquest-firewall.service` | `/etc/systemd/system/mealquest-firewall.service` (root, 0644) | + +The unit is `Type=oneshot`, `RemainAfterExit=yes`, `Requires=` and +`PartOf=docker.service`, and `WantedBy=multi-user.target docker.service`: +it re-applies the rules whenever Docker starts or restarts (Docker +recreates `DOCKER-USER` empty), and a failure fails this unit only. + +Install (done on 10.60.1.14 on 2026-10-05): + +```sh +cd /home/overseer/mealquest +sudo install -o root -g root -m 0755 deploy/systemd/mealquest-firewall.sh /usr/local/sbin/mealquest-firewall +sudo install -o root -g root -m 0644 deploy/systemd/mealquest-firewall.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now mealquest-firewall +``` + +Check: + +```sh +systemctl is-enabled mealquest-firewall; systemctl is-active mealquest-firewall # enabled, active +sudo iptables -S DOCKER-USER +``` + +Expected head of the chain (recorded 2026-10-05; NestQuest's rules follow): + +``` +-A DOCKER-USER -s 10.60.1.33/32 -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j RETURN +-A DOCKER-USER -s 10.60.1.14/32 -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j RETURN +-A DOCKER-USER -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j DROP +``` + +From any other LAN machine, `curl -m 5 http://10.60.1.14:8090/health` must +time out; from 10.60.1.33 and 10.60.1.14 it returns 200. + +## Routine deploy and rollback + +Production runs `main` (a release); only `overseer` merges `dev` → `main`. +Every deploy builds the image on the host from the checked-out commit. + +### Deploy + +```sh +cd /home/overseer/mealquest +git fetch origin +git rev-parse --short HEAD # the SHA being replaced: note it for rollback +git checkout --detach origin/main # or a release tag +docker compose -f deploy/compose.yaml up -d --build +docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up +curl -fsS http://10.60.1.14:8090/health +curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 +git rev-parse HEAD # the deployed SHA +``` + +Record the deployed SHA, the previous SHA and the date in the Maestro task +for the release. Migrations run at start-up; the service refuses to start +if the database has an applied migration it does not know. + +If `deploy/.env.example` gained a variable, add it to `deploy/.env` before +`up`; the service fails fast at boot naming any missing required variable. + +### Rollback to a previous SHA + +```sh +cd /home/overseer/mealquest +git checkout --detach +docker compose -f deploy/compose.yaml up -d --build +docker compose -f deploy/compose.yaml ps +curl -fsS http://10.60.1.14:8090/health +``` + +A rollback across a release that added a migration fails at start-up +(the older binary refuses the newer schema). In that case restore the +database to a point before the upgrade — "Restore the database" with +`-timestamp` set just before the deploy — then start the older SHA. +Changes made since that point are lost; say so to the household. + ## Backups ### Database — Litestream @@ -225,6 +537,44 @@ python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db Before and after must match exactly. +## Restore photos (media) + +Recipe and dinner photos live in `media/` in the data volume and are +backed up nightly to `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/` +(see "Media — nightly rsync"). The nightly job mirrors with `--delete`: +**stop the timer first**, or a run against an emptied or partial `media/` +would delete the NAS copy you are about to restore from. + +Run from `/home/overseer/mealquest`: + +```sh +M=/var/lib/docker/volumes/mealquest-data/_data/media + +# 1. Stop the nightly mirror and the app (no writes into media/ meanwhile). +sudo systemctl stop mealquest-media-backup.timer +docker compose -f deploy/compose.yaml stop mealquest + +# 2. Recreate the directory if it is gone, owned by the app user. +sudo install -d -o 10001 -g 10001 -m 0755 "$M" + +# 3. Copy back from the NAS as uid 10001's files. +sudo rsync -a --stats --chown=10001:10001 \ + -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \ + joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \ + "$M/" +sudo find "$M" -type f | wc -l # compare with the file count on the NAS + +# 4. Start the app and the timer again. +docker compose -f deploy/compose.yaml up -d +curl -fsS http://10.60.1.14:8090/health +sudo systemctl start mealquest-media-backup.timer +``` + +When the database is restored too, restore it first ("Restore the +database"), then the media, so every photo the database refers to is +present. This media restore has not been run against production yet; the +backup direction was checked on 2026-10-05 (see "Media backup checks"). + ## Restore drill record — 2026-10-05 Drill on 10.60.1.14 against the live (pre-launch, empty-household) @@ -316,3 +666,89 @@ integrity ok. | `media` replaced by a symlink to `/etc` | `Result=exit-code`, status 1; `/run/mealquest-data/media is a symlink to /etc; refusing to back it up` | | `media` removed | `Result=exit-code`, status 1; `media directory /run/mealquest-data/media is missing` | | `sudo -u mealquest` writing into the volume via `/var/lib/docker/...` | `Permission denied` — uid 10001 cannot traverse `/var/lib/docker` (mode 0710 root) | + +## Secret rotation + +Rotate a secret when it may have leaked, when someone with access leaves, +or on the owner's schedule. Never paste a secret into a ticket, chat or +this file. + +### App key (`deploy/secrets/secret_key`) + +The app key encrypts every stored secret (the `_ct` columns: the LLM API +key, the recipe-search API key, the Google Calendar refresh token and the +Google Chat webhook URL). +**Rotating it re-encrypts nothing.** Values encrypted with the old key can +no longer be read and must be entered again in the app. A +`mealquest rotate-key` helper that re-encrypts in place is planned but does +not exist yet. + +```sh +cd /home/overseer/mealquest +sudo cp -a deploy/secrets/secret_key /var/tmp/secret_key.old # keep until verified, then shred +head -c 32 /dev/urandom | base64 | sudo tee deploy/secrets/secret_key >/dev/null +sudo chown 10001:10001 deploy/secrets/secret_key +sudo chmod 600 deploy/secrets/secret_key +docker compose -f deploy/compose.yaml up -d --force-recreate mealquest +curl -fsS http://10.60.1.14:8090/health +``` + +Then re-enter every stored secret (the LLM key in Settings → AI assistant, +the recipe-search key, reconnect Google Calendar, the Google Chat webhook) +and confirm each works. +Finally `sudo shred -u /var/tmp/secret_key.old`. + +### OIDC client secret (`deploy/secrets/oidc_client_secret`) + +In Authentik, open the provider `mealquest` and regenerate the Client +Secret. Write it to the file (same owner and mode as above) and recreate +the service: + +```sh +printf '%s\n' '' | sudo tee deploy/secrets/oidc_client_secret >/dev/null +sudo chown 10001:10001 deploy/secrets/oidc_client_secret +sudo chmod 600 deploy/secrets/oidc_client_secret +docker compose -f deploy/compose.yaml up -d --force-recreate mealquest +``` + +Sign in at to confirm; sign-ins +fail between regenerating the secret in Authentik and recreating the +service. + +### LLM API key + +Stored encrypted in the database, not in `deploy/`. In the app, open +**Settings → AI assistant**, choose **Replace**, paste the new key and +**Verify**, and check the verified state is shown. Then revoke the +old key in the provider's console. (The AI assistant settings page is +delivered by its own feature; until it ships there is no stored LLM key +to rotate.) + +### VAPID keys (Web Push) + +Planned with Web Push (Feature 19): the public key goes in +`deploy/.env` as `MEALQUEST_VAPID_PUBLIC_KEY`, the private key in +`deploy/secrets/vapid_private_key` (owner `10001:10001`, mode 0600), and +the pair is generated with `cmd/vapidgen`. **Changing the VAPID pair +invalidates every push subscription**: each device must turn +notifications on again (`/settings/notifications`) before it receives +pushes. Rotate only when the private key may have leaked, write both +halves, recreate the service with `--force-recreate mealquest`, and tell +the household to re-enable notifications on each device. + +### Litestream / NAS key (`deploy/secrets/litestream_key`) + +Generate a new keypair as in "Database — Litestream", add the new public +key to `joshua@10.60.1.50:~/.ssh/authorized_keys`, replace the file +(owner `10001:10001`, mode 0600), run +`docker compose -f deploy/compose.yaml up -d --force-recreate litestream` +and `sudo systemctl start mealquest-media-backup.service`, check both +(`litestream ltx` and the journal), then remove the old public key from +the NAS. + +## Context records + +The confirmed values in this runbook (port 8090, checkout path, Authentik +group `mealquest-admin`, DNS, firewall, backup target) are also recorded in +the CTXD project `mealquest`: `RUNBOOKS.MD` (deployment procedure) and the +open questions in `CONTEXT.MD`, updated for Feature 07 task 876134bc. -- 2.55.0 From d55fe5a060b580097d3ea4a9eb29aeb7d2a8447f Mon Sep 17 00:00:00 2001 From: dev_agent Date: Mon, 5 Oct 2026 20:03:06 -0400 Subject: [PATCH 2/4] Address runbook review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Media restore now stops both mealquest-media-backup.timer and the service, checks that neither is active and no rsync runs, and records the NAS file count before media/ is touched; a post-condition compares local and NAS counts before the timer starts again, so an rsync --delete never sees an empty or partial media directory. The shortcut restore snippet in "Media — nightly rsync" now points at that procedure. First deploy sets MEALQUEST_BIND_ADDRESS to 10.60.1.14 explicitly (the template ships 127.0.0.1) and shows the resulting 10.60.1.14:8090 publish. Record the Authentik client ID and the actual DNS records. App-key rotation no longer sends the operator to re-entry flows that do not exist: it pre-checks that no _ct column holds a value, refuses to rotate otherwise, and marks the LLM, recipe-search (Feature 10), Calendar (Feature 16) and Chat (Feature 19) re-entry flows as planned. Task: 876134bc-9f1c-46b3-bc3c-0d889c2fd5f6 Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 146 +++++++++++++++++++++++++++++++++------------- 1 file changed, 105 insertions(+), 41 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 93433a9..522500b 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -96,8 +96,27 @@ cp deploy/.env.example deploy/.env chmod 600 deploy/.env ``` -Values in use (the client id is not a secret but is read from the -Authentik provider, not repeated here): +Then edit `deploy/.env` to the values below. In particular +**`MEALQUEST_BIND_ADDRESS` must be changed**: `.env.example` ships +`127.0.0.1` (a dev machine), and left as is the service listens only on +the host's loopback, which Traefik cannot reach. Set it to the LAN address +and check: + +```sh +sed -i 's/^MEALQUEST_BIND_ADDRESS=.*/MEALQUEST_BIND_ADDRESS=10.60.1.14/' deploy/.env +grep '^MEALQUEST_BIND_ADDRESS=' deploy/.env # MEALQUEST_BIND_ADDRESS=10.60.1.14 +docker compose -f deploy/compose.yaml config | grep -A4 'ports:' +# host_ip: 10.60.1.14, target: 8080, published: "8090" +``` + +After step 5 the listener is `10.60.1.14:8090` (container port 8080): + +```sh +docker compose -f deploy/compose.yaml port mealquest 8080 # 10.60.1.14:8090 +sudo ss -ltnp 'sport = :8090' # 10.60.1.14:8090 only, no 0.0.0.0 or 127.0.0.1 +``` + +Values in use: | Variable | Value | | --- | --- | @@ -107,7 +126,7 @@ Authentik provider, not repeated here): | `MEALQUEST_TRUSTED_PROXY` | `10.60.1.33` | | `MEALQUEST_DEFAULT_TZ` | `America/New_York` | | `MEALQUEST_OIDC_ISSUER` | `https://auth.cubecraftlabs.com/application/o/mealquest/` | -| `MEALQUEST_OIDC_CLIENT_ID` | the Client ID of the Authentik provider `mealquest` | +| `MEALQUEST_OIDC_CLIENT_ID` | `i6zraqmWsF552b8nmte070LXz0SI5pRdArBxFxeq` (not a secret) | | `MEALQUEST_OIDC_ALLOWED_GROUPS` | `mealquest-admin` | | `MEALQUEST_SESSION_IDLE` / `MEALQUEST_SESSION_MAX` | `720h` / `4320h` | @@ -188,7 +207,7 @@ Sign-in is OpenID Connect against Authentik at | Scopes | MealQuest requests `openid profile email groups offline_access`; the provider's scope mappings include openid, profile, email and offline_access, and the `groups` claim is delivered by a property mapping (the default profile mapping carries `groups`) | | Application | name MealQuest, slug `mealquest`, provider `mealquest` | | Group | `mealquest-admin`, bound to the application; household members are added to it | -| Client ID | in `deploy/.env` as `MEALQUEST_OIDC_CLIENT_ID` | +| Client ID | `i6zraqmWsF552b8nmte070LXz0SI5pRdArBxFxeq` (not a secret; in `deploy/.env` as `MEALQUEST_OIDC_CLIENT_ID`) | | Client Secret | in `deploy/secrets/oidc_client_secret` only | MealQuest refuses a sign-in unless the user's `groups` claim (from the ID @@ -264,9 +283,9 @@ curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/hea | Where | Record | | --- | --- | -| Cloudflare, zone `cubecraftlabs.com` | `A mealquest` → the public address, **proxied** (public resolvers answer with Cloudflare addresses) | -| Pi-hole primary 10.60.1.101 (`dns.ccl`), Local DNS records | `mealquest.cubecraftlabs.com` → `10.60.1.33` | -| Pi-hole secondary 10.60.1.85 (`dns2.ccl`), Local DNS records | `mealquest.cubecraftlabs.com` → `10.60.1.33` | +| Cloudflare, zone `cubecraftlabs.com` | `A mealquest` (`mealquest.cubecraftlabs.com`), **proxied** (orange cloud), pointing at the site's public origin address. Because it is proxied, public resolvers answer with Cloudflare's anycast addresses, A and AAAA (AAAA on 2026-10-05: `2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75`), never the origin. The origin address itself is not recorded in this runbook or in CTXD; read it from the Cloudflare dashboard | +| Pi-hole primary 10.60.1.101 (`dns.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | +| Pi-hole secondary 10.60.1.85 (`dns2.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | The local record must be on **both** Pi-holes, or clients using the secondary go out through Cloudflare. Check: @@ -451,15 +470,8 @@ sudo systemctl start mealquest-media-backup.service journalctl -u mealquest-media-backup --no-pager -n 30 ``` -Restore media by reversing the copy into the (empty or stale) media -directory as uid 10001: - -```sh -sudo rsync -a --chown=10001:10001 \ - -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \ - joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \ - /var/lib/docker/volumes/mealquest-data/_data/media/ -``` +To restore media, follow "Restore photos (media)" — never copy back while +the timer or the service can still run. ## Restore the database @@ -541,35 +553,58 @@ Before and after must match exactly. Recipe and dinner photos live in `media/` in the data volume and are backed up nightly to `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/` -(see "Media — nightly rsync"). The nightly job mirrors with `--delete`: -**stop the timer first**, or a run against an emptied or partial `media/` -would delete the NAS copy you are about to restore from. +(see "Media — nightly rsync"). The backup service mirrors with +`rsync --delete`. **An `rsync --delete` run must never see an empty or +partial `media/`**: it would delete the NAS copy you are about to restore +from. Stopping the timer is not enough — it does not stop a run that is +already in flight, and the service can still be started by hand. So stop +**both** the timer and the service, confirm no rsync is running, and do +not start the service until the restore is verified. Run from `/home/overseer/mealquest`: ```sh M=/var/lib/docker/volumes/mealquest-data/_data/media +NAS=/volume1/homes/Joshua/mealquest-backups/media +SSH="ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" -# 1. Stop the nightly mirror and the app (no writes into media/ meanwhile). -sudo systemctl stop mealquest-media-backup.timer +# 1. Stop the nightly mirror: the timer AND any run in flight. +sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service + +# 2. Pre-check — all three must hold before media/ is touched. If any +# fails, stop here and find out why. +systemctl is-active mealquest-media-backup.timer # inactive +systemctl is-active mealquest-media-backup.service # inactive (or failed) +pgrep -a -u mealquest rsync || echo "no rsync" # no rsync +sudo $SSH joshua@10.60.1.50 "find $NAS -type f | wc -l" +# ^ note this NAS file count; it must be non-zero and is the target below. + +# 3. Stop the app (no writes into media/ meanwhile). docker compose -f deploy/compose.yaml stop mealquest -# 2. Recreate the directory if it is gone, owned by the app user. +# 4. Recreate the directory if it is gone, owned by the app user. sudo install -d -o 10001 -g 10001 -m 0755 "$M" -# 3. Copy back from the NAS as uid 10001's files. -sudo rsync -a --stats --chown=10001:10001 \ - -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \ - joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \ - "$M/" -sudo find "$M" -type f | wc -l # compare with the file count on the NAS +# 5. Copy back from the NAS as uid 10001's files. No --delete in this +# direction: the NAS copy is the source and is only read. +sudo rsync -a --stats --chown=10001:10001 -e "$SSH" "joshua@10.60.1.50:$NAS/" "$M/" -# 4. Start the app and the timer again. +# 6. Post-condition — before anything can mirror again: +sudo find "$M" -type f | wc -l # >= the NAS count from step 2 +sudo $SSH joshua@10.60.1.50 "find $NAS -type f | wc -l" # unchanged from step 2 + +# 7. Start the app, then the timer again. Only now may the service run. docker compose -f deploy/compose.yaml up -d curl -fsS http://10.60.1.14:8090/health sudo systemctl start mealquest-media-backup.timer +systemctl is-active mealquest-media-backup.timer # active ``` +The local count may exceed the NAS count only if `media/` still held +files the NAS never received (photos added since the last nightly run); +the next nightly run then copies them up. If the local count is lower, +the copy did not complete: leave the timer stopped and re-run step 5. + When the database is restored too, restore it first ("Restore the database"), then the media, so every photo the database refers to is present. This media restore has not been run against production yet; the @@ -675,16 +710,45 @@ this file. ### App key (`deploy/secrets/secret_key`) -The app key encrypts every stored secret (the `_ct` columns: the LLM API -key, the recipe-search API key, the Google Calendar refresh token and the -Google Chat webhook URL). -**Rotating it re-encrypts nothing.** Values encrypted with the old key can -no longer be read and must be entered again in the app. A -`mealquest rotate-key` helper that re-encrypts in place is planned but does -not exist yet. +The app key encrypts every stored secret: the `_ct` columns +`ai_settings.api_key_ct` (LLM API key), `recipe_search_settings.api_key_ct` +(recipe-search API key), `integrations_google.refresh_token_ct` (Google +Calendar refresh token) and `chat_settings.webhook_url_ct` (Google Chat +webhook URL). **Rotating it re-encrypts nothing**: a value encrypted with +the old key can no longer be read. A `mealquest rotate-key` helper that +re-encrypts in place is planned but does not exist yet. + +**What is recoverable today.** The service currently exposes only +household settings; no shipped flow writes or re-enters any of these four +secrets. The re-entry flows are planned: the LLM key in Settings → AI +assistant (the AI settings feature), the recipe-search key (Feature 10), +Google Calendar reconnect (Feature 16) and the Google Chat webhook +(Feature 19). So today the rotation is safe **only if no `_ct` value is +stored**, which the pre-check below proves. If the pre-check finds a +stored value, do not rotate: it could not be re-entered afterwards. Keep +the current key and wait for the matching flow (or the `rotate-key` +helper). + +Pre-check, on a **copy** of the database (as in "Count check"): + +```sh +D=/var/lib/docker/volumes/mealquest-data/_data +rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck +sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck +python3 - /tmp/mqcheck/mealquest.db <<'PY' +import sqlite3, sys +c = sqlite3.connect(sys.argv[1]) +for t, col in (("ai_settings", "api_key_ct"), ("recipe_search_settings", "api_key_ct"), + ("integrations_google", "refresh_token_ct"), ("chat_settings", "webhook_url_ct")): + print(t, col, c.execute(f"select count(*) from {t} where {col} is not null").fetchone()[0]) +PY +rm -rf /tmp/mqcheck +# every count must be 0; otherwise stop here +``` + +Rotate, from `/home/overseer/mealquest`: ```sh -cd /home/overseer/mealquest sudo cp -a deploy/secrets/secret_key /var/tmp/secret_key.old # keep until verified, then shred head -c 32 /dev/urandom | base64 | sudo tee deploy/secrets/secret_key >/dev/null sudo chown 10001:10001 deploy/secrets/secret_key @@ -693,10 +757,10 @@ docker compose -f deploy/compose.yaml up -d --force-recreate mealquest curl -fsS http://10.60.1.14:8090/health ``` -Then re-enter every stored secret (the LLM key in Settings → AI assistant, -the recipe-search key, reconnect Google Calendar, the Google Chat webhook) -and confirm each works. -Finally `sudo shred -u /var/tmp/secret_key.old`. +Then sign in and open household settings to confirm the service runs on +the new key. Finally `sudo shred -u /var/tmp/secret_key.old`. Once the +planned flows ship, this section gains a step per secret to re-enter it +and confirm it works. ### OIDC client secret (`deploy/secrets/oidc_client_secret`) -- 2.55.0 From 0f679439cc805fa94faacd7e8e7f131c59cae724 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Mon, 5 Oct 2026 20:09:59 -0400 Subject: [PATCH 3/4] State the exact Cloudflare record in the runbook DNS section Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 522500b..4f4cca3 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -283,7 +283,7 @@ curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/hea | Where | Record | | --- | --- | -| Cloudflare, zone `cubecraftlabs.com` | `A mealquest` (`mealquest.cubecraftlabs.com`), **proxied** (orange cloud), pointing at the site's public origin address. Because it is proxied, public resolvers answer with Cloudflare's anycast addresses, A and AAAA (AAAA on 2026-10-05: `2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75`), never the origin. The origin address itself is not recorded in this runbook or in CTXD; read it from the Cloudflare dashboard | +| Cloudflare, zone `cubecraftlabs.com` | `CNAME mealquest.cubecraftlabs.com` → `cubecraftlabs.com`, **proxied = true** (orange cloud). Because it is proxied, public resolvers answer with Cloudflare's anycast addresses, A and AAAA (AAAA on 2026-10-05: `2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75`), never the origin | | Pi-hole primary 10.60.1.101 (`dns.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | | Pi-hole secondary 10.60.1.85 (`dns2.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | -- 2.55.0 From c90d13d01aa1bf27e7532d4efa0a312bcd010bb0 Mon Sep 17 00:00:00 2001 From: dev_agent Date: Mon, 5 Oct 2026 20:17:55 -0400 Subject: [PATCH 4/4] Run the app-key pre-check and rotation with the stack stopped A raw copy of mealquest.db and its WAL taken while the stack runs is not a consistent snapshot: a checkpoint between the two copies can drop committed rows, so the _ct pre-check could report a false zero. Stop the app and Litestream before the pre-check and keep them stopped until the new key is in place. Co-Authored-By: Claude Opus 5.5 --- deploy/RUNBOOK.md | 39 +++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 4f4cca3..df1cc3b 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -729,12 +729,34 @@ stored value, do not rotate: it could not be re-entered afterwards. Keep the current key and wait for the matching flow (or the `rotate-key` helper). -Pre-check, on a **copy** of the database (as in "Count check"): +**The pre-check and the key replacement run with the whole stack +stopped**, from the pre-check until the new key is in place. A raw copy of +`mealquest.db` and `mealquest.db-wal` taken while the stack runs is **not** +a consistent snapshot: the two files are copied one after the other, and a +checkpoint (by the app or by Litestream) between the two copies can move +committed pages out of the WAL into `mealquest.db` after `mealquest.db` was +already copied, then reset the WAL before it is copied. The copy then +lacks those commits — possibly a stored `_ct` value — and the pre-check +reports a false 0. A running app could also store a `_ct` value after a +correct pre-check and before the key is replaced. Stopping both containers +removes every writer (Litestream also writes and checkpoints), and the +clean close checkpoints the WAL into `mealquest.db`, so the file alone is +complete. + +Run from `/home/overseer/mealquest`: ```sh D=/var/lib/docker/volumes/mealquest-data/_data + +# 1. Stop the whole stack (app and Litestream) and leave it stopped until +# step 4. Nothing may hold the database open. +docker compose -f deploy/compose.yaml down +sudo ls -la "$D" # mealquest.db and media/ only: no -wal, no -shm; if + # either exists, the close was not clean — stop here + +# 2. Pre-check, on a copy of the now-quiescent database. rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck -sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" && sudo chown -R "$USER" /tmp/mqcheck +sudo cp -a "$D/mealquest.db" /tmp/mqcheck/ && sudo chown -R "$USER" /tmp/mqcheck python3 - /tmp/mqcheck/mealquest.db <<'PY' import sqlite3, sys c = sqlite3.connect(sys.argv[1]) @@ -743,17 +765,18 @@ for t, col in (("ai_settings", "api_key_ct"), ("recipe_search_settings", "api_ke print(t, col, c.execute(f"select count(*) from {t} where {col} is not null").fetchone()[0]) PY rm -rf /tmp/mqcheck -# every count must be 0; otherwise stop here -``` +# every count must be 0; otherwise start the stack again on the current +# key (docker compose -f deploy/compose.yaml up -d) and stop here -Rotate, from `/home/overseer/mealquest`: - -```sh +# 3. Replace the key, still with the stack stopped. sudo cp -a deploy/secrets/secret_key /var/tmp/secret_key.old # keep until verified, then shred head -c 32 /dev/urandom | base64 | sudo tee deploy/secrets/secret_key >/dev/null sudo chown 10001:10001 deploy/secrets/secret_key sudo chmod 600 deploy/secrets/secret_key -docker compose -f deploy/compose.yaml up -d --force-recreate mealquest + +# 4. Start the stack on the new key and check health. +docker compose -f deploy/compose.yaml up -d +docker compose -f deploy/compose.yaml ps curl -fsS http://10.60.1.14:8090/health ``` -- 2.55.0