diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index 89534f5..df1cc3b 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -1,9 +1,10 @@ # MealQuest deployment runbook -How MealQuest is deployed on office-srvr (10.60.1.14) and how its data is -backed up and restored. Sections marked *task 876134bc* are completed by -that task (Feature 07 deployment runbook); this file currently records the -reverse-proxy route, the backups and the restore procedure. +How MealQuest is deployed on office-srvr (10.60.1.14), how it is reached +through Authentik, Traefik and DNS, how it is updated and rolled back, how +its data is backed up and restored, and how its secrets are rotated. Values +are the ones in use on 2026-10-05. Secrets themselves are never written +here. Paths used below: @@ -17,10 +18,213 @@ Paths used below: | Backup target (NAS, over SSH) | `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups` | | Same target over SFTP (Synology chroot) | `/homes/Joshua/mealquest-backups` | +## Prerequisites + +On office-srvr (10.60.1.14), as recorded on 2026-10-05: + +| Item | Value | +| --- | --- | +| OS | Ubuntu 26.04.1 LTS | +| Docker Engine / Compose plugin | 29.8.2 / 5.6.0 | +| git, rsync, iptables | 2.53.0, 3.4.1, 1.8.11 (nf_tables) | +| Account | `overseer`, with sudo | +| Network | LAN reachability to Traefik (10.60.1.33), Authentik (`auth.cubecraftlabs.com`) and the NAS (10.60.1.50, SSH) | + +Elsewhere: + +- Admin access to Authentik at . +- SSH access to the Traefik host 10.60.1.33. +- Cloudflare access for the `cubecraftlabs.com` zone, and admin access to + both Pi-holes (10.60.1.101 `dns.ccl`, 10.60.1.85 `dns2.ccl`). +- A NAS account `joshua` on 10.60.1.50 with the backup directory + `/volume1/homes/Joshua/mealquest-backups` (see "Backups"). +- A Gitea token able to read `Maniacs_Mansion/MealQuest`, for the clone. + ## Stack and first deploy -*Completed by task 876134bc.* The compose stack is `deploy/compose.yaml`; -start it with `docker compose -f deploy/compose.yaml up -d --build`. +The compose stack is `deploy/compose.yaml`: the `mealquest` service (built +from the repository `Dockerfile`, runs as uid/gid 10001, listens on 8080 in +the container) and the `litestream` service. Data lives in the named volume +`mealquest-data`. + +### 1. Choose and check the port + +The container's 8080 is published on `10.60.1.14:8090`. Ports 8080 and 8081 +on this host belong to NestQuest, so 8090 was chosen. Before the first +deploy it was checked free with: + +```sh +ss -ltnp | grep 8090 # printed nothing +``` + +After the deploy the same host shows NestQuest on 8080/8081 and MealQuest +on 8090: + +``` +LISTEN 0 4096 10.60.1.14:8080 0.0.0.0:* +LISTEN 0 4096 10.60.1.14:8081 0.0.0.0:* +LISTEN 0 4096 10.60.1.14:8090 0.0.0.0:* +``` + +The published host port is fixed in `deploy/compose.yaml`, in +`deploy/systemd/mealquest-firewall.sh` and in the Traefik route; change all +three together. + +### 2. Clone + +The checkout lives at `/home/overseer/mealquest`, next to NestQuest's +`~/nestquest` (not `/opt/mealquest`). Clone over HTTPS with a Gitea token, +then reset the remote to the clean URL so the token is not stored in +`.git/config`: + +```sh +cd /home/overseer +git clone https://:@code.cubecraftlabs.com/Maniacs_Mansion/MealQuest.git mealquest +cd mealquest +git remote set-url origin https://code.cubecraftlabs.com/Maniacs_Mansion/MealQuest.git +git remote -v # must show no token +git checkout main +``` + +### 3. Environment file + +`deploy/.env` (git-ignored) is copied from `deploy/.env.example` and is +mode 0600, owned by `overseer`: + +```sh +cp deploy/.env.example deploy/.env +chmod 600 deploy/.env +``` + +Then edit `deploy/.env` to the values below. In particular +**`MEALQUEST_BIND_ADDRESS` must be changed**: `.env.example` ships +`127.0.0.1` (a dev machine), and left as is the service listens only on +the host's loopback, which Traefik cannot reach. Set it to the LAN address +and check: + +```sh +sed -i 's/^MEALQUEST_BIND_ADDRESS=.*/MEALQUEST_BIND_ADDRESS=10.60.1.14/' deploy/.env +grep '^MEALQUEST_BIND_ADDRESS=' deploy/.env # MEALQUEST_BIND_ADDRESS=10.60.1.14 +docker compose -f deploy/compose.yaml config | grep -A4 'ports:' +# host_ip: 10.60.1.14, target: 8080, published: "8090" +``` + +After step 5 the listener is `10.60.1.14:8090` (container port 8080): + +```sh +docker compose -f deploy/compose.yaml port mealquest 8080 # 10.60.1.14:8090 +sudo ss -ltnp 'sport = :8090' # 10.60.1.14:8090 only, no 0.0.0.0 or 127.0.0.1 +``` + +Values in use: + +| Variable | Value | +| --- | --- | +| `MEALQUEST_BIND_ADDRESS` | `10.60.1.14` | +| `MEALQUEST_PUBLIC_URL` | `https://mealquest.cubecraftlabs.com` | +| `MEALQUEST_SHUTDOWN_TIMEOUT` | `10s` | +| `MEALQUEST_TRUSTED_PROXY` | `10.60.1.33` | +| `MEALQUEST_DEFAULT_TZ` | `America/New_York` | +| `MEALQUEST_OIDC_ISSUER` | `https://auth.cubecraftlabs.com/application/o/mealquest/` | +| `MEALQUEST_OIDC_CLIENT_ID` | `i6zraqmWsF552b8nmte070LXz0SI5pRdArBxFxeq` (not a secret) | +| `MEALQUEST_OIDC_ALLOWED_GROUPS` | `mealquest-admin` | +| `MEALQUEST_SESSION_IDLE` / `MEALQUEST_SESSION_MAX` | `720h` / `4320h` | + +`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR` and the `*_FILE` variables are +set by `deploy/compose.yaml` and must not appear in `deploy/.env`. + +### 4. Secrets + +`deploy/secrets/` (git-ignored) is mode 0700, owned by `overseer`. Each +file holds one value, is owned by `10001:10001` (host user `mealquest`) and +is mode 0600. Compose bind-mounts file secrets with their host owner and +mode, so any other owner or mode stops the container reading them and the +service refuses to start. + +| File | Content | Mounted at | +| --- | --- | --- | +| `secret_key` | base64 of 32 random bytes; encrypts stored secrets (AES-256-GCM) | `/run/secrets/secret_key` | +| `oidc_client_secret` | Client Secret of the Authentik provider `mealquest` | `/run/secrets/oidc_client_secret` | +| `litestream_key` | dedicated SSH private key for the NAS (see "Backups") | `/etc/litestream/sftp_key` | + +```sh +mkdir -p deploy/secrets && chmod 700 deploy/secrets +head -c 32 /dev/urandom | base64 > deploy/secrets/secret_key +printf '%s\n' '' > deploy/secrets/oidc_client_secret +# litestream_key: see "Backups" > "Database — Litestream" +sudo chown 10001:10001 deploy/secrets/* +sudo chmod 600 deploy/secrets/* +sudo ls -la deploy/secrets # three files, mealquest mealquest, -rw------- +``` + +The `mealquest` host user (uid/gid 10001) is created in "Media — nightly +rsync"; until it exists `ls` shows the numeric ids. + +### 5. Firewall, then the stack + +Install the firewall unit first (see "Host firewall"), so port 8090 is +never open to the LAN. Create the media directory (see "Media — nightly +rsync"), then build and start: + +```sh +sudo systemctl start mealquest-firewall +docker compose -f deploy/compose.yaml up -d --build +docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up +curl -fsS http://10.60.1.14:8090/health +curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 +``` + +### Host checkout state on 2026-10-05 + +The first deploy ran before Feature 07 reached `main`. The checkout is at +`main` `9cddbbb` and the Feature 07 files under `deploy/` were copied in +from the feature branch: `deploy/compose.yaml` and `deploy/.env.example` +are listed in `.git/info/exclude`, the rest show as untracked. Before the +first routine deploy of a `main` that contains `deploy/`, remove those +`.git/info/exclude` lines and the copied files (keep `deploy/.env` and +`deploy/secrets/`), or `git checkout` refuses to overwrite them: + +```sh +cd /home/overseer/mealquest +git status --porcelain --untracked-files=all deploy # the copied files +sed -i '/^deploy\/\(compose.yaml\|.env.example\)$/d' .git/info/exclude +git clean -n deploy # dry run: must not list .env or secrets/ (both git-ignored) +git clean -f deploy +``` + +Then follow "Routine deploy"; the checkout brings the tracked copies back. + +## Authentik + +Sign-in is OpenID Connect against Authentik at +. + +| Object | Setting | +| --- | --- | +| Provider | OAuth2/OpenID provider `mealquest`, client type **Confidential** | +| Redirect URI | `https://mealquest.cubecraftlabs.com/auth/callback` (strict) | +| Signing | RS256, with the instance's signing key | +| Scopes | MealQuest requests `openid profile email groups offline_access`; the provider's scope mappings include openid, profile, email and offline_access, and the `groups` claim is delivered by a property mapping (the default profile mapping carries `groups`) | +| Application | name MealQuest, slug `mealquest`, provider `mealquest` | +| Group | `mealquest-admin`, bound to the application; household members are added to it | +| Client ID | `i6zraqmWsF552b8nmte070LXz0SI5pRdArBxFxeq` (not a secret; in `deploy/.env` as `MEALQUEST_OIDC_CLIENT_ID`) | +| Client Secret | in `deploy/secrets/oidc_client_secret` only | + +MealQuest refuses a sign-in unless the user's `groups` claim (from the ID +token, or from userinfo when the token has none) contains a group in +`MEALQUEST_OIDC_ALLOWED_GROUPS`. + +Check discovery from the Docker host — the service discovers the issuer +at boot and will not start if it cannot: + +```sh +curl -s -o /dev/null -w "%{http_code}\n" \ + https://auth.cubecraftlabs.com/application/o/mealquest/.well-known/openid-configuration # 200 +``` + +On 2026-10-05 discovery returned 200 with issuer +`https://auth.cubecraftlabs.com/application/o/mealquest/` and +`id_token_signing_alg_values_supported: ["RS256"]`. ## Reverse proxy (Traefik) @@ -67,6 +271,133 @@ http: frameDeny: true ``` +Traefik watches the dynamic directory, so saving the file loads the route; +the certificate comes from `certResolver: myresolver` (Let's Encrypt, +DNS-01 through Cloudflare). Check from the LAN: + +```sh +curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 +``` + +## DNS + +| Where | Record | +| --- | --- | +| Cloudflare, zone `cubecraftlabs.com` | `CNAME mealquest.cubecraftlabs.com` → `cubecraftlabs.com`, **proxied = true** (orange cloud). Because it is proxied, public resolvers answer with Cloudflare's anycast addresses, A and AAAA (AAAA on 2026-10-05: `2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75`), never the origin | +| Pi-hole primary 10.60.1.101 (`dns.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | +| Pi-hole secondary 10.60.1.85 (`dns2.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | + +The local record must be on **both** Pi-holes, or clients using the +secondary go out through Cloudflare. Check: + +```sh +for s in 10.60.1.101 10.60.1.85; do dig +short @$s mealquest.cubecraftlabs.com A; done +# 10.60.1.33 from each +``` + +**AAAA caveat.** The Pi-hole local record is A only, so both Pi-holes still +forward AAAA queries and return Cloudflare's IPv6 addresses +(`2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75` on 2026-10-05). An +IPv6-capable LAN client may prefer those and reach MealQuest through +Cloudflare instead of directly through Traefik. It still works (Cloudflare +proxies to the public route), but it is not the LAN path. If that matters, +add a local AAAA override or block AAAA for this name on both Pi-holes. + +## Host firewall + +Docker publishes `10.60.1.14:8090` past the host's INPUT rules, so the +port is restricted in the `DOCKER-USER` chain instead: only Traefik +(10.60.1.33) and the host itself (10.60.1.14) may connect; every other +source is dropped. + +Docker DNATs the published port to the container address and port 8080 +**before** `DOCKER-USER`, so a plain `-p tcp --dport 8090` rule never +matches. The rules match the original destination `10.60.1.14:8090` with +conntrack (`--ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL`) +and are tagged with the comment `mealquest-firewall`. The script deletes +its own tagged rules and re-inserts them at positions 1–3, so re-runs are +idempotent. NestQuest's 8080/8081 rules below them are left untouched. + +| Repository file | Installed as | +| --- | --- | +| `deploy/systemd/mealquest-firewall.sh` | `/usr/local/sbin/mealquest-firewall` (root, 0755) | +| `deploy/systemd/mealquest-firewall.service` | `/etc/systemd/system/mealquest-firewall.service` (root, 0644) | + +The unit is `Type=oneshot`, `RemainAfterExit=yes`, `Requires=` and +`PartOf=docker.service`, and `WantedBy=multi-user.target docker.service`: +it re-applies the rules whenever Docker starts or restarts (Docker +recreates `DOCKER-USER` empty), and a failure fails this unit only. + +Install (done on 10.60.1.14 on 2026-10-05): + +```sh +cd /home/overseer/mealquest +sudo install -o root -g root -m 0755 deploy/systemd/mealquest-firewall.sh /usr/local/sbin/mealquest-firewall +sudo install -o root -g root -m 0644 deploy/systemd/mealquest-firewall.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now mealquest-firewall +``` + +Check: + +```sh +systemctl is-enabled mealquest-firewall; systemctl is-active mealquest-firewall # enabled, active +sudo iptables -S DOCKER-USER +``` + +Expected head of the chain (recorded 2026-10-05; NestQuest's rules follow): + +``` +-A DOCKER-USER -s 10.60.1.33/32 -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j RETURN +-A DOCKER-USER -s 10.60.1.14/32 -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j RETURN +-A DOCKER-USER -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j DROP +``` + +From any other LAN machine, `curl -m 5 http://10.60.1.14:8090/health` must +time out; from 10.60.1.33 and 10.60.1.14 it returns 200. + +## Routine deploy and rollback + +Production runs `main` (a release); only `overseer` merges `dev` → `main`. +Every deploy builds the image on the host from the checked-out commit. + +### Deploy + +```sh +cd /home/overseer/mealquest +git fetch origin +git rev-parse --short HEAD # the SHA being replaced: note it for rollback +git checkout --detach origin/main # or a release tag +docker compose -f deploy/compose.yaml up -d --build +docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up +curl -fsS http://10.60.1.14:8090/health +curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 +git rev-parse HEAD # the deployed SHA +``` + +Record the deployed SHA, the previous SHA and the date in the Maestro task +for the release. Migrations run at start-up; the service refuses to start +if the database has an applied migration it does not know. + +If `deploy/.env.example` gained a variable, add it to `deploy/.env` before +`up`; the service fails fast at boot naming any missing required variable. + +### Rollback to a previous SHA + +```sh +cd /home/overseer/mealquest +git checkout --detach +docker compose -f deploy/compose.yaml up -d --build +docker compose -f deploy/compose.yaml ps +curl -fsS http://10.60.1.14:8090/health +``` + +A rollback across a release that added a migration fails at start-up +(the older binary refuses the newer schema). In that case restore the +database to a point before the upgrade — "Restore the database" with +`-timestamp` set just before the deploy — then start the older SHA. +Changes made since that point are lost; say so to the household. + ## Backups ### Database — Litestream @@ -139,15 +470,8 @@ sudo systemctl start mealquest-media-backup.service journalctl -u mealquest-media-backup --no-pager -n 30 ``` -Restore media by reversing the copy into the (empty or stale) media -directory as uid 10001: - -```sh -sudo rsync -a --chown=10001:10001 \ - -e "ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" \ - joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ \ - /var/lib/docker/volumes/mealquest-data/_data/media/ -``` +To restore media, follow "Restore photos (media)" — never copy back while +the timer or the service can still run. ## Restore the database @@ -225,6 +549,67 @@ python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db Before and after must match exactly. +## Restore photos (media) + +Recipe and dinner photos live in `media/` in the data volume and are +backed up nightly to `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/` +(see "Media — nightly rsync"). The backup service mirrors with +`rsync --delete`. **An `rsync --delete` run must never see an empty or +partial `media/`**: it would delete the NAS copy you are about to restore +from. Stopping the timer is not enough — it does not stop a run that is +already in flight, and the service can still be started by hand. So stop +**both** the timer and the service, confirm no rsync is running, and do +not start the service until the restore is verified. + +Run from `/home/overseer/mealquest`: + +```sh +M=/var/lib/docker/volumes/mealquest-data/_data/media +NAS=/volume1/homes/Joshua/mealquest-backups/media +SSH="ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" + +# 1. Stop the nightly mirror: the timer AND any run in flight. +sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service + +# 2. Pre-check — all three must hold before media/ is touched. If any +# fails, stop here and find out why. +systemctl is-active mealquest-media-backup.timer # inactive +systemctl is-active mealquest-media-backup.service # inactive (or failed) +pgrep -a -u mealquest rsync || echo "no rsync" # no rsync +sudo $SSH joshua@10.60.1.50 "find $NAS -type f | wc -l" +# ^ note this NAS file count; it must be non-zero and is the target below. + +# 3. Stop the app (no writes into media/ meanwhile). +docker compose -f deploy/compose.yaml stop mealquest + +# 4. Recreate the directory if it is gone, owned by the app user. +sudo install -d -o 10001 -g 10001 -m 0755 "$M" + +# 5. Copy back from the NAS as uid 10001's files. No --delete in this +# direction: the NAS copy is the source and is only read. +sudo rsync -a --stats --chown=10001:10001 -e "$SSH" "joshua@10.60.1.50:$NAS/" "$M/" + +# 6. Post-condition — before anything can mirror again: +sudo find "$M" -type f | wc -l # >= the NAS count from step 2 +sudo $SSH joshua@10.60.1.50 "find $NAS -type f | wc -l" # unchanged from step 2 + +# 7. Start the app, then the timer again. Only now may the service run. +docker compose -f deploy/compose.yaml up -d +curl -fsS http://10.60.1.14:8090/health +sudo systemctl start mealquest-media-backup.timer +systemctl is-active mealquest-media-backup.timer # active +``` + +The local count may exceed the NAS count only if `media/` still held +files the NAS never received (photos added since the last nightly run); +the next nightly run then copies them up. If the local count is lower, +the copy did not complete: leave the timer stopped and re-run step 5. + +When the database is restored too, restore it first ("Restore the +database"), then the media, so every photo the database refers to is +present. This media restore has not been run against production yet; the +backup direction was checked on 2026-10-05 (see "Media backup checks"). + ## Restore drill record — 2026-10-05 Drill on 10.60.1.14 against the live (pre-launch, empty-household) @@ -316,3 +701,141 @@ integrity ok. | `media` replaced by a symlink to `/etc` | `Result=exit-code`, status 1; `/run/mealquest-data/media is a symlink to /etc; refusing to back it up` | | `media` removed | `Result=exit-code`, status 1; `media directory /run/mealquest-data/media is missing` | | `sudo -u mealquest` writing into the volume via `/var/lib/docker/...` | `Permission denied` — uid 10001 cannot traverse `/var/lib/docker` (mode 0710 root) | + +## Secret rotation + +Rotate a secret when it may have leaked, when someone with access leaves, +or on the owner's schedule. Never paste a secret into a ticket, chat or +this file. + +### App key (`deploy/secrets/secret_key`) + +The app key encrypts every stored secret: the `_ct` columns +`ai_settings.api_key_ct` (LLM API key), `recipe_search_settings.api_key_ct` +(recipe-search API key), `integrations_google.refresh_token_ct` (Google +Calendar refresh token) and `chat_settings.webhook_url_ct` (Google Chat +webhook URL). **Rotating it re-encrypts nothing**: a value encrypted with +the old key can no longer be read. A `mealquest rotate-key` helper that +re-encrypts in place is planned but does not exist yet. + +**What is recoverable today.** The service currently exposes only +household settings; no shipped flow writes or re-enters any of these four +secrets. The re-entry flows are planned: the LLM key in Settings → AI +assistant (the AI settings feature), the recipe-search key (Feature 10), +Google Calendar reconnect (Feature 16) and the Google Chat webhook +(Feature 19). So today the rotation is safe **only if no `_ct` value is +stored**, which the pre-check below proves. If the pre-check finds a +stored value, do not rotate: it could not be re-entered afterwards. Keep +the current key and wait for the matching flow (or the `rotate-key` +helper). + +**The pre-check and the key replacement run with the whole stack +stopped**, from the pre-check until the new key is in place. A raw copy of +`mealquest.db` and `mealquest.db-wal` taken while the stack runs is **not** +a consistent snapshot: the two files are copied one after the other, and a +checkpoint (by the app or by Litestream) between the two copies can move +committed pages out of the WAL into `mealquest.db` after `mealquest.db` was +already copied, then reset the WAL before it is copied. The copy then +lacks those commits — possibly a stored `_ct` value — and the pre-check +reports a false 0. A running app could also store a `_ct` value after a +correct pre-check and before the key is replaced. Stopping both containers +removes every writer (Litestream also writes and checkpoints), and the +clean close checkpoints the WAL into `mealquest.db`, so the file alone is +complete. + +Run from `/home/overseer/mealquest`: + +```sh +D=/var/lib/docker/volumes/mealquest-data/_data + +# 1. Stop the whole stack (app and Litestream) and leave it stopped until +# step 4. Nothing may hold the database open. +docker compose -f deploy/compose.yaml down +sudo ls -la "$D" # mealquest.db and media/ only: no -wal, no -shm; if + # either exists, the close was not clean — stop here + +# 2. Pre-check, on a copy of the now-quiescent database. +rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck +sudo cp -a "$D/mealquest.db" /tmp/mqcheck/ && sudo chown -R "$USER" /tmp/mqcheck +python3 - /tmp/mqcheck/mealquest.db <<'PY' +import sqlite3, sys +c = sqlite3.connect(sys.argv[1]) +for t, col in (("ai_settings", "api_key_ct"), ("recipe_search_settings", "api_key_ct"), + ("integrations_google", "refresh_token_ct"), ("chat_settings", "webhook_url_ct")): + print(t, col, c.execute(f"select count(*) from {t} where {col} is not null").fetchone()[0]) +PY +rm -rf /tmp/mqcheck +# every count must be 0; otherwise start the stack again on the current +# key (docker compose -f deploy/compose.yaml up -d) and stop here + +# 3. Replace the key, still with the stack stopped. +sudo cp -a deploy/secrets/secret_key /var/tmp/secret_key.old # keep until verified, then shred +head -c 32 /dev/urandom | base64 | sudo tee deploy/secrets/secret_key >/dev/null +sudo chown 10001:10001 deploy/secrets/secret_key +sudo chmod 600 deploy/secrets/secret_key + +# 4. Start the stack on the new key and check health. +docker compose -f deploy/compose.yaml up -d +docker compose -f deploy/compose.yaml ps +curl -fsS http://10.60.1.14:8090/health +``` + +Then sign in and open household settings to confirm the service runs on +the new key. Finally `sudo shred -u /var/tmp/secret_key.old`. Once the +planned flows ship, this section gains a step per secret to re-enter it +and confirm it works. + +### OIDC client secret (`deploy/secrets/oidc_client_secret`) + +In Authentik, open the provider `mealquest` and regenerate the Client +Secret. Write it to the file (same owner and mode as above) and recreate +the service: + +```sh +printf '%s\n' '' | sudo tee deploy/secrets/oidc_client_secret >/dev/null +sudo chown 10001:10001 deploy/secrets/oidc_client_secret +sudo chmod 600 deploy/secrets/oidc_client_secret +docker compose -f deploy/compose.yaml up -d --force-recreate mealquest +``` + +Sign in at to confirm; sign-ins +fail between regenerating the secret in Authentik and recreating the +service. + +### LLM API key + +Stored encrypted in the database, not in `deploy/`. In the app, open +**Settings → AI assistant**, choose **Replace**, paste the new key and +**Verify**, and check the verified state is shown. Then revoke the +old key in the provider's console. (The AI assistant settings page is +delivered by its own feature; until it ships there is no stored LLM key +to rotate.) + +### VAPID keys (Web Push) + +Planned with Web Push (Feature 19): the public key goes in +`deploy/.env` as `MEALQUEST_VAPID_PUBLIC_KEY`, the private key in +`deploy/secrets/vapid_private_key` (owner `10001:10001`, mode 0600), and +the pair is generated with `cmd/vapidgen`. **Changing the VAPID pair +invalidates every push subscription**: each device must turn +notifications on again (`/settings/notifications`) before it receives +pushes. Rotate only when the private key may have leaked, write both +halves, recreate the service with `--force-recreate mealquest`, and tell +the household to re-enable notifications on each device. + +### Litestream / NAS key (`deploy/secrets/litestream_key`) + +Generate a new keypair as in "Database — Litestream", add the new public +key to `joshua@10.60.1.50:~/.ssh/authorized_keys`, replace the file +(owner `10001:10001`, mode 0600), run +`docker compose -f deploy/compose.yaml up -d --force-recreate litestream` +and `sudo systemctl start mealquest-media-backup.service`, check both +(`litestream ltx` and the journal), then remove the old public key from +the NAS. + +## Context records + +The confirmed values in this runbook (port 8090, checkout path, Authentik +group `mealquest-admin`, DNS, firewall, backup target) are also recorded in +the CTXD project `mealquest`: `RUNBOOKS.MD` (deployment procedure) and the +open questions in `CONTEXT.MD`, updated for Feature 07 task 876134bc.