From 1a781bfdb8e4b103153d5fb73980be6c8db461f9 Mon Sep 17 00:00:00 2001 From: orchestrator-agent <20+orchestrator-agent@code.cubecraftcreations.com> Date: Thu, 8 Oct 2026 10:02:40 -0400 Subject: [PATCH] Move the deployment runbook out of the repository into CTXD (#137) Co-authored-by: orchestrator-agent <20+orchestrator-agent@code.cubecraftcreations.com> --- README.md | 64 +- deploy/.env.example | 41 +- deploy/RUNBOOK.md | 1417 ----------------- deploy/compose.dev.yaml | 9 +- deploy/compose.yaml | 22 +- deploy/systemd/mealquest-firewall.service | 5 +- deploy/systemd/mealquest-media-backup.service | 7 +- deploy/systemd/mealquest-media-backup.sh | 5 +- 8 files changed, 50 insertions(+), 1520 deletions(-) delete mode 100644 deploy/RUNBOOK.md diff --git a/README.md b/README.md index 97d7c82..1f60a80 100644 --- a/README.md +++ b/README.md @@ -11,61 +11,14 @@ single Go service with the React PWA embedded in the binary. > `/health`, embedded PWA skeleton, Makefile, Dockerfile, CI). The planned > application is described in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). -## Quick start +## Running it locally -Requirements: Go 1.25 and Node.js `^22.22.2 || ^24.15.0 || >=26.0.0` — that -is, 22.22.2 or later on the 22 line, 24.15.0 or later on the 24 line, or 26.0.0 -or later. Node.js 23, 24.0–24.14 and 25 are not supported: this is the -`engines` range of jsdom in the committed lockfile, the strictest of all the -web dependencies. - -```sh -(cd web && npm ci) -mkdir -p ~/.mealquest-dev # outside the repository: it holds a secret -chmod 700 ~/.mealquest-dev -head -c 32 /dev/urandom | base64 > ~/.mealquest-dev/secret_key -chmod 600 ~/.mealquest-dev/secret_key -export MEALQUEST_HTTP_ADDR=:8080 -export MEALQUEST_PUBLIC_URL=http://localhost:5173 -export MEALQUEST_DATA_DIR="$HOME/.mealquest-dev" -export MEALQUEST_SHUTDOWN_TIMEOUT=10s -export MEALQUEST_SECRET_KEY_FILE="$HOME/.mealquest-dev/secret_key" -# Authentik OAuth2/OpenID provider for MealQuest (redirect URI: -# $MEALQUEST_PUBLIC_URL/auth/callback). Save its client secret to the file. -export MEALQUEST_OIDC_ISSUER=https://auth.example.com/application/o/mealquest/ -export MEALQUEST_OIDC_CLIENT_ID=mealquest -export MEALQUEST_OIDC_CLIENT_SECRET_FILE="$HOME/.mealquest-dev/oidc_client_secret" -export MEALQUEST_OIDC_ALLOWED_GROUPS=mealquest-admin -# Optional: sessions end after MEALQUEST_SESSION_IDLE unused (default 720h) -# and always MEALQUEST_SESSION_MAX after sign-in (default 4320h). -# export MEALQUEST_SESSION_IDLE=720h -# export MEALQUEST_SESSION_MAX=4320h -# Optional: comma-separated IPs/CIDRs of reverse proxies whose -# X-Forwarded-For/-Proto headers are honoured (production: Traefik). Unset -# trusts no peer, which is right for local development. -# export MEALQUEST_TRUSTED_PROXY=10.60.1.33 -# Optional: a SearXNG instance the Settings screen suggests as the recipe -# search URL until one is saved. -# export MEALQUEST_SEARXNG_URL=http://localhost:8888 -export MEALQUEST_DEFAULT_TZ=America/New_York -make dev -``` - -`make dev` runs the Go server on `:8080` and the Vite dev server on `:5173` -(which proxies `/api`, `/auth` and `/media` to `:8080`). Open -http://localhost:5173. Configuration is environment-only and fails fast: -every variable above is required except the commented-out session -lifetimes (when set they must be positive, with MAX at least IDLE), -trusted proxy list (every entry must be an IP or CIDR), the SearXNG URL -(an absolute `http://` or `https://` URL without credentials, a query or a -fragment) and `MEALQUEST_DEFAULT_TZ`, and -secrets are read only from files named by `*_FILE` variables. `MEALQUEST_PUBLIC_URL`, `MEALQUEST_OIDC_ISSUER` and the -endpoints the issuer advertises must be `https://`; plain `http://` is -accepted only for `localhost`, `127.0.0.1` and `[::1]`. -`MEALQUEST_DEFAULT_TZ` is the IANA zone the household settings are created -with on first boot: until those settings exist, boot fails unless it is a -valid IANA zone. Later boots ignore it, set or not (change the zone in -household settings instead). +The developer setup (prerequisites, Authentik and Google clients, the +`MEALQUEST_*` environment, `make dev` and the optional SearXNG sidecar) is +runbook information and lives in CTXD: project `mealquest`, file +`RUNBOOKS.MD`, section "Local development". Configuration is read only from +the environment, and the service refuses to start, naming every missing or +invalid variable. ## Common commands @@ -78,7 +31,8 @@ household settings instead). ## Where to read next -- [AGENTS.md](AGENTS.md) — house rules, approved dependencies, conventions. +- [AGENTS.md](AGENTS.md) — house rules, approved dependencies, the quality + floor every change must pass, repository layout, conventions. - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — architecture of record. - [design/MealQuest Requirements.md](design/MealQuest%20Requirements.md) — product requirements. - [design/MealQuest UI Designs/design_handoff_mealquest/README.md](design/MealQuest%20UI%20Designs/design_handoff_mealquest/README.md) — UI handoff. diff --git a/deploy/.env.example b/deploy/.env.example index 6262452..fc2b906 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -1,20 +1,15 @@ -# MealQuest production environment. Copy to deploy/.env (never committed) -# and replace the placeholders. The service fails fast: a missing or invalid -# required variable stops it at boot with the variable named. +# MealQuest production environment: the template for deploy/.env (never +# committed). The service fails fast: a missing or invalid required variable +# stops it at boot with the variable named. Creating deploy/.env: CTXD +# project mealquest, RUNBOOKS.MD "Stack and first deploy" > "3. Environment +# file". # # Secrets are never set here. They live in deploy/secrets/ (never committed), -# one value per file, owned by the container user and readable only by it: -# -# mkdir -p deploy/secrets && chmod 700 deploy/secrets -# head -c 32 /dev/urandom | base64 > deploy/secrets/secret_key -# printf '%s\n' '' > deploy/secrets/oidc_client_secret -# printf '%s\n' '' > deploy/secrets/google_client_secret -# sudo chown 10001:10001 deploy/secrets/* && sudo chmod 600 deploy/secrets/* -# -# SearXNG's secret key is the exception: compose reads it itself, so it -# stays owned by you (mode 0600). Create it after the chown above: -# -# (umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env) +# one value per file, owned by the container user 10001:10001 with mode +# 0600: secret_key, oidc_client_secret and google_client_secret. SearXNG's +# searxng.env is the exception: compose reads it itself, so it is owned by +# the host account that runs compose, mode 0600. Creating them: CTXD project +# mealquest, RUNBOOKS.MD "Stack and first deploy" > "4. Secrets". # # deploy/compose.yaml mounts them at /run/secrets/ and points the # *_FILE variables at them. @@ -42,8 +37,7 @@ MEALQUEST_TRUSTED_PROXY=10.60.1.33 # IANA zone the household settings are created with on first boot. Required # until those settings exist; ignored afterwards (change the zone in -# household settings instead). Confirm the household's zone before the first -# deploy. +# household settings instead). MEALQUEST_DEFAULT_TZ=America/New_York # --- Authentik OIDC ---------------------------------------------------------- @@ -57,10 +51,10 @@ MEALQUEST_OIDC_ALLOWED_GROUPS=mealquest-admin # --- Google Calendar (Push to Calendar) ------------------------------------ -# Client ID of the Google OAuth web client (deploy/RUNBOOK.md "Google -# Calendar OAuth client"). Required: the service will not start without it -# and the client secret file. Its authorized redirect URI is -# /api/v1/integrations/google/callback. +# Client ID of the Google OAuth web client (CTXD project mealquest, +# RUNBOOKS.MD "Google Calendar OAuth client"). Required: the service will +# not start without it and the client secret file. Its authorized redirect +# URI is /api/v1/integrations/google/callback. MEALQUEST_GOOGLE_CLIENT_ID=.apps.googleusercontent.com # Optional session lifetimes. A session ends after IDLE unused and always MAX @@ -82,13 +76,14 @@ MEALQUEST_SESSION_MAX=4320h # The SearXNG recipe-search sidecar on the compose network. The Settings # screen pre-fills it as the SearXNG URL until one is saved; the saved # provider and base URL come from PATCH /api/v1/settings/recipe-search. -# See deploy/RUNBOOK.md "Recipe search (SearXNG)". +# See CTXD project mealquest, RUNBOOKS.MD "Recipe search (SearXNG)". # --- Planned; not read by the service yet ------------------------------------- # # Added to deploy/compose.yaml and this file by the features that use them. # -# Web Push (Feature 19). Generate the key pair with cmd/vapidgen. +# Web Push (Feature 19). Key pair: CTXD project mealquest, RUNBOOKS.MD +# "VAPID keys (Web Push)". # MEALQUEST_VAPID_PUBLIC_KEY= # MEALQUEST_VAPID_PRIVATE_KEY_FILE=/run/secrets/vapid_private_key # MEALQUEST_VAPID_SUBJECT=mailto: diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md deleted file mode 100644 index 61137ae..0000000 --- a/deploy/RUNBOOK.md +++ /dev/null @@ -1,1417 +0,0 @@ -# MealQuest deployment runbook - -How MealQuest is deployed on office-srvr (10.60.1.14), how it is reached -through Authentik, Traefik and DNS, how it is updated and rolled back, how -its data is backed up and restored, and how its secrets are rotated. Values -are the ones in use on 2026-10-05. Secrets themselves are never written -here. - -Paths used below: - -| Name | Path | -| --- | --- | -| Checkout on the host | `/home/overseer/mealquest` | -| Data volume (`mealquest-data`) on the host | `/var/lib/docker/volumes/mealquest-data/_data` | -| Same volume inside the containers | `/data` | -| Database | `mealquest.db` (+ `mealquest.db-wal`, `mealquest.db-shm` while open) | -| Media | `media/` in the data volume | -| Backup target (NAS, over SSH) | `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups` | -| Same target over SFTP (Synology chroot) | `/homes/Joshua/mealquest-backups` | - -## Prerequisites - -On office-srvr (10.60.1.14), as recorded on 2026-10-05: - -| Item | Value | -| --- | --- | -| OS | Ubuntu 26.04.1 LTS | -| Docker Engine / Compose plugin | 29.8.2 / 5.6.0 | -| git, rsync, iptables | 2.53.0, 3.4.1, 1.8.11 (nf_tables) | -| Account | `overseer`, with sudo | -| Network | LAN reachability to Traefik (10.60.1.33), Authentik (`auth.cubecraftlabs.com`) and the NAS (10.60.1.50, SSH) | - -Elsewhere: - -- Admin access to Authentik at . -- Owner access to the Google Cloud project that holds MealQuest's OAuth - client (see "Google Calendar OAuth client"). -- SSH access to the Traefik host 10.60.1.33. -- Cloudflare access for the `cubecraftlabs.com` zone, and admin access to - both Pi-holes (10.60.1.101 `dns.ccl`, 10.60.1.85 `dns2.ccl`). -- A NAS account `joshua` on 10.60.1.50 with the backup directory - `/volume1/homes/Joshua/mealquest-backups` (see "Backups"). -- A Gitea token able to read `Maniacs_Mansion/MealQuest`, for the clone. - -## Stack and first deploy - -The compose stack is `deploy/compose.yaml`: the `mealquest` service (built -from the repository `Dockerfile`, runs as uid/gid 10001, listens on 8080 in -the container), the `litestream` service and the `searxng` recipe-search -sidecar (see "Recipe search (SearXNG)"). Data lives in the named volume -`mealquest-data`. - -### 1. Choose and check the port - -The container's 8080 is published on `10.60.1.14:8090`. Ports 8080 and 8081 -on this host belong to NestQuest, so 8090 was chosen. Before the first -deploy it was checked free with: - -```sh -ss -ltnp | grep 8090 # printed nothing -``` - -After the deploy the same host shows NestQuest on 8080/8081 and MealQuest -on 8090: - -``` -LISTEN 0 4096 10.60.1.14:8080 0.0.0.0:* -LISTEN 0 4096 10.60.1.14:8081 0.0.0.0:* -LISTEN 0 4096 10.60.1.14:8090 0.0.0.0:* -``` - -The published host port is fixed in `deploy/compose.yaml`, in -`deploy/systemd/mealquest-firewall.sh` and in the Traefik route; change all -three together. - -### 2. Clone - -The checkout lives at `/home/overseer/mealquest`, next to NestQuest's -`~/nestquest` (not `/opt/mealquest`). Clone over HTTPS with a Gitea token, -then reset the remote to the clean URL so the token is not stored in -`.git/config`: - -```sh -cd /home/overseer -git clone https://:@code.cubecraftlabs.com/Maniacs_Mansion/MealQuest.git mealquest -cd mealquest -git remote set-url origin https://code.cubecraftlabs.com/Maniacs_Mansion/MealQuest.git -git remote -v # must show no token -git checkout main -``` - -### 3. Environment file - -`deploy/.env` (git-ignored) is copied from `deploy/.env.example` and is -mode 0600, owned by `overseer`: - -```sh -cp deploy/.env.example deploy/.env -chmod 600 deploy/.env -``` - -Then edit `deploy/.env` to the values below. In particular -**`MEALQUEST_BIND_ADDRESS` must be changed**: `.env.example` ships -`127.0.0.1` (a dev machine), and left as is the service listens only on -the host's loopback, which Traefik cannot reach. Set it to the LAN address -and check: - -```sh -sed -i 's/^MEALQUEST_BIND_ADDRESS=.*/MEALQUEST_BIND_ADDRESS=10.60.1.14/' deploy/.env -grep '^MEALQUEST_BIND_ADDRESS=' deploy/.env # MEALQUEST_BIND_ADDRESS=10.60.1.14 -docker compose -f deploy/compose.yaml config | grep -A4 'ports:' -# host_ip: 10.60.1.14, target: 8080, published: "8090" -``` - -After step 5 the listener is `10.60.1.14:8090` (container port 8080): - -```sh -docker compose -f deploy/compose.yaml port mealquest 8080 # 10.60.1.14:8090 -sudo ss -ltnp 'sport = :8090' # 10.60.1.14:8090 only, no 0.0.0.0 or 127.0.0.1 -``` - -Values in use: - -| Variable | Value | -| --- | --- | -| `MEALQUEST_BIND_ADDRESS` | `10.60.1.14` | -| `MEALQUEST_PUBLIC_URL` | `https://mealquest.cubecraftlabs.com` | -| `MEALQUEST_SHUTDOWN_TIMEOUT` | `10s` | -| `MEALQUEST_TRUSTED_PROXY` | `10.60.1.33` | -| `MEALQUEST_DEFAULT_TZ` | `America/New_York` | -| `MEALQUEST_OIDC_ISSUER` | `https://auth.cubecraftlabs.com/application/o/mealquest/` | -| `MEALQUEST_OIDC_CLIENT_ID` | `i6zraqmWsF552b8nmte070LXz0SI5pRdArBxFxeq` (not a secret) | -| `MEALQUEST_OIDC_ALLOWED_GROUPS` | `mealquest-admin` | -| `MEALQUEST_GOOGLE_CLIENT_ID` | Client ID of the Google OAuth web client (not a secret; see "Google Calendar OAuth client") | -| `MEALQUEST_SESSION_IDLE` / `MEALQUEST_SESSION_MAX` | `720h` / `4320h` | - -`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR` and the `*_FILE` variables are -set by `deploy/compose.yaml` and must not appear in `deploy/.env`. - -### 4. Secrets - -`deploy/secrets/` (git-ignored) is mode 0700, owned by `overseer`. Each -file holds one value, is owned by `10001:10001` (host user `mealquest`) and -is mode 0600. Compose bind-mounts file secrets with their host owner and -mode, so any other owner or mode stops the container reading them and the -service refuses to start. - -| File | Content | Mounted at | -| --- | --- | --- | -| `secret_key` | base64 of 32 random bytes; encrypts stored secrets (AES-256-GCM) | `/run/secrets/secret_key` | -| `oidc_client_secret` | Client Secret of the Authentik provider `mealquest` | `/run/secrets/oidc_client_secret` | -| `google_client_secret` | Client secret of the Google OAuth web client | `/run/secrets/google_client_secret` | -| `litestream_key` | dedicated SSH private key for the NAS (see "Backups") | `/etc/litestream/sftp_key` | -| `searxng.env` | `SEARXNG_SECRET=<64 hex characters>`; SearXNG's secret key | not mounted; compose passes it as an environment variable | - -`searxng.env` is the one exception to the owner rule: compose itself reads -it (as `env_file`), so it is owned by `overseer`, mode 0600. - -```sh -mkdir -p deploy/secrets && chmod 700 deploy/secrets -head -c 32 /dev/urandom | base64 > deploy/secrets/secret_key -printf '%s\n' '' > deploy/secrets/oidc_client_secret -printf '%s\n' '' > deploy/secrets/google_client_secret -# litestream_key: see "Backups" > "Database — Litestream" -sudo chown 10001:10001 deploy/secrets/secret_key deploy/secrets/oidc_client_secret deploy/secrets/google_client_secret deploy/secrets/litestream_key -sudo chmod 600 deploy/secrets/* -(umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env) -sudo ls -la deploy/secrets # four files mealquest mealquest, searxng.env overseer overseer; all -rw------- -``` - -The `mealquest` host user (uid/gid 10001) is created in "Media — nightly -rsync"; until it exists `ls` shows the numeric ids. - -### 5. Firewall, then the stack - -Install the firewall unit and the stack unit first (see "Host firewall" -and "Stack unit"), so port 8090 is never open to the LAN. Create the media -directory (see "Media — nightly rsync"), then build and start: - -```sh -docker compose -f deploy/compose.yaml build -sudo systemctl enable --now mealquest # starts mealquest-firewall first -docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up, searxng healthy -curl -fsS http://10.60.1.14:8090/health -curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 -``` - -### Host checkout state on 2026-10-05 - -The first deploy ran before Feature 07 reached `main`. The checkout is at -`main` `9cddbbb` and the Feature 07 files under `deploy/` were copied in -from the feature branch: `deploy/compose.yaml` and `deploy/.env.example` -are listed in `.git/info/exclude`, the rest show as untracked. Before the -first routine deploy of a `main` that contains `deploy/`, remove those -`.git/info/exclude` lines and the copied files (keep `deploy/.env` and -`deploy/secrets/`), or `git checkout` refuses to overwrite them: - -```sh -cd /home/overseer/mealquest -git status --porcelain --untracked-files=all deploy # the copied files -sed -i '/^deploy\/\(compose.yaml\|.env.example\)$/d' .git/info/exclude -git clean -n deploy # dry run: must not list .env or secrets/ (both git-ignored) -git clean -f deploy -``` - -On 2026-10-06 `deploy/compose.yaml` was replaced and -`deploy/compose.dev.yaml` and `deploy/searxng/settings.yml` were copied in -from the Feature 10 branch (task 641f468b) to start the SearXNG sidecar; -they too must go before that checkout (`git clean -n -d deploy` lists -them). - -Then follow "Routine deploy"; the checkout brings the tracked copies back. - -## Authentik - -Sign-in is OpenID Connect against Authentik at -. - -| Object | Setting | -| --- | --- | -| Provider | OAuth2/OpenID provider `mealquest`, client type **Confidential** | -| Redirect URI | `https://mealquest.cubecraftlabs.com/auth/callback` (strict) | -| Signing | RS256, with the instance's signing key | -| Scopes | MealQuest requests `openid profile email groups offline_access`; the provider's scope mappings include openid, profile, email and offline_access, and the `groups` claim is delivered by a property mapping (the default profile mapping carries `groups`) | -| Application | name MealQuest, slug `mealquest`, provider `mealquest` | -| Group | `mealquest-admin`, bound to the application; household members are added to it | -| Client ID | `i6zraqmWsF552b8nmte070LXz0SI5pRdArBxFxeq` (not a secret; in `deploy/.env` as `MEALQUEST_OIDC_CLIENT_ID`) | -| Client Secret | in `deploy/secrets/oidc_client_secret` only | - -MealQuest refuses a sign-in unless the user's `groups` claim (from the ID -token, or from userinfo when the token has none) contains a group in -`MEALQUEST_OIDC_ALLOWED_GROUPS`. - -Check discovery from the Docker host — the service discovers the issuer -at boot and will not start if it cannot: - -```sh -curl -s -o /dev/null -w "%{http_code}\n" \ - https://auth.cubecraftlabs.com/application/o/mealquest/.well-known/openid-configuration # 200 -``` - -On 2026-10-05 discovery returned 200 with issuer -`https://auth.cubecraftlabs.com/application/o/mealquest/` and -`id_token_signing_alg_values_supported: ["RS256"]`. - -## Google Calendar OAuth client - -Push to Calendar connects the household's Google account with a Google -OAuth web client. MealQuest calls Google's OAuth and Calendar REST -endpoints directly. **The service will not start without -`MEALQUEST_GOOGLE_CLIENT_ID` and `deploy/secrets/google_client_secret`**, -so create the client before deploying a release that contains Feature 16. - -In the Google Cloud console, in the project that holds MealQuest: - -| Object | Setting | -| --- | --- | -| API | Google Calendar API enabled | -| OAuth consent screen | User type External; app name MealQuest; publishing status **In production** (a client left in Testing gets refresh tokens that expire after 7 days, so Push to Calendar would stop working a week after connecting) | -| Scopes | `https://www.googleapis.com/auth/calendar.events` and `https://www.googleapis.com/auth/calendar.calendarlist.readonly` | -| Credentials | OAuth client ID, application type **Web application**, name `mealquest` | -| Authorized redirect URI | `https://mealquest.cubecraftlabs.com/api/v1/integrations/google/callback` (exactly; it is `MEALQUEST_PUBLIC_URL`'s origin plus that path) | -| Client ID | in `deploy/.env` as `MEALQUEST_GOOGLE_CLIENT_ID` (not a secret) | -| Client secret | in `deploy/secrets/google_client_secret` only | - -The app is unverified, so Google shows an "unverified app" warning on the -consent screen; choose to continue. That is expected for a single-household -deployment. - -To connect, a signed-in household member opens -`https://mealquest.cubecraftlabs.com/api/v1/integrations/google/connect` -(the Settings screen links there once it ships). MealQuest asks for offline -access with consent always prompted, and lands back on -`/settings/calendar` once the refresh token is stored. Disconnecting -(`DELETE /api/v1/integrations/google`) revokes the token at Google and -forgets it. - -## Reverse proxy (Traefik) - -MealQuest is published at by the -shared Traefik instance on 10.60.1.33 (Feature 07, task 0c1dc172). The -route is a dynamic-configuration file that lives on the Traefik host only, -at `/home/overseer/Code/traefik/config/dynamic/mealquest-ccl.yml`. Its exact -contents: - -```yaml -# MealQuest - https://mealquest.cubecraftlabs.com -# Feature 07 task 0c1dc172. Single Go binary (API + embedded PWA) on -# office-srvr 10.60.1.14:8090. HTTP -> HTTPS is handled globally by -# http-conf.yml, so only a websecure router is defined here. -# This file lives on the Traefik host only; the repo copy is recorded in -# deploy/RUNBOOK.md by task 876134bc. -http: - routers: - mealquest: - entryPoints: - - websecure - rule: "Host(`mealquest.cubecraftlabs.com`)" - priority: 100 - middlewares: - - mealquest-headers - service: mealquest - tls: - certResolver: myresolver - domains: - - main: mealquest.cubecraftlabs.com - - services: - mealquest: - loadBalancer: - servers: - - url: "http://10.60.1.14:8090" - - middlewares: - mealquest-headers: - headers: - stsSeconds: 31536000 - stsIncludeSubdomains: true - contentTypeNosniff: true - frameDeny: true -``` - -Traefik watches the dynamic directory, so saving the file loads the route; -the certificate comes from `certResolver: myresolver` (Let's Encrypt, -DNS-01 through Cloudflare). Check from the LAN: - -```sh -curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 -``` - -## DNS - -| Where | Record | -| --- | --- | -| Cloudflare, zone `cubecraftlabs.com` | `CNAME mealquest.cubecraftlabs.com` → `cubecraftlabs.com`, **proxied = true** (orange cloud). Because it is proxied, public resolvers answer with Cloudflare's anycast addresses, A and AAAA (AAAA on 2026-10-05: `2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75`), never the origin | -| Pi-hole primary 10.60.1.101 (`dns.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | -| Pi-hole secondary 10.60.1.85 (`dns2.ccl`), Local DNS records | `A mealquest.cubecraftlabs.com` → `10.60.1.33` (Traefik) | - -The local record must be on **both** Pi-holes, or clients using the -secondary go out through Cloudflare. Check: - -```sh -for s in 10.60.1.101 10.60.1.85; do dig +short @$s mealquest.cubecraftlabs.com A; done -# 10.60.1.33 from each -``` - -**AAAA caveat.** The Pi-hole local record is A only, so both Pi-holes still -forward AAAA queries and return Cloudflare's IPv6 addresses -(`2606:4700:3032::6815:34ed`, `2606:4700:3035::ac43:cd75` on 2026-10-05). An -IPv6-capable LAN client may prefer those and reach MealQuest through -Cloudflare instead of directly through Traefik. It still works (Cloudflare -proxies to the public route), but it is not the LAN path. If that matters, -add a local AAAA override or block AAAA for this name on both Pi-holes. - -## Host firewall - -Docker publishes `10.60.1.14:8090` past the host's INPUT rules, so the -port is restricted in the `DOCKER-USER` chain instead: only Traefik -(10.60.1.33) and the host itself (10.60.1.14) may connect; every other -source is dropped. - -Docker DNATs the published port to the container address and port 8080 -**before** `DOCKER-USER`, so a plain `-p tcp --dport 8090` rule never -matches. The rules match the original destination `10.60.1.14:8090` with -conntrack (`--ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL`) -and are tagged with the comment `mealquest-firewall`. The port is never -open while the rules change: if the chain does not already start with -exactly the three rules, the script first inserts the DROP at position 1, -then deletes its other tagged rules and inserts the two RETURN rules above -the DROP. A re-run on a correct chain changes nothing. NestQuest's -8080/8081 rules below them are left untouched. - -| Repository file | Installed as | -| --- | --- | -| `deploy/systemd/mealquest-firewall.sh` | `/usr/local/sbin/mealquest-firewall` (root, 0755) | -| `deploy/systemd/mealquest-firewall.service` | `/etc/systemd/system/mealquest-firewall.service` (root, 0644) | - -The unit is `Type=oneshot`, `RemainAfterExit=yes`, `Requires=` and -`PartOf=docker.service`, and `WantedBy=multi-user.target docker.service`: -it re-applies the rules whenever Docker starts or restarts (Docker -recreates `DOCKER-USER` empty), and a failure fails this unit only. - -Install (done on 10.60.1.14 on 2026-10-05): - -```sh -cd /home/overseer/mealquest -sudo install -o root -g root -m 0755 deploy/systemd/mealquest-firewall.sh /usr/local/sbin/mealquest-firewall -sudo install -o root -g root -m 0644 deploy/systemd/mealquest-firewall.service /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now mealquest-firewall -``` - -Check: - -```sh -systemctl is-enabled mealquest-firewall; systemctl is-active mealquest-firewall # enabled, active -sudo iptables -S DOCKER-USER -``` - -Expected head of the chain (recorded 2026-10-05; NestQuest's rules follow): - -``` --A DOCKER-USER -s 10.60.1.33/32 -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j RETURN --A DOCKER-USER -s 10.60.1.14/32 -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j RETURN --A DOCKER-USER -m conntrack --ctorigdst 10.60.1.14 --ctorigdstport 8090 --ctdir ORIGINAL -m comment --comment mealquest-firewall -j DROP -``` - -From any other LAN machine, `curl -m 5 http://10.60.1.14:8090/health` must -time out; from 10.60.1.33 and 10.60.1.14 it returns 200. - -## Stack unit - -`deploy/systemd/mealquest.service` owns the compose stack and is the only -way it starts. Both compose services use `restart: "no"`, so Docker never -starts them on its own; the unit has `Requires=` and -`After=mealquest-firewall.service`, so the stack starts only after the -firewall rules are applied, and not at all if that unit fails. It is -`PartOf=docker.service`, so restarting Docker (which recreates -`DOCKER-USER` empty) stops the stack, re-applies the firewall, then starts -the stack again. `ExecStart` is `docker compose … up -d` and `ExecStop` is -`docker compose … down`; neither builds, so build before starting. - -Containers no longer restart after a crash; `systemctl restart mealquest` -brings them back. - -| Repository file | Installed as | -| --- | --- | -| `deploy/systemd/mealquest.service` | `/etc/systemd/system/mealquest.service` (root, 0644) | - -Install (done on 10.60.1.14 on 2026-10-05): - -```sh -cd /home/overseer/mealquest -sudo install -o root -g root -m 0644 deploy/systemd/mealquest.service /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now mealquest -``` - -Check: - -```sh -systemctl is-enabled mealquest; systemctl is-active mealquest # enabled, active -systemctl show mealquest -p Requires -p After # both list mealquest-firewall.service -``` - -Start, stop and restart the whole stack with `sudo systemctl start|stop|restart -mealquest`, never with `docker compose up` or `down`. Per-service compose -commands (`stop mealquest`, `up -d --force-recreate litestream`) are fine -while the unit is active, because the firewall is already in place. - -## Recipe search (SearXNG) - -The `searxng` service in `deploy/compose.yaml` is a private SearXNG -instance for recipe search. On the compose network it is reachable at -`http://searxng:8080`; that is the address to configure in the app. - -Starting the sidecar does not connect the app to it. The recipe-search -provider and base URL live in the `recipe_search_settings` row and are set -through `GET`/`PATCH /api/v1/settings/recipe-search` (the API behind the -recipe search card on Settings → AI assistant). Until that row exists, -recipe search answers 409 `not_configured`. - -`deploy/compose.yaml` sets `MEALQUEST_SEARXNG_URL=http://searxng:8080`. -The service only suggests it: `GET` returns it as `default_base_url`, and -the recipe search card pre-fills the SearXNG URL with it while no URL is -saved. Nothing is stored until the card is saved, and a saved `base_url` -always wins. A set value must be an absolute `http://` or `https://` URL -without credentials, a query or a fragment, or the service refuses to -start; unset, `default_base_url` is `null`. - -To set it up, open Settings → AI assistant, check that the recipe search -card shows SearXNG with `http://searxng:8080`, select **Save recipe search** -and then **Test**. Or, from a signed-in session (every non-`GET` request -needs the `X-MealQuest-CSRF` header set to `csrfToken` from -`GET /api/v1/me`): - -1. `PATCH /api/v1/settings/recipe-search` with - `{"provider": "searxng", "base_url": "http://searxng:8080"}`. -2. `GET /api/v1/settings/recipe-search` shows `"provider": "searxng"` and - `"base_url": "http://searxng:8080"`. -3. `POST /api/v1/settings/recipe-search/test` searches - "chicken tikka masala" and must answer 200 with - `{"query": "chicken tikka masala", "count": }`. A 502 - `search_failed` carries SearXNG's error. - -| Item | Value | -| --- | --- | -| Image | `searxng/searxng:2026.10.4-d48c4b555`, pinned by digest `sha256:76b0bf285aca014c7191fc4d9234c4bfb358624ac33d8883833d496c059ec072` | -| Settings | `deploy/searxng/settings.yml`, mounted read-only at `/etc/searxng/settings.yml` | -| Secret | `deploy/secrets/searxng.env` (`SEARXNG_SECRET`), see "Secrets" | -| Network | `mealquest` only; **no published port** | -| Health | `wget http://127.0.0.1:8080/healthz` inside the container | - -The settings enable the `json` output format, turn off the public limiter -and bot detection (the only client is the `mealquest` service), and keep -only the general web search engines DuckDuckGo, Brave, Google, Bing, -Mojeek and Startpage. Engines that block the host (CAPTCHA, "too many -requests") are suspended by SearXNG for a while and listed under -`unresponsive_engines` in the JSON; the others still answer. - -Check from inside the app container (the only place it is reachable): - -```sh -docker compose -f deploy/compose.yaml exec mealquest \ - wget -qO- 'http://searxng:8080/search?q=chicken+tikka+masala+recipe&format=json' -docker compose -f deploy/compose.yaml port searxng 8080 # prints ":0": nothing published -``` - -On 2026-10-06 that query returned 17 results. From this host four of the -six engines were blocked (Brave, DuckDuckGo, Google and Startpage were -rate-limited or served a CAPTCHA), so the results came from Bing and -Mojeek. SearXNG retries suspended engines after their suspension ends, so -the set that answers can change. From 10.60.1.85 nothing on 10.60.1.14 answered as SearXNG: -8888 refused the connection and 8080 is NestQuest's API. - -For development, `deploy/compose.dev.yaml` publishes SearXNG on the -dev machine's loopback only (`127.0.0.1:8888`). It is never used on -office-srvr. - -To move to a newer image, pull the new tag, take its digest from -`docker image inspect --format '{{json .RepoDigests}}' searxng/searxng:`, -update both in `deploy/compose.yaml`, then deploy as usual. - -## Routine deploy and rollback - -Production runs `main` (a release); only `overseer` merges `dev` → `main`. -Every deploy builds the image on the host from the checked-out commit. - -### Deploy - -1. Check out and build the release. Nothing restarts yet; production - keeps running the previous SHA. - -```sh -cd /home/overseer/mealquest && -git fetch origin && -git rev-parse --short HEAD && # the SHA being replaced: note it for rollback -git checkout --detach origin/main && # or a release tag -docker compose -f deploy/compose.yaml build -``` - - Each command runs only if the one before it succeeded. If `echo $?` - then prints anything but `0`, stop: nothing has restarted. - -2. **Verify a restorable pre-deploy point — before the restart.** The - restart applies any new migrations, and an older SHA refuses the - upgraded schema, so the only rollback is restoring the database to a - point before the restart (see "Rollback to a previous SHA"). Check that - Litestream is replicating and that its replica restores to the current - data, with the commands from "Database — Litestream", "Restore the - database" and "Count check". First write `/tmp/mqcount.py` and - `/tmp/mqsnap.py` with the `cat` command in "Count check", then: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the verification with a nonzero status -D=/var/lib/docker/volumes/mealquest-data/_data -fail() { echo "$*" >&2; exit 1; } -point= -# On the way out, run both clean-ups whatever happens, keep any failure, -# and print the restore point only if every check and clean-up succeeded. -trap 'st=$? -rm -rf /tmp/mqcheck || st=1 -sudo sh -c "rm -f $D/predeploy-check.db*" || st=1 -if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" -else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi -exit "$st"' EXIT -rm -rf /tmp/mqcheck -mkdir /tmp/mqcheck - -# a. Replication is running: litestream up, no errors in its log, and the -# newest LTX file (listed) is from the latest writes. Each command's own -# status is checked: grep exits 0 on a match, 1 on none, 2+ on an error. -id=$(docker compose -f deploy/compose.yaml ps --status running --quiet litestream) || fail "docker compose ps failed" -[ -n "$id" ] || fail "litestream is not running" -docker compose -f deploy/compose.yaml logs --since 24h litestream > /tmp/mqcheck/litestream.log || fail "docker compose logs failed" -gs=0 -grep -i error /tmp/mqcheck/litestream.log > /tmp/mqcheck/litestream.errors || gs=$? -case "$gs" in - 0) cat /tmp/mqcheck/litestream.errors; fail "litestream logged errors in the last 24h";; - 1) ;; # no match: no errors logged - *) fail "the litestream log scan failed (status $gs)";; -esac -docker compose -f deploy/compose.yaml exec -T litestream \ - litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db - -# b. Restore the replica to a scratch file beside the live database (the -# live files are not touched) and move it off the volume; then take a -# consistent snapshot of the live database next to it. -docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ - restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db -sudo sh -c "mv $D/predeploy-check.db* /tmp/mqcheck/" -sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/live.db -sudo chown -R "$USER" /tmp/mqcheck -sudo ls -ln "$D" # mealquest.db-wal and -shm owned by 10001, not 0: see "Count check" - -# c. Count check both: the restored replica must report integrity ok and -# exactly the live snapshot's counts and content-sha256. -python3 /tmp/mqcount.py /tmp/mqcheck/predeploy-check.db > /tmp/mqcheck/restored.txt -python3 /tmp/mqcount.py /tmp/mqcheck/live.db > /tmp/mqcheck/live.txt -cat /tmp/mqcheck/restored.txt -grep -qx 'integrity ok' /tmp/mqcheck/restored.txt || fail "the restored replica failed integrity_check" -diff /tmp/mqcheck/restored.txt /tmp/mqcheck/live.txt || fail "the restored replica differs from the live snapshot" - -# d. The pre-deploy restore point: now, after every check passed, before -# the restart. It is printed on the way out, once the clean-ups succeed. -point=$(date -u +%Y-%m-%dT%H:%M:%SZ) -) -``` - - The verification runs in a subshell that stops at the first failed - step. On the way out it always attempts both clean-ups — removing - `/tmp/mqcheck` and any `predeploy-check.db` left on the volume — and a - failed clean-up fails the verification too. The restore point is - printed last, and only when every check and both clean-ups succeeded. - Read it like this: - - - `pre-deploy restore point:` followed by a UTC time, and `echo $?` - prints `0` — the backup is verified and cleaned up. Note the time for - rollback, with the previous SHA, and go on to step 3. - - `verification FAILED`, any other error, or a nonzero `echo $?` — no - restore point was printed. If only the counts differed, a write - landed between the restore and the snapshot: run the block again. If - a clean-up failed, remove what it names by hand before running the - block again. - - The `ls` in step b shows `mealquest.db-wal` or `-shm` owned by `0` - — stop, as in "Count check", even if the rest passed. - - **If any check fails, do not restart.** Production stays on the - previous SHA (only the new image was built). Restore replication first - — `docker compose -f deploy/compose.yaml up -d --force-recreate - litestream`, then read its log for the cause (NAS reachable, key, - host key) — and repeat this step until it passes. Never deploy a - release on an unverified backup. - -3. Restart and check the deployed release. - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the check with a nonzero status -fail() { echo "$*" >&2; exit 1; } -sha= -# On the way out, print the deployed SHA only if every check passed. -trap 'st=$? -if [ "$st" -eq 0 ] && [ -n "$sha" ]; then echo "deployed SHA: $sha" -else [ "$st" -ne 0 ] || st=1; echo "deploy FAILED (status $st): no SHA is deployed" >&2; fi -exit "$st"' EXIT -sudo systemctl restart mealquest # applies the firewall, then down + up -d -docker compose -f deploy/compose.yaml ps -for s in mealquest litestream searxng; do - id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") || fail "docker compose ps failed" - [ -n "$id" ] || fail "$s is not running" -done -curl -fsS http://10.60.1.14:8090/health -code=$(curl -sS -o /dev/null -w "%{http_code}" https://mealquest.cubecraftlabs.com/health) -[ "$code" = 200 ] || fail "the public /health returned $code, not 200" -sha=$(git rev-parse HEAD) # the checked-out SHA, now running and healthy -) -``` - - The check stops at the first failed step and prints the SHA last, only - when the restart, all three services running and both health checks - succeeded. `restart` returns once `mealquest` is healthy (Litestream - waits for it); SearXNG may still show `health: starting` for its first - 15 seconds. - - - `deployed SHA:` followed by the SHA, and `echo $?` prints `0` — - record that SHA, the previous SHA, the pre-deploy restore point and - the date in the Maestro task for the release. - - `deploy FAILED`, any other error, or a nonzero `echo $?` — record - nothing as deployed: the release is not running healthy, and after a - failed restart the stack may be down. Read - `journalctl -u mealquest` and `docker compose -f deploy/compose.yaml - logs`, then fix the cause or roll back ("Rollback to a previous SHA"). - -Migrations run at start-up; the service refuses to start if the database -has an applied migration it does not know. - -If `deploy/.env.example` gained a variable, add it to `deploy/.env` before -the restart; the service fails fast at boot naming any missing required variable. - -### Rollback to a previous SHA - -First check whether the rollback crosses a migration, with the rule the -service applies at start-up (`internal/store/migrate.go`): the older build -refuses a database holding a version in `schema_migrations` that is not one -of its own migration files — the `*.sql` files directly in `migrations/` -(`migrations/queries/` holds sqlc queries, not migrations). Compare the -versions a consistent snapshot of the live database has applied with the -files the previous SHA has. First write `/tmp/mqsnap.py` with the `cat` -command in "Count check", then: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the check with a nonzero status -D=/var/lib/docker/volumes/mealquest-data/_data -P= -# On the way out, remove /tmp/mqcheck whatever happens, keep any failure, -# and report every nonzero exit. -trap 'st=$? -rm -rf /tmp/mqcheck || st=1 -if [ "$st" -ne 0 ]; then echo "check FAILED (status $st): stop and fix the cause" >&2; fi -exit "$st"' EXIT -rm -rf /tmp/mqcheck -mkdir /tmp/mqcheck -sudo python3 /tmp/mqsnap.py "$D/mealquest.db" /tmp/mqcheck/mealquest.db -sudo chown -R "$USER" /tmp/mqcheck -git ls-tree --name-only "$P" migrations/ > /tmp/mqcheck/known # top level only -python3 - /tmp/mqcheck/mealquest.db /tmp/mqcheck/known <<'PY' -import sqlite3, sys -applied = {v for (v,) in sqlite3.connect(sys.argv[1]).execute("select version from schema_migrations")} -known = {n[len("migrations/"):-len(".sql")] for n in open(sys.argv[2]).read().split() if n.endswith(".sql")} -if not applied or not known: - sys.exit("no versions read: check the snapshot and the previous SHA, then run again") -unknown = sorted(applied - known) -print("unknown to the previous SHA:", " ".join(unknown) if unknown else "none") -PY -) -``` - -The check runs in a subshell that stops at the first failed step — the -snapshot or its integrity check, `git ls-tree` or the comparison. On the -way out it always attempts to remove `/tmp/mqcheck`, and a failed removal -fails the check too; every nonzero exit prints `check FAILED`. Read it -like this: - -- `unknown to the previous SHA: none` and `echo $?` prints `0` — no - migration crossed: take the plain restart below. -- `unknown to the previous SHA:` followed by versions, and `echo $?` - prints `0` — the older build does not know them: the migration-crossing - path below is required. -- `check FAILED`, any other error, or a nonzero `echo $?` — stop and fix - the cause, then run the check again. Do not pick a path, even if an - `unknown to the previous SHA:` line was printed before it. If the - clean-up failed, remove `/tmp/mqcheck` by hand first. - -**No migration crossed** — rebuild and restart the older SHA: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the rollback with a nonzero status -P= -fail() { echo "$*" >&2; exit 1; } -sha= -# On the way out, print the running SHA only if every check passed. -trap 'st=$? -if [ "$st" -eq 0 ] && [ -n "$sha" ]; then echo "rolled back to: $sha" -else [ "$st" -ne 0 ] || st=1; echo "rollback FAILED (status $st)" >&2; fi -exit "$st"' EXIT -git checkout --detach "$P" -docker compose -f deploy/compose.yaml build -sudo systemctl restart mealquest -docker compose -f deploy/compose.yaml ps -for s in mealquest litestream searxng; do - id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") || fail "docker compose ps failed" - [ -n "$id" ] || fail "$s is not running" -done -curl -fsS http://10.60.1.14:8090/health -sha=$(git rev-parse HEAD) -) -``` - -Each step runs only if the one before it succeeded, so the health -check answers only for the older SHA. Only `rolled back to:` followed by -the SHA, with `echo $?` printing `0`, means the rollback is done. On -`rollback FAILED` or a nonzero `echo $?` it is not: the first error names -the failed step, and if it came before the restart, production is still -on the release being rolled back from. - -**A migration crossed** — the older binary refuses the newer schema, so the -database is restored first and the older SHA started after. Changes made -since the pre-deploy restore point are lost; say so to the household. - -1. Check out and build the older SHA. Nothing restarts yet. - -```sh -git checkout --detach && -docker compose -f deploy/compose.yaml build -``` - - If `echo $?` then prints anything but `0`, stop: the database has not - been touched and nothing has restarted. - -2. Run "Restore the database" steps 1–4, adding - `-timestamp ` (recorded in step 2 of - "Deploy") to the `restore` command. Its step 5 then starts the older - SHA and checks health. - -## Backups - -### Database — Litestream - -The `litestream` compose service streams every committed change to -`mealquest.db` to the NAS over SFTP (`deploy/litestream.yml`). It takes a -full snapshot daily (compaction level 9) and keeps a week of snapshots -plus the incremental LTX files between them. - -One-time setup (already done on 10.60.1.14): - -1. Generate the dedicated keypair, git-ignored, owned by the container - user: - - ```sh - cd /home/overseer/mealquest - ssh-keygen -t ed25519 -N '' -C mealquest-litestream -f deploy/secrets/litestream_key - # append deploy/secrets/litestream_key.pub to joshua@10.60.1.50:~/.ssh/authorized_keys - rm deploy/secrets/litestream_key.pub - sudo chown 10001:10001 deploy/secrets/litestream_key - sudo chmod 0600 deploy/secrets/litestream_key - ``` - -2. The NAS host key is pinned in `deploy/litestream.yml` (`host-key`). - -Check replication — list every LTX file at every level. Level 9 holds the -full snapshots; the default listing (`-level 0`) shows only L0 files: - -```sh -docker compose -f deploy/compose.yaml exec -T litestream \ - litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db -``` - -### Media — nightly rsync - -`mealquest-media-backup.timer` runs `mealquest-media-backup.service` at -03:30 every night. The service runs `/usr/local/sbin/mealquest-media-backup` -(`deploy/systemd/mealquest-media-backup.sh`) as the unprivileged host user -`mealquest` (uid/gid 10001, the app's own user) with no capabilities. -systemd bind-mounts the data volume read-only at `/run/mealquest-data` and -passes the Litestream key in as the credential `nas_key`; uid 10001 cannot -otherwise reach `/var/lib/docker` or `deploy/secrets/`. The script refuses a -`media` path that is a symlink and fails (exit 1) if `media` is missing. - -One-time setup (already done on 10.60.1.14): - -```sh -cd /home/overseer/mealquest -# Host user for the backup; ssh needs a passwd entry for its uid. -sudo groupadd --system --gid 10001 mealquest -sudo useradd --system --uid 10001 --gid 10001 --no-create-home \ - --home-dir /nonexistent --shell /usr/sbin/nologin mealquest -# Pinned NAS host keys, root-owned and world-readable. -sudo install -d -o root -g root -m 0755 /etc/mealquest -sudo install -o root -g root -m 0644 deploy/known_hosts /etc/mealquest/known_hosts -# The media directory; the backup fails if it is ever missing. -sudo install -d -o 10001 -g 10001 -m 0755 /var/lib/docker/volumes/mealquest-data/_data/media -# Script and units. -sudo install -o root -g root -m 0755 deploy/systemd/mealquest-media-backup.sh /usr/local/sbin/mealquest-media-backup -sudo install -o root -g root -m 0644 deploy/systemd/mealquest-media-backup.service \ - deploy/systemd/mealquest-media-backup.timer /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now mealquest-media-backup.timer -``` - -Run it by hand and read the result: - -```sh -sudo systemctl start mealquest-media-backup.service -journalctl -u mealquest-media-backup --no-pager -n 30 -``` - -To restore media, follow "Restore photos (media)" — never copy back while -the timer or the service can still run. - -## Restore the database - -SQLite in WAL mode keeps recent commits in `mealquest.db-wal`, indexed by -`mealquest.db-shm`. **A restored database must never sit beside an old WAL -or shared-memory file**: on open, SQLite would replay the old WAL's pages -onto the restored database and corrupt it or mix in state from the -database it replaced. So all three files are moved aside together, with -Litestream's local state directory `.mealquest.db-litestream`, before the -restore — and the restore runs into an empty volume. - -Steps 1–3: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the restore with a nonzero status -D=/var/lib/docker/volumes/mealquest-data/_data -A=/var/tmp/mealquest-restore-aside-$(date +%Y%m%d%H%M) -fail() { echo "$*" >&2; exit 1; } -# On the way out, name the aside directory; on any failure the stack -# stays stopped. -trap 'st=$? -if [ "$st" -eq 0 ]; then echo "restored mealquest.db; the old files are in $A: run step 4" -else echo "restore FAILED (status $st): leave the stack stopped; anything moved aside is in $A" >&2; fi -exit "$st"' EXIT - -# 1. Stop the whole stack (app and Litestream). Nothing may hold the -# database open; a clean close checkpoints the WAL into mealquest.db. -sudo systemctl stop mealquest - -# 2. Move the database, its -wal and -shm files and Litestream's state -# aside — all of them, every time. Keep the directory until the -# restored stack is verified. -sudo mkdir "$A" -for f in mealquest.db mealquest.db-wal mealquest.db-shm .mealquest.db-litestream; do - if sudo test -e "$D/$f"; then sudo mv "$D/$f" "$A/"; fi -done -left=$(sudo ls -A "$D") -[ "$left" = media ] || fail "the volume must hold only media/, not: $left" - -# 3. Restore from the NAS. --no-deps keeps the app stopped so it cannot -# create a fresh, empty database first. -docker compose -f deploy/compose.yaml run --rm --no-deps litestream \ - restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db -left=$(sudo ls -A "$D") -[ "$left" = "$(printf 'mealquest.db\nmedia')" ] || fail "the volume must hold only mealquest.db and media/ (no -wal, no -shm), not: $left" -) -``` - -To restore to a point in time, add `-timestamp 2026-10-05T23:00:00Z` (or -`-txid`) to the `restore` command. - -The block stops at the first failed step. Only `restored mealquest.db` -with `echo $?` printing `0` lets you go on; on `restore FAILED` leave the -stack stopped, fix the cause and run the block again (a new aside -directory is made; the earlier one keeps what it holds). - -4. Check the restored file before starting the app: run the copy block - in "Count check". - -5. Start the stack and check health, only after step 4 passed: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the check with a nonzero status -fail() { echo "$*" >&2; exit 1; } -sha= -# On the way out, print the running SHA only if every check passed. -trap 'st=$? -if [ "$st" -eq 0 ] && [ -n "$sha" ]; then echo "stack running and healthy at $sha" -else [ "$st" -ne 0 ] || st=1; echo "start FAILED (status $st)" >&2; fi -exit "$st"' EXIT -sudo systemctl start mealquest -docker compose -f deploy/compose.yaml ps -for s in mealquest litestream searxng; do - id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s") || fail "docker compose ps failed" - [ -n "$id" ] || fail "$s is not running" -done -curl -fsS http://10.60.1.14:8090/health -sha=$(git rev-parse HEAD) -) -``` - - Only `stack running and healthy at` with `echo $?` printing `0` ends - the restore; then the aside directory may go. - -### Count check - -`/tmp/mqcount.py` prints row counts for key tables, the number of -application tables, a SHA-256 over every application row (first 16 hex -digits; Litestream's own `_litestream*` tables are excluded) and -`PRAGMA integrity_check`. Run it on a **copy** of the database, never on the -live file. - -While the app or Litestream has the database open, the copy is a snapshot -from `/tmp/mqsnap.py`: it opens the live database read-only, copies it -with SQLite's online backup API (one consistent read, whatever is still in -the WAL included) and exits nonzero unless the snapshot's -`PRAGMA integrity_check` is `ok`. Never copy `mealquest.db` and its `-wal` -as files while they are open: a checkpoint between the two copies leaves -an older state that still reads cleanly. "Deploy" step 2 and "Rollback to -a previous SHA" use the snapshot. With the stack stopped, as in "Restore -the database" step 4, the files are closed and checkpointed, and a plain -copy is consistent (a read-only open there would leave a `-wal` and -`-shm` beside the restored file). - -`mode=ro` stops writes through the snapshot's own connection, but SQLite -can still create a missing `-wal` or `-shm` beside the database, and under -`sudo` it would be owned by root. The live snapshot therefore expects the -app and Litestream to be running and holding the existing -`mealquest.db-wal` and `mealquest.db-shm`, owned by `10001`. If -`sudo ls -ln "$D"` after a live snapshot shows either sidecar owned by -`0` — the snapshot created it — stop before going further: check -`docker compose -f deploy/compose.yaml ps` (mealquest and litestream -running) and, with the cause understood, return the sidecars to -`10001:10001` before proceeding. - -`/tmp` is temporary, so write the scripts before every check: - -```sh -cat > /tmp/mqcount.py <<'EOF' && -import sqlite3, sys, hashlib -c = sqlite3.connect(sys.argv[1]) -tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")] -for t in ("dinners", "week_plans", "schema_migrations"): - print(t, c.execute(f"select count(*) from {t}").fetchone()[0]) -h = hashlib.sha256() -for t in tabs: - for row in c.execute(f"select * from \"{t}\" order by rowid"): - h.update(repr(row).encode()) -print("tables", len(tabs), "content-sha256", h.hexdigest()[:16]) -print("integrity", c.execute("pragma integrity_check").fetchone()[0]) -EOF -cat > /tmp/mqsnap.py <<'EOF' -import os, sqlite3, sys -src, dst = sys.argv[1], sys.argv[2] -if os.path.exists(dst): - sys.exit(f"{dst} already exists") -s = sqlite3.connect(f"file:{src}?mode=ro", uri=True) -d = sqlite3.connect(dst) -s.backup(d) -s.close() -result = d.execute("pragma integrity_check").fetchall() -d.close() -if result != [("ok",)]: - sys.exit(f"snapshot integrity_check failed: {result}") -print("snapshot", dst, "integrity ok") -EOF -``` - -The second file is written only if the first was; `echo $?` must then -print `0` before either script is used. - -With the stack stopped, copy the database and run it: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the check with a nonzero status -D=/var/lib/docker/volumes/mealquest-data/_data -fail() { echo "$*" >&2; exit 1; } -# On the way out, remove /tmp/mqcheck whatever happens, keep any failure, -# and give the verdict only if every step and the clean-up succeeded. -trap 'st=$? -rm -rf /tmp/mqcheck || st=1 -if [ "$st" -eq 0 ]; then echo "count check: integrity ok" -else echo "count check FAILED (status $st): do not start the stack" >&2; fi -exit "$st"' EXIT -rm -rf /tmp/mqcheck -mkdir /tmp/mqcheck -sudo sh -c "cp -a $D/mealquest.db* /tmp/mqcheck/" -sudo chown -R "$USER" /tmp/mqcheck -python3 /tmp/mqcount.py /tmp/mqcheck/mealquest.db > /tmp/mqcheck/counts.txt -cat /tmp/mqcheck/counts.txt -grep -qx 'integrity ok' /tmp/mqcheck/counts.txt || fail "the copy failed integrity_check" -) -``` - -The check stops at the first failed step — the clean-up of an earlier -`/tmp/mqcheck`, the copy, the count or the integrity check — so it never -reads a file left by an earlier run. Only `count check: integrity ok` -with `echo $?` printing `0` passes; then the counts printed above it, -before and after, must match exactly. - -## Restore photos (media) - -Recipe and dinner photos live in `media/` in the data volume and are -backed up nightly to `joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/` -(see "Media — nightly rsync"). The backup service mirrors with -`rsync --delete`. **An `rsync --delete` run must never see an empty or -partial `media/`**: it would delete the NAS copy you are about to restore -from. Stopping the timer is not enough — it does not stop a run that is -already in flight, and the service can still be started by hand. So stop -**both** the timer and the service, confirm no rsync is running, and do -not start the service until the restore is verified. - -Run from `/home/overseer/mealquest`: - -```sh -cd /home/overseer/mealquest && ( -set -eu # any failed step ends the restore with a nonzero status -M=/var/lib/docker/volumes/mealquest-data/_data/media -NAS=/volume1/homes/Joshua/mealquest-backups/media -SSH="ssh -i deploy/secrets/litestream_key -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/mealquest/known_hosts -o StrictHostKeyChecking=yes" -fail() { echo "$*" >&2; exit 1; } -nas= here= w= -# On the way out, give the counts only if every step passed; on any -# failure, report the timer's actual state, not an assumed one. -trap 'st=$? -[ -z "$w" ] || rm -rf "$w" -if [ "$st" -eq 0 ] && [ -n "$here" ]; then echo "media restored: $here files here, $nas on the NAS, all $nas match the NAS manifest" -else [ "$st" -ne 0 ] || st=1; t=$(systemctl is-active mealquest-media-backup.timer) || : - if [ "$t" = inactive ]; then echo "media restore FAILED (status $st): the timer is inactive; leave it stopped and find out why" >&2 - else echo "media restore FAILED (status $st): the timer is ${t:-unknown}, not inactive; run sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service and confirm it is inactive before anything else" >&2; fi; fi -exit "$st"' EXIT -w=$(mktemp -d) - -# 1. Stop the nightly mirror: the timer AND any run in flight. -sudo systemctl stop mealquest-media-backup.timer mealquest-media-backup.service - -# 2. Pre-check — all must hold before media/ is touched. is-active exits -# nonzero for every state but active, so its printed state is checked; -# pgrep exits 0 on a match, 1 on none, 2+ on an error. -ts=$(systemctl is-active mealquest-media-backup.timer) || : -[ "$ts" = inactive ] || fail "the timer is ${ts:-unknown}, not inactive" -ss=$(systemctl is-active mealquest-media-backup.service) || : -case "$ss" in - inactive|failed) ;; - *) fail "the service is ${ss:-unknown}, not inactive or failed";; -esac -ps=0 -pgrep -a -u mealquest rsync || ps=$? -case "$ps" in - 0) fail "an rsync is still running as mealquest";; - 1) ;; # no match: no rsync - *) fail "pgrep failed (status $ps)";; -esac -l=$(sudo $SSH joshua@10.60.1.50 "find $NAS -type f") || fail "listing the NAS copy failed" -[ -n "$l" ] || fail "the NAS copy holds no files" -nas=$(printf '%s\n' "$l" | wc -l) # the target below -# The NAS manifest: one "sha256 ./path" line per file, taken now, before -# the copy. Every check below compares content against it, not counts. -sudo $SSH joshua@10.60.1.50 "cd $NAS && find . -type f -exec sha256sum {} +" >"$w/raw" || fail "hashing the NAS copy failed" -LC_ALL=C sort "$w/raw" >"$w/nas" -[ "$(wc -l <"$w/nas")" -eq "$nas" ] || fail "the NAS manifest has $(wc -l <"$w/nas") files, the listing $nas" - -# 3. Stop the app (no writes into media/ meanwhile). -docker compose -f deploy/compose.yaml stop mealquest - -# 4. Recreate the directory if it is gone, owned by the app user. -sudo install -d -o 10001 -g 10001 -m 0755 "$M" - -# 5. Copy back from the NAS as uid 10001's files. No --delete in this -# direction: the NAS copy is the source and is only read. --checksum -# compares content, so a local file with the NAS size and mtime but -# other bytes is still replaced. -sudo rsync -a --checksum --stats --chown=10001:10001 -e "$SSH" "joshua@10.60.1.50:$NAS/" "$M/" - -# 6. Post-condition — before anything can mirror again: at least the NAS -# count here, and the NAS count unchanged. -l=$(sudo find "$M" -type f) || fail "listing $M failed" -[ -n "$l" ] || fail "$M holds no files after the copy" -n=$(printf '%s\n' "$l" | wc -l) -[ "$n" -ge "$nas" ] || fail "only $n of the NAS's $nas files are here: the copy did not complete" -l=$(sudo $SSH joshua@10.60.1.50 "find $NAS -type f") || fail "listing the NAS copy failed" -[ -n "$l" ] && [ "$(printf '%s\n' "$l" | wc -l)" -eq "$nas" ] || fail "the NAS file count changed from $nas" -# Content: every file in the NAS manifest is here with the same bytes, -# and the NAS still matches the manifest taken before the copy. -sudo sh -c 'cd "$1" && find . -type f -exec sha256sum {} +' sh "$M" >"$w/raw" || fail "hashing $M failed" -LC_ALL=C sort "$w/raw" >"$w/here" -LC_ALL=C comm -23 "$w/nas" "$w/here" >"$w/bad" -[ ! -s "$w/bad" ] || { cat "$w/bad" >&2; fail "$(wc -l <"$w/bad") NAS file(s) above are missing here or differ in content"; } -sudo $SSH joshua@10.60.1.50 "cd $NAS && find . -type f -exec sha256sum {} +" >"$w/raw" || fail "hashing the NAS copy failed" -LC_ALL=C sort "$w/raw" | cmp -s - "$w/nas" || fail "the NAS copy changed during the restore" -here=$n - -# 7. Start the app, then the timer again. Only now may the service run. -docker compose -f deploy/compose.yaml up -d -curl -fsS http://10.60.1.14:8090/health -sudo systemctl start mealquest-media-backup.timer -systemctl is-active mealquest-media-backup.timer # active; exits nonzero otherwise -) -``` - -The block stops at the first failed step and gives the counts last, only -when every step passed and every file in the NAS manifest (a SHA-256 of -each file, taken before the copy) is present locally with the same bytes. -On `media restore FAILED` or a nonzero `echo $?` the message gives the -timer's actual state: if it is not `inactive`, stop it as the message says -before anything else; then leave it stopped. The local count may exceed the NAS count only if -`media/` still held files the NAS never received (photos added since the -last nightly run); the next nightly run then copies them up. If the local count is lower, -the copy did not complete: fix the cause and run the block again. - -When the database is restored too, restore it first ("Restore the -database"), then the media, so every photo the database refers to is -present. This media restore has not been run against production yet; the -backup direction was checked on 2026-10-05 (see "Media backup checks"). - -## Restore drill record — 2026-10-05 - -Drill on 10.60.1.14 against the live (pre-launch, empty-household) -database, Litestream 0.5.17. Commands as in "Restore the database"; -output trimmed only of compose progress lines. - -Replica before the drill (`ltx -level all`; level 9 is the full snapshot): - -``` -level min_txid max_txid size created -0 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z -1 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z -1 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z -2 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z -2 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z -3 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z -9 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z -``` - -1. `docker compose -f deploy/compose.yaml down` — both containers stopped - and removed. The volume then held `mealquest.db`, - `.mealquest.db-litestream/` and `media/`; the clean close had - checkpointed and removed `-wal` and `-shm`. -2. Before (copy of the stopped database): - - ``` - dinners 0 - week_plans 0 - schema_migrations 7 - tables 26 content-sha256 836e230e16c58dc7 - integrity ok - ``` - -3. Moved aside to `/var/tmp/mealquest-restore-aside-20261005`: - `mealquest.db` and `.mealquest.db-litestream` (no `-wal`/`-shm` existed - after the clean stop; the loop moves them whenever they do). Volume - afterwards: `media/` only. -4. `docker compose -f deploy/compose.yaml run --rm --no-deps litestream restore -config /etc/litestream.yml -o /data/mealquest.db /data/mealquest.db` - — volume afterwards: `mealquest.db` (258048 bytes) and `media/`; no - `-wal`, no `-shm`. -5. After (copy of the restored file, before the app started): - - ``` - dinners 0 - week_plans 0 - schema_migrations 7 - tables 26 content-sha256 836e230e16c58dc7 - integrity ok - ``` - -6. `docker compose -f deploy/compose.yaml up -d` — `mealquest-mealquest-1` - healthy, `mealquest-litestream-1` up; `GET /health` → `200`. Counts on a - copy of the running database (`mealquest.db`, `-wal`, `-shm`) were - identical to the above. -7. Litestream resumed against the restored database: - - ``` - level=INFO msg="detected database behind replica" db=mealquest.db db_txid=0000000000000000 replica_txid=0000000000000003 - level=INFO msg="fetched latest L0 file from replica" db=mealquest.db min_txid=0000000000000003 max_txid=0000000000000003 - level=INFO msg="compaction complete" db=mealquest.db level=1 txid.min=0000000000000004 txid.max=0000000000000004 size=14110 - ``` - - Replica after the drill (`ltx -level all`): - - ``` - level min_txid max_txid size created - 0 0000000000000004 0000000000000004 14110 2026-10-05T23:27:21Z - 1 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z - 1 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z - 1 0000000000000004 0000000000000004 14110 2026-10-05T23:27:21Z - 2 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z - 2 0000000000000003 0000000000000003 14110 2026-10-05T23:13:30Z - 3 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z - 9 0000000000000001 0000000000000002 14110 2026-10-05T23:12:38Z - ``` - -The first drill earlier the same day (before `-level all` was recorded) -gave the same before/after result: checksum `836e230e16c58dc7`, 26 tables, -integrity ok. - -### Media backup checks — 2026-10-05 - -`sudo systemctl start mealquest-media-backup.service` on 10.60.1.14: - -| Case | Result | -| --- | --- | -| Normal run | `Result=success`; journal: `synced /run/mealquest-data/media to joshua@10.60.1.50:/volume1/homes/Joshua/mealquest-backups/media/ as uid 10001` | -| A test file added | `Number of regular files transferred: 1`; removing it then gave `Number of deleted files: 1 (reg: 1)` | -| `media` replaced by a symlink to `/etc` | `Result=exit-code`, status 1; `/run/mealquest-data/media is a symlink to /etc; refusing to back it up` | -| `media` removed | `Result=exit-code`, status 1; `media directory /run/mealquest-data/media is missing` | -| `sudo -u mealquest` writing into the volume via `/var/lib/docker/...` | `Permission denied` — uid 10001 cannot traverse `/var/lib/docker` (mode 0710 root) | - -## Secret rotation - -Rotate a secret when it may have leaked, when someone with access leaves, -or on the owner's schedule. Never paste a secret into a ticket, chat or -this file. - -### App key (`deploy/secrets/secret_key`) - -The app key encrypts every stored secret: the `_ct` columns -`ai_settings.api_key_ct` (LLM API key), `recipe_search_settings.api_key_ct` -(recipe-search API key), `integrations_google.refresh_token_ct` (Google -Calendar refresh token) and `chat_settings.webhook_url_ct` (Google Chat -webhook URL). **Rotating it re-encrypts nothing**: a value encrypted with -the old key can no longer be read. A `mealquest rotate-key` helper that -re-encrypts in place is planned but does not exist yet. - -**What is recoverable today.** The service currently exposes only -household settings; no shipped flow writes or re-enters any of these four -secrets. The re-entry flows are planned: the LLM key in Settings → AI -assistant (the AI settings feature), the recipe-search key (Feature 10), -Google Calendar reconnect (Feature 16) and the Google Chat webhook -(Feature 19). So today the rotation is safe **only if no `_ct` value is -stored**, which the pre-check below proves. If the pre-check finds a -stored value, do not rotate: it could not be re-entered afterwards. Keep -the current key and wait for the matching flow (or the `rotate-key` -helper). - -**The pre-check and the key replacement run with the whole stack -stopped**, from the pre-check until the new key is in place. A raw copy of -`mealquest.db` and `mealquest.db-wal` taken while the stack runs is **not** -a consistent snapshot: the two files are copied one after the other, and a -checkpoint (by the app or by Litestream) between the two copies can move -committed pages out of the WAL into `mealquest.db` after `mealquest.db` was -already copied, then reset the WAL before it is copied. The copy then -lacks those commits — possibly a stored `_ct` value — and the pre-check -reports a false 0. A running app could also store a `_ct` value after a -correct pre-check and before the key is replaced. Stopping both containers -removes every writer (Litestream also writes and checkpoints), and the -clean close checkpoints the WAL into `mealquest.db`, so the file alone is -complete. - -Run from `/home/overseer/mealquest`: - -```sh -D=/var/lib/docker/volumes/mealquest-data/_data - -# 1. Stop the whole stack (app and Litestream) and leave it stopped until -# step 4. Nothing may hold the database open. -sudo systemctl stop mealquest -sudo ls -la "$D" # mealquest.db and media/ only: no -wal, no -shm; if - # either exists, the close was not clean — stop here - -# 2. Pre-check, on a copy of the now-quiescent database. -rm -rf /tmp/mqcheck && mkdir /tmp/mqcheck -sudo cp -a "$D/mealquest.db" /tmp/mqcheck/ && sudo chown -R "$USER" /tmp/mqcheck -python3 - /tmp/mqcheck/mealquest.db <<'PY' -import sqlite3, sys -c = sqlite3.connect(sys.argv[1]) -for t, col in (("ai_settings", "api_key_ct"), ("recipe_search_settings", "api_key_ct"), - ("integrations_google", "refresh_token_ct"), ("chat_settings", "webhook_url_ct")): - print(t, col, c.execute(f"select count(*) from {t} where {col} is not null").fetchone()[0]) -PY -rm -rf /tmp/mqcheck -# every count must be 0; otherwise start the stack again on the current -# key (sudo systemctl start mealquest) and stop here - -# 3. Replace the key, still with the stack stopped. -sudo cp -a deploy/secrets/secret_key /var/tmp/secret_key.old # keep until verified, then shred -head -c 32 /dev/urandom | base64 | sudo tee deploy/secrets/secret_key >/dev/null -sudo chown 10001:10001 deploy/secrets/secret_key -sudo chmod 600 deploy/secrets/secret_key - -# 4. Start the stack on the new key and check health. -sudo systemctl start mealquest -docker compose -f deploy/compose.yaml ps -curl -fsS http://10.60.1.14:8090/health -``` - -Then sign in and open household settings to confirm the service runs on -the new key. Finally `sudo shred -u /var/tmp/secret_key.old`. Once the -planned flows ship, this section gains a step per secret to re-enter it -and confirm it works. - -### OIDC client secret (`deploy/secrets/oidc_client_secret`) - -In Authentik, open the provider `mealquest` and regenerate the Client -Secret. Write it to the file (same owner and mode as above) and recreate -the service: - -```sh -printf '%s\n' '' | sudo tee deploy/secrets/oidc_client_secret >/dev/null -sudo chown 10001:10001 deploy/secrets/oidc_client_secret -sudo chmod 600 deploy/secrets/oidc_client_secret -docker compose -f deploy/compose.yaml up -d --force-recreate mealquest -``` - -Sign in at to confirm; sign-ins -fail between regenerating the secret in Authentik and recreating the -service. - -### Google OAuth client secret (`deploy/secrets/google_client_secret`) - -In the Google Cloud console, open the OAuth client `mealquest`, add a new -client secret, write it to the file (same owner and mode as above) and -recreate the service: - -```sh -printf '%s\n' '' | sudo tee deploy/secrets/google_client_secret >/dev/null -sudo chown 10001:10001 deploy/secrets/google_client_secret -sudo chmod 600 deploy/secrets/google_client_secret -docker compose -f deploy/compose.yaml up -d --force-recreate mealquest -``` - -Then disable and delete the old secret in the console. A new client secret -leaves the household's stored refresh token valid, so nothing needs -reconnecting. - -### LLM API key - -Stored encrypted in the database, not in `deploy/`. In the app, open -**Settings → AI assistant**, choose **Replace**, paste the new key and -**Verify**, and check the verified state is shown. Then revoke the -old key in the provider's console. (The AI assistant settings page is -delivered by its own feature; until it ships there is no stored LLM key -to rotate.) - -### VAPID keys (Web Push) - -Planned with Web Push (Feature 19): the public key goes in -`deploy/.env` as `MEALQUEST_VAPID_PUBLIC_KEY`, the private key in -`deploy/secrets/vapid_private_key` (owner `10001:10001`, mode 0600), and -the pair is generated with `cmd/vapidgen`. **Changing the VAPID pair -invalidates every push subscription**: each device must turn -notifications on again (`/settings/notifications`) before it receives -pushes. Rotate only when the private key may have leaked, write both -halves, recreate the service with `--force-recreate mealquest`, and tell -the household to re-enable notifications on each device. - -### SearXNG secret (`deploy/secrets/searxng.env`) - -Only signs SearXNG's own cookies and image-proxy links; nothing stored -depends on it. Replace it and recreate the service: - -```sh -(umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env) -docker compose -f deploy/compose.yaml up -d --force-recreate searxng -docker compose -f deploy/compose.yaml ps searxng # healthy -``` - -### Litestream / NAS key (`deploy/secrets/litestream_key`) - -Generate a new keypair as in "Database — Litestream", add the new public -key to `joshua@10.60.1.50:~/.ssh/authorized_keys`, replace the file -(owner `10001:10001`, mode 0600), run -`docker compose -f deploy/compose.yaml up -d --force-recreate litestream` -and `sudo systemctl start mealquest-media-backup.service`, check both -(`litestream ltx` and the journal), then remove the old public key from -the NAS. - -## Context records - -The confirmed values in this runbook (port 8090, checkout path, Authentik -group `mealquest-admin`, DNS, firewall, backup target) are also recorded in -the CTXD project `mealquest`: `RUNBOOKS.MD` (deployment procedure) and the -open questions in `CONTEXT.MD`, updated for Feature 07 task 876134bc. diff --git a/deploy/compose.dev.yaml b/deploy/compose.dev.yaml index ee071ba..3515700 100644 --- a/deploy/compose.dev.yaml +++ b/deploy/compose.dev.yaml @@ -1,8 +1,7 @@ -# Development override: publishes SearXNG on the dev machine's loopback so -# it can be queried directly, e.g. -# -# docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml up -d searxng -# curl 'http://127.0.0.1:8888/search?q=chicken+tikka+masala+recipe&format=json' +# Development override layered on deploy/compose.yaml: publishes SearXNG +# on the dev machine's loopback (127.0.0.1:8888) so it can be queried +# directly. Using it: CTXD project mealquest, RUNBOOKS.MD "Local +# development". # # Loopback only, never a LAN address. Never used in production: the # systemd unit starts deploy/compose.yaml alone. diff --git a/deploy/compose.yaml b/deploy/compose.yaml index f54b159..a85f106 100644 --- a/deploy/compose.yaml +++ b/deploy/compose.yaml @@ -1,18 +1,16 @@ -# Production stack for MealQuest on 10.60.1.14 (D-018). Build on the host, -# then start through systemd: -# -# docker compose -f deploy/compose.yaml build -# sudo systemctl restart mealquest +# Production stack for MealQuest on 10.60.1.14 (D-018). Building, deploying +# and operating it: CTXD project mealquest, RUNBOOKS.MD "Stack unit" and +# "Routine deploy and rollback". # # The services use restart: "no" on purpose: Docker must not start them # before the DOCKER-USER firewall is applied. deploy/systemd/mealquest.service # owns start and stop and requires mealquest-firewall.service first. # -# Configuration comes from deploy/.env (copy deploy/.env.example) and secrets -# from deploy/secrets/; neither is committed. Compose bind-mounts file secrets -# with their host owner and mode, so each file in deploy/secrets/ must be -# owned by the container user 10001:10001 with mode 0600 (except -# searxng.env, read by compose itself; see the searxng service). +# Configuration comes from deploy/.env (template: deploy/.env.example) and +# secrets from deploy/secrets/; neither is committed. Compose bind-mounts +# file secrets with their host owner and mode, so each file in +# deploy/secrets/ must be owned by the container user 10001:10001 with mode +# 0600 (except searxng.env, read by compose itself; see the searxng service). name: mealquest services: @@ -65,8 +63,8 @@ services: # the app's own user (owner of the volume) with a read-only root # filesystem, no privilege escalation and no Linux capabilities. # - # Restore: follow "Restore the database" in deploy/RUNBOOK.md (stop the - # stack, move the database with its -wal and -shm aside, restore, check). + # Restoring the database: CTXD project mealquest, RUNBOOKS.MD "Restore + # the database". litestream: image: litestream/litestream:0.5.17 user: "10001:10001" diff --git a/deploy/systemd/mealquest-firewall.service b/deploy/systemd/mealquest-firewall.service index c7c5f8e..90d2fdc 100644 --- a/deploy/systemd/mealquest-firewall.service +++ b/deploy/systemd/mealquest-firewall.service @@ -5,8 +5,9 @@ # are one-way: a failure here fails this unit visibly and never fails # docker.service. # -# Start MealQuest only after this unit has succeeded: -# systemctl start mealquest-firewall && docker compose up -d +# mealquest.service requires this unit and starts the stack only after it +# has succeeded. Installing and operating both: CTXD project mealquest, +# RUNBOOKS.MD "Host firewall" and "Stack unit". [Unit] Description=Restrict MealQuest port 8090 to Traefik (DOCKER-USER) After=docker.service diff --git a/deploy/systemd/mealquest-media-backup.service b/deploy/systemd/mealquest-media-backup.service index 9f5a61c..9222684 100644 --- a/deploy/systemd/mealquest-media-backup.service +++ b/deploy/systemd/mealquest-media-backup.service @@ -1,8 +1,7 @@ # Runs the nightly MealQuest media backup (/usr/local/sbin/mealquest-media-backup). -# Triggered by mealquest-media-backup.timer; run by hand with -# systemctl start mealquest-media-backup -# and read the result with -# journalctl -u mealquest-media-backup +# Triggered by mealquest-media-backup.timer; output goes to the journal. +# Installing it and running it by hand: CTXD project mealquest, +# RUNBOOKS.MD "Media — nightly rsync". # # The script runs as the app's user (10001, host user "mealquest") with no # capabilities. systemd, as root, only bind-mounts the data volume's root diff --git a/deploy/systemd/mealquest-media-backup.sh b/deploy/systemd/mealquest-media-backup.sh index f4ee14d..fd99363 100755 --- a/deploy/systemd/mealquest-media-backup.sh +++ b/deploy/systemd/mealquest-media-backup.sh @@ -15,8 +15,9 @@ # swapping the path afterwards cannot redirect it. Symlinks inside the # directory are copied as links (-a), never followed. # -# The media directory is created at deploy time (deploy/RUNBOOK.md). If it -# is missing the job fails; it is never treated as "nothing to back up". +# The media directory is created at deploy time (CTXD project mealquest, +# RUNBOOKS.MD "Media — nightly rsync"). If it is missing the job fails; it +# is never treated as "nothing to back up". # # rsync mirrors into .../mealquest-backups/media/ only: the target ends in # a slash, so --delete removes files from that directory and nothing