Docs refresh: architecture, agents and README reflect what was built (Maestro 3b22e48b) (#214)

Co-authored-by: orchestrator-agent <20+orchestrator-agent@code.cubecraftcreations.com>

AGENTS.md

@@ -78,17 +78,18 @@ behaviour, exercise it (hit the endpoint, render the page).
 | Path | Purpose |
 | --- | --- |
 | `cmd/server` | Entry point. Wiring only — no business logic. |
+| `cmd/vapidgen` | Prints a new VAPID key pair for Web Push. |
 | `internal/config` | `MEALQUEST_*` environment config; secrets from `*_FILE`. |
 | `internal/server` | HTTP server, routes, `/health`. |
 | `internal/frontend` | `go:embed` of the built PWA with SPA fallback. |
-| `internal/*` (planned) | `auth`, `db`, `store`, `secretbox`, `household`, `dinners`, `recipes`, `ai/llm`, `ai/harness`, `ai/skills`, `suggest`, `plan`, `shopping`, `ideas`, `gcal`, `notify`, `push`, `chat`, `jobs`, `scheduler`. |
-| `migrations/` | SQL migrations (planned). |
+| `internal/*` | `auth`, `forwarded`, `secureurl`, `db`, `store`, `secretbox`, `household`, `dinners`, `recipes`, `ai/llm`, `ai/harness`, `ai/skills`, `ai/aisettings`, `suggest`, `plan`, `shopping`, `ideas`, `home`, `gcal`, `notify`, `push`, `chat`, `vapid`, `jobs`, `scheduler` — one line each in [ARCHITECTURE.md](docs/ARCHITECTURE.md#repository-layout). |
+| `migrations/` | SQL migrations, embedded and applied at boot by `internal/store`; sqlc queries in `migrations/queries/`. |
 | `web/` | React PWA source. |
-| `deploy/` | Deployment files (planned). |
+| `deploy/` | Deployment files: compose stack, systemd units, [deploy runbook](deploy/RUNBOOK.md) and the automated deploy (`auto-deploy.sh` with `systemd/mealquest-auto-deploy.*`). |
 | `docs/` | Project documentation, including [ARCHITECTURE.md](docs/ARCHITECTURE.md). |
 | `design/` | Requirements and UI handoff. **Read-only reference** — do not edit. |
 | `Dockerfile` | Multi-stage image build, at the repository root. |
-| `Makefile` | `dev`, `test`, `build`, `docker`. |
+| `Makefile` | `dev`, `test`, `build`, `docker`; `sqlc` and `sqlc-check` regenerate and check `internal/db`. |
 | `.gitea/workflows/ci.yml` | CI quality floor. |
 
 Design references:
@@ -104,7 +105,10 @@ Design references:
 - **Task branches:** `task/<slug>_<task-uuid>`, branched from the feature branch.
 - Task pull requests **squash-merge into the feature branch**.
 - The feature pull request merges into `dev`.
-- **Only `overseer` merges `dev` → `main`** (a release).
+- **Only `overseer` merges `dev` → `main`** (a release). On the production
+  host, `mealquest-auto-deploy.timer` then deploys the new `main` once its
+  CI passes ([deploy/RUNBOOK.md](deploy/RUNBOOK.md) "Automated deploys");
+  nothing deploys `dev` or a feature branch.
 - Never push directly to `dev` or `main`.
 - Authors never review, approve, merge or close their own work.
 - Never store credentials in the repository; environment or ignored files only.

README.md

@@ -7,9 +7,27 @@ builds a weekly plan, pushes it to Google Calendar, and generates a shopping
 list grouped by dinner, kept as a browsable weekly history. It ships as a
 single Go service with the React PWA embedded in the binary.
 
-> Status: this repository is the Feature 01 scaffold (Go service with
-> `/health`, embedded PWA skeleton, Makefile, Dockerfile, CI). The planned
-> application is described in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
+Production runs at <https://mealquest.cubecraftlabs.com> for one household,
+signed in through Authentik.
+
+## Stack
+
+- **Backend:** Go 1.25 (`CGO_ENABLED=0`, one static binary), chi router,
+  SQLite through `modernc.org/sqlite` with sqlc queries and embedded
+  migrations, Authentik OIDC, `log/slog`. AI calls go to an
+  OpenAI-compatible, Anthropic or Ollama endpoint over plain HTTP; recipes
+  come from SearXNG or Brave search and the recipe page's schema.org data.
+  Google Calendar, Google Chat and Web Push are the integrations.
+- **Frontend:** React 19, TypeScript, Vite, React Router 7, TanStack Query 5,
+  `vite-plugin-pwa`, CSS Modules over the design tokens; tests with Vitest,
+  Testing Library and jsdom.
+- **Deployment:** a Docker Compose stack (the service, Litestream
+  replication to the NAS, a SearXNG sidecar) behind Traefik, with systemd
+  units for the firewall, nightly media backup and the automated deploy of
+  `main`.
+
+The approved dependency list is in [AGENTS.md](AGENTS.md); nothing else is
+added without the owner's approval.
 
 ## Running it locally
 
@@ -28,6 +46,15 @@ invalid variable.
 | `make test`    | `gofmt` check, `go vet`, `go test -race`, web typecheck and tests. |
 | `make build`   | Build the PWA, embed it, build the static binary `bin/mealquest`. |
 | `make docker`  | Build the container image from the root `Dockerfile`.           |
+| `make sqlc-check` | Regenerate `internal/db` with sqlc and fail if it drifted.   |
+
+## Deploying and operating
+
+[deploy/RUNBOOK.md](deploy/RUNBOOK.md) is the deploy runbook: build, ship and
+restart, verifying the deployed version, rollback and the automated deploy.
+Database restore, the boot configuration and other host procedures are in
+CTXD: project `mealquest`, file `RUNBOOKS.MD`. Only `overseer` releases, by
+merging `dev` into `main`.
 
 ## Where to read next
 

deploy/RUNBOOK.md

@@ -0,0 +1,357 @@
+# MealQuest deploy runbook
+
+The one way to deploy MealQuest to production: the docker compose project
+`mealquest` on office-srvr `10.60.1.14` (see
+[ARCHITECTURE.md](../docs/ARCHITECTURE.md#topology)). Production runs `main`
+or a release tag; only `overseer` merges `dev` → `main`. Every deploy builds
+on the host, from the checked-out commit, with `make docker`.
+
+Host setup (clone, firewall, the systemd unit, Traefik, DNS, Litestream) and
+the database restore are in CTXD project `mealquest`, `RUNBOOKS.MD`. This
+file is the build, restart, verification and rollback procedure; where it
+uses a CTXD section, it names it.
+
+All commands run on `10.60.1.14` as `overseer`, in bash, from the checkout
+`/home/overseer/mealquest`. A block that ends with `FAILED` or a non-zero
+`$?` stops the deploy: do not go on to the next step.
+
+## How the image carries its version
+
+`make docker` runs `docker build --build-arg VERSION=<version>` and tags the
+image `mealquest:<version>`, where `<version>` is
+`git describe --tags --always` of the checkout (a release tag such as
+`v1.0.0`, otherwise the short SHA), with `-dirty` added when a tracked file
+is modified. The [Dockerfile](../Dockerfile) links it into the binary, and
+`/health` reports it:
+
+```json
+{"status":"ok","version":"v1.0.0"}
+```
+
+Never pass `--build-arg VERSION` by hand, and never build with
+`docker compose build` or `docker compose up --build`: compose passes no
+version, so that image reports `dev`.
+
+The compose service `mealquest` runs the image named `mealquest-mealquest`
+(compose's name for the image it would build; check with
+`docker compose -f deploy/compose.yaml config --images mealquest`). It
+already exists, so `docker compose up` uses it and never builds. A deploy
+points that name at the `make docker` image with `docker tag`. Every
+`mealquest:<version>` image is kept for rollback; do not prune them.
+
+## Prerequisites
+
+- `make`, `git`, `curl`, `python3` and Docker Engine with Compose on the
+  host (`make` is not in the base install: `sudo apt-get install make`).
+- `deploy/.env` (git-ignored, owner `overseer`, mode 0600) sets every
+  variable [`deploy/.env.example`](.env.example) sets, with
+  `MEALQUEST_BIND_ADDRESS=10.60.1.14`. It must not set the variables
+  `deploy/compose.yaml` fixes (`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR`,
+  `MEALQUEST_SEARXNG_URL` and the `*_FILE` variables). Creating it: CTXD
+  `RUNBOOKS.MD` "Stack and first deploy" > "3. Environment file".
+- `deploy/secrets/` (git-ignored, mode 0700) holds `secret_key`,
+  `oidc_client_secret`, `google_client_secret`, `vapid_private_key` and
+  `litestream_key`, each owned by `10001:10001` with mode 0600, and
+  `searxng.env`, owned by `overseer` with mode 0600. Creating them: CTXD
+  `RUNBOOKS.MD` "Stack and first deploy" > "4. Secrets".
+- The compose file is the checkout's [`deploy/compose.yaml`](compose.yaml);
+  the systemd unit [`mealquest.service`](systemd/mealquest.service) starts
+  and stops it and is the only way the stack starts.
+- DNS and Traefik already route `https://mealquest.cubecraftlabs.com` to
+  `10.60.1.14:8090` (CTXD `RUNBOOKS.MD` "Reverse proxy (Traefik)", "DNS").
+
+Check `deploy/.env` and the secrets before every deploy (a new release may
+add a variable; the service refuses to boot without it). Both commands must
+print nothing:
+
+```sh
+cd /home/overseer/mealquest
+comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \
+         <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u)   # variables missing from deploy/.env
+sudo find deploy/secrets -type f ! -perm 600                       # secrets not mode 0600
+```
+
+## Deploy
+
+### 1. Build
+
+Record the release production is running now, check out the new release
+and build its image. Nothing restarts.
+
+The rollback target is the version the running service reports on
+`/health`, read before the checkout changes, never the checkout's `HEAD`:
+after a failed or deferred deploy the checkout can be ahead of production.
+The block resolves that version to a commit and prints it as the rollback
+SHA. A running version of `dev` or `*-dirty` names no commit: the block
+fails, and the rollback target must be established by hand before going on.
+
+The checkout must be clean (the same test as the Makefile's `GIT_DIRTY`:
+no modified tracked file outside the generated `internal/frontend/dist`).
+The block computes the image tag exactly as `make docker` does, `-dirty`
+included, and checks that image exists after the build.
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+built=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$built" ]; then echo "built: $built"
+else [ "$st" -ne 0 ] || st=1; echo "build FAILED (status $st)" >&2; fi
+exit "$st"' EXIT
+bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-)
+[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
+running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+  || fail "http://$bind:8090/health did not return a version: no running release to roll back to"
+case "$running" in
+  dev|*-dirty) fail "the running release reports $running, which names no commit" ;;
+esac
+git fetch --tags origin
+rollback=$(git rev-parse --verify --quiet "$running^{commit}") \
+  || fail "running version $running is not a commit in this repository"
+echo "rollback SHA: $rollback (running version $running)"   # RECORD it for rollback
+git checkout --detach "refs/remotes/origin/main^{commit}"   # or the release tag, e.g. v1.0.0
+dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist')
+[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first:
+$dirty"
+v=$(git describe --tags --always)${dirty:+-dirty}   # the Makefile's VERSION
+make docker
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed"
+built="mealquest:$v"
+)
+```
+
+### 2. Ship and restart
+
+1. **Pre-restart gates.** Run CTXD `RUNBOOKS.MD` "Routine deploy and
+   rollback" > "Deploy" step 2 (a verified pre-deploy restore point; record
+   it with the rollback SHA) and step 3 (boot-required configuration). The
+   restart applies migrations the previous release refuses, so without that
+   restore point there is no rollback. If either fails, do not restart:
+   production stays on the previous release.
+2. **Point the stack at the new image and restart.** `docker compose pull`
+   fetches the pinned `litestream` and `searxng` images before anything
+   stops; `mealquest` is never pulled. The restart applies the firewall,
+   then runs `down` and `up -d`, and returns once `mealquest` passes its
+   healthcheck (`/health` inside the container).
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+up=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$up" ]; then echo "restarted on: $up"
+else [ "$st" -ne 0 ] || st=1; echo "restart FAILED (status $st)" >&2; fi
+exit "$st"' EXIT
+v=$(git describe --tags --always)
+docker compose -f deploy/compose.yaml pull litestream searxng
+docker tag "mealquest:$v" mealquest-mealquest
+sudo systemctl restart mealquest
+for s in mealquest litestream searxng; do
+  id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s")
+  [ -n "$id" ] || fail "$s is not running"
+done
+id=$(docker compose -f deploy/compose.yaml ps --quiet mealquest)
+health=$(docker inspect -f '{{.State.Health.Status}}' "$id")
+[ "$health" = healthy ] || fail "mealquest is $health, not healthy"
+up="mealquest:$v"
+)
+```
+
+On failure read `journalctl -u mealquest` and
+`docker compose -f deploy/compose.yaml logs mealquest`: a configuration
+error names the variable. Fix it or roll back.
+
+### 3. Verify the deployed version and a real screen
+
+A `/health` 200 alone proves only that some build is up. This step checks,
+on the LAN address and through Traefik, that:
+
+- `/health` reports exactly the checkout's `git describe --tags --always`;
+- `/` and the Plan screen's route `/plan` serve the built PWA shell (the
+  React root and a hashed `/assets/` script, not the placeholder page an
+  unbuilt embed serves);
+- that script is served as JavaScript.
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+want=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$want" ]; then echo "deployed and verified: $want ($(git rev-parse HEAD))"
+else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): the deploy is not verified" >&2; fi
+exit "$st"' EXIT
+v=$(git describe --tags --always)
+for base in http://10.60.1.14:8090 https://mealquest.cubecraftlabs.com; do
+  got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+    || fail "$base/health did not return a version"
+  [ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v"
+  for path in / /plan; do
+    page=$(curl -fsS "$base$path") || fail "$base$path did not return 200"
+    grep -q '<div id="root"></div>' <<< "$page" || fail "$base$path is not the PWA shell"
+    js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true
+    [ -n "$js" ] || fail "$base$path references no built /assets/ script"
+    type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200"
+    case "$type" in
+      text/javascript*) ;;
+      *) fail "$base$js is $type, not JavaScript" ;;
+    esac
+  done
+  echo "$base: version $got, PWA shell and $js served"
+done
+want=$v
+)
+```
+
+- `deployed and verified: <version> (<SHA>)` and `$?` = `0` → open
+  `https://mealquest.cubecraftlabs.com` in a browser: sign in and the Home
+  screen renders with the household's data. Then record the deployed
+  version and SHA, the rollback SHA, the pre-deploy restore point and the
+  date in the release's Maestro task.
+- `verification FAILED`, any error or a non-zero `$?` → the deploy is not
+  done. A version mismatch means the stack runs a different image than the
+  checkout: repeat step 2, or roll back.
+
+## Rollback to a previous release
+
+Rolling back points `mealquest-mealquest` at the previous release's
+`mealquest:<version>` image again; that release's checkout keeps the
+compose file and the version check consistent with it.
+
+1. **Check whether the rollback crosses a migration** with CTXD
+   `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a previous
+   SHA" (its first block), run against the rollback SHA that deploy step 1
+   recorded. If it prints versions unknown to the rollback SHA, restore the
+   database to the pre-deploy restore point first (CTXD `RUNBOOKS.MD`
+   "Restore the database", with `-timestamp <restore point>`); changes
+   since then are lost, so tell the household.
+2. **Check out the rollback SHA and retag its image.** An image built before
+   this runbook was not tagged `mealquest:<version>`; `make docker` builds
+   it from the same commit.
+
+```sh
+cd /home/overseer/mealquest && (
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+P=<rollback SHA>   # recorded by step 1, from the running release's /health
+back=
+trap 'st=$?
+if [ "$st" -eq 0 ] && [ -n "$back" ]; then echo "rolled back to: $back"
+else [ "$st" -ne 0 ] || st=1; echo "rollback FAILED (status $st)" >&2; fi
+exit "$st"' EXIT
+git checkout --detach "$P"
+v=$(git describe --tags --always)
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || make docker
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v"
+docker tag "mealquest:$v" mealquest-mealquest
+sudo systemctl restart mealquest
+back="mealquest:$v"
+)
+```
+
+3. **Run step 3 above** unchanged. The rollback is done only when it prints
+   `deployed and verified:` with the previous release's version.
+
+## Automated deploys
+
+[`auto-deploy.sh`](auto-deploy.sh), installed as
+`/usr/local/sbin/mealquest-auto-deploy`, runs this runbook's deploy unattended:
+step 1 (build), the prerequisite `deploy/.env` and secrets check, CTXD
+`RUNBOOKS.MD` "Deploy" steps 2 (pre-deploy restore point) and 3
+(boot-required configuration), step 2 (ship and restart) and step 3
+(verify). No check is skipped or weakened, and it stops at the first
+failure. Every failure before the restart leaves production on the previous
+release.
+
+### What runs, and when
+
+[`mealquest-auto-deploy.timer`](systemd/mealquest-auto-deploy.timer) starts
+[`mealquest-auto-deploy.service`](systemd/mealquest-auto-deploy.service)
+at every quarter hour (`OnCalendar=*-*-* *:0/15:00`), plus up to 5 minutes'
+random delay; a check missed while the host was down runs as soon as it is
+back (`Persistent=true`). The service runs the script as `overseer` from the
+checkout, and only while `mealquest.service` is active: a stack stopped on
+purpose (a database or media restore) is skipped, not redeployed. Output goes
+to the journal (`journalctl -u mealquest-auto-deploy`).
+
+Each run fetches `origin` and compares `origin/main` with the last
+`deployed=` SHA in the deploy record `/home/overseer/mealquest-auto-deploy.log`:
+
+| `origin/main` | Result |
+| --- | --- |
+| is the last deployed SHA | No-op: exit 0, no output. |
+| CI (Gitea commit status) `pending` | Exit 0, no output; the next run checks again. |
+| CI `failure`, `error`, `warning`, or no status yet | Error in the journal, exit 1, nothing deployed. |
+| the last record line is `FAILED` for this SHA | Error in the journal, exit 1, nothing deployed: a failed deploy is never retried by the timer. |
+| CI `success` | Deploy. |
+
+The CI status is read from
+`https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits/<SHA>/status`
+(public; no credential). A commit merged moments before a run can briefly
+report no status, which fails that run; the next run reads it again.
+
+A deploy appends one line to the record:
+
+```text
+2026-10-12T15:04:05Z previous=<rollback SHA> deployed=<SHA> restore_point=2026-10-12T15:01:59Z
+```
+
+`previous` is the rollback SHA (from the running release's `/health`,
+never the checkout) and `restore_point` the verified pre-deploy restore
+point: record both in the release's Maestro task. A failed deploy appends
+`<date> FAILED target=<SHA> status=<exit status>` and logs
+`auto-deploy FAILED (status <n>)` to the journal, adding whether it failed
+before the restart (production untouched) or after it (read
+`journalctl -u mealquest`, then fix or roll back). Either way the checkout is
+left at the target commit; the running release on `/health` stays the
+rollback target.
+
+### Install on the host
+
+The script and units are installed copies: repeat this whenever a release
+changes `deploy/auto-deploy.sh` or the units. From the checkout, at a release
+that contains them:
+
+```sh
+cd /home/overseer/mealquest
+sudo install -m 0755 deploy/auto-deploy.sh /usr/local/sbin/mealquest-auto-deploy
+sudo install -m 0644 deploy/systemd/mealquest-auto-deploy.service \
+  deploy/systemd/mealquest-auto-deploy.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+```
+
+The timer refuses to deploy until the record holds a deploy, so seed it with
+one manual deploy of the release production should run (below), then start
+the timer:
+
+```sh
+sudo /usr/local/sbin/mealquest-auto-deploy --ref refs/remotes/origin/main
+sudo systemctl enable --now mealquest-auto-deploy.timer
+systemctl list-timers mealquest-auto-deploy.timer
+```
+
+To stop automated deploys: `sudo systemctl disable --now mealquest-auto-deploy.timer`.
+
+### Manual override
+
+```sh
+sudo /usr/local/sbin/mealquest-auto-deploy --ref <ref>
+```
+
+deploys an explicit ref (a SHA, tag or `origin/<branch>`) through the same
+blocks, bypassing the CI and no-op gates: for a roll-forward fix, a re-run
+after a `FAILED` record, or seeding the record. Run as root, the script
+re-executes itself as `overseer`; it holds a lock, so it never runs alongside
+the timer's run. It prints `deployed and verified:` and exits 0 only when
+step 3 passed.
+
+### Rollback
+
+Rollback is not automated: follow "Rollback to a previous release" above
+and CTXD `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a
+previous SHA", using the `previous` SHA and `restore_point` from the record.
+The timer does not undo a manual rollback: `origin/main` is still the last
+recorded deploy, so its runs are no-ops until `main` moves, and a new `main`
+then deploys as usual. Disable the timer if the rollback must hold.

deploy/auto-deploy.sh

@@ -0,0 +1,251 @@
+#!/usr/bin/env bash
+# MealQuest automated deploy: deploy/RUNBOOK.md, run unattended.
+#
+#   mealquest-auto-deploy              deploy origin/main if it is new and CI passed
+#   mealquest-auto-deploy --ref <ref>  deploy <ref> now, bypassing the CI and no-op gates
+#
+# Installed as /usr/local/sbin/mealquest-auto-deploy and run as overseer by
+# mealquest-auto-deploy.service (deploy/systemd); install, semantics and the
+# manual override: deploy/RUNBOOK.md "Automated deploys". Run as root (sudo)
+# it re-executes itself as overseer.
+#
+# Without --ref: exit 0 with no output when origin/main is the last deployed
+# SHA in the deploy record or its CI is pending; exit 1, deploying nothing,
+# when its CI failed or reports no status, or when the last attempt to deploy
+# that SHA failed. Every deploy runs the runbook's blocks in order and stops
+# at the first failure; nothing before the restart touches production. A
+# verified deploy appends one line to the deploy record:
+#   <UTC date> previous=<SHA> deployed=<SHA> restore_point=<UTC date>
+# a failed one appends:
+#   <UTC date> FAILED target=<SHA> status=<exit status>
+set -euo pipefail
+
+CHECKOUT=/home/overseer/mealquest
+RECORD=/home/overseer/mealquest-auto-deploy.log
+LOCK=/home/overseer/mealquest-auto-deploy.lock
+STATUS_API=https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits
+PUBLIC=https://mealquest.cubecraftlabs.com
+D=/var/lib/docker/volumes/mealquest-data/_data
+COMPOSE=(docker compose -f deploy/compose.yaml)
+
+fail() { echo "$*" >&2; exit 1; }
+
+if [ "$(id -u)" -eq 0 ]; then
+  exec sudo -u overseer -H -- "$0" "$@"
+fi
+[ "$(id -un)" = overseer ] || fail "run as overseer (or root), not $(id -un)"
+
+ref=
+case "$#" in
+  0) ;;
+  2) [ "$1" = --ref ] && [ -n "$2" ] || fail "usage: $0 [--ref <ref>]"; ref=$2 ;;
+  *) fail "usage: $0 [--ref <ref>]" ;;
+esac
+
+exec 9>> "$LOCK"
+flock -n 9 || fail "another mealquest-auto-deploy run holds $LOCK"
+
+cd "$CHECKOUT"
+git fetch --quiet --tags origin || fail "git fetch origin failed"
+
+if [ -n "$ref" ]; then
+  target=$(git rev-parse --verify --quiet "$ref^{commit}") || fail "$ref is not a commit in $CHECKOUT"
+else
+  # The full remote-tracking ref: a bare origin/main resolves a tag or local
+  # branch of that name first, which could deploy a commit not on main.
+  target=$(git rev-parse --verify --quiet "refs/remotes/origin/main^{commit}") \
+    || fail "refs/remotes/origin/main is not a commit in $CHECKOUT"
+  [ -f "$RECORD" ] || fail "no deploy record at $RECORD: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")"
+  last=$(grep -oE 'deployed=[0-9a-f]{40}' "$RECORD" | tail -n 1 | cut -d = -f 2) || true
+  [ -n "$last" ] || fail "$RECORD records no deploy: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")"
+  [ "$target" != "$last" ] || exit 0
+  if tail -n 1 "$RECORD" | grep -q " FAILED target=$target status="; then
+    fail "auto-deploy of $target already failed (see $RECORD and the journal): fix it, then deploy with --ref"
+  fi
+  state=$(curl -fsS --max-time 30 "$STATUS_API/$target/status" \
+    | python3 -c 'import json, sys; d = json.load(sys.stdin); print(d["state"] if d["total_count"] else "absent")') \
+    || fail "could not read the CI status of $target from $STATUS_API"
+  case "$state" in
+    success) ;;
+    pending) exit 0 ;;
+    *) fail "CI for origin/main $target is $state, not success: not deploying" ;;
+  esac
+fi
+
+echo "auto-deploy: deploying $target${ref:+ (--ref $ref)}"
+restarted=
+W=
+on_exit() {
+  local st=$?
+  [ -z "$W" ] || rm -rf "$W" || st=1
+  if [ "$st" -ne 0 ]; then
+    echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) FAILED target=$target status=$st" >> "$RECORD"
+    if [ -n "$restarted" ]; then
+      echo "auto-deploy FAILED (status $st) after the restart: production may be down or unverified; read journalctl -u mealquest, then fix or roll back (deploy/RUNBOOK.md)" >&2
+    else
+      echo "auto-deploy FAILED (status $st): production untouched" >&2
+    fi
+  fi
+  exit "$st"
+}
+trap on_exit EXIT
+
+# deploy/RUNBOOK.md "Deploy" > "1. Build": record the rollback SHA from the
+# RUNNING release, check out the target, reject a dirty checkout, build.
+bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-) || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
+[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
+running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+  || fail "http://$bind:8090/health did not return a version: no running release to roll back to"
+case "$running" in
+  dev|*-dirty) fail "the running release reports $running, which names no commit" ;;
+esac
+rollback=$(git rev-parse --verify --quiet "$running^{commit}") \
+  || fail "running version $running is not a commit in this repository"
+echo "rollback SHA: $rollback (running version $running)"
+git checkout --quiet --detach "$target"
+dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist')
+[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first:
+$dirty"
+v=$(git describe --tags --always)${dirty:+-dirty}   # the Makefile's VERSION
+make docker
+docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed"
+echo "built: mealquest:$v"
+
+# deploy/RUNBOOK.md "Prerequisites": deploy/.env sets every variable of the
+# target's .env.example, and every secret is mode 0600.
+missing=$(comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \
+                   <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u))
+[ -z "$missing" ] || fail "deploy/.env is missing: $missing"
+loose=$(sudo find deploy/secrets -type f ! -perm 600)
+[ -z "$loose" ] || fail "secrets not mode 0600: $loose"
+
+# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 2: a verified
+# pre-deploy restore point. The scripts are CTXD "Count check". They and the
+# check files live in a private mktemp directory (mode 0700), not at shared
+# /tmp paths, and python runs with -I: the snapshot runs as root, and a
+# shared directory on its import path would let any local user run code as
+# root.
+W=$(mktemp -d)
+cat > "$W/mqcount.py" <<'EOF'
+import sqlite3, sys, hashlib
+c = sqlite3.connect(sys.argv[1])
+tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
+for t in ("dinners", "week_plans", "schema_migrations"):
+    print(t, c.execute(f"select count(*) from {t}").fetchone()[0])
+h = hashlib.sha256()
+for t in tabs:
+    for row in c.execute(f"select * from \"{t}\" order by rowid"):
+        h.update(repr(row).encode())
+print("tables", len(tabs), "content-sha256", h.hexdigest()[:16])
+print("integrity", c.execute("pragma integrity_check").fetchone()[0])
+EOF
+cat > "$W/mqsnap.py" <<'EOF'
+import os, sqlite3, sys
+src, dst = sys.argv[1], sys.argv[2]
+if os.path.exists(dst):
+    sys.exit(f"{dst} already exists")
+s = sqlite3.connect(f"file:{src}?mode=ro", uri=True)
+d = sqlite3.connect(dst)
+s.backup(d)
+s.close()
+result = d.execute("pragma integrity_check").fetchall()
+d.close()
+if result != [("ok",)]:
+    sys.exit(f"snapshot integrity_check failed: {result}")
+print("snapshot", dst, "integrity ok")
+EOF
+point=$(
+set -eu
+point=
+trap 'st=$?
+rm -rf "$W" || st=1
+sudo sh -c "rm -f $D/predeploy-check.db*" || st=1
+if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" >&2; echo "$point"
+else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi
+exit "$st"' EXIT
+exec 3>&1 1>&2   # block output to the journal; only the restore point to $point
+mkdir "$W/mqcheck"
+# a. Replication running, no log errors (grep: 0 match, 1 none, 2+ error), newest LTX listed
+id=$("${COMPOSE[@]}" ps --status running --quiet litestream) || fail "docker compose ps failed"
+[ -n "$id" ] || fail "litestream is not running"
+"${COMPOSE[@]}" logs --since 24h litestream > "$W/mqcheck/litestream.log" || fail "docker compose logs failed"
+gs=0
+grep -i error "$W/mqcheck/litestream.log" > "$W/mqcheck/litestream.errors" || gs=$?
+case "$gs" in
+  0) cat "$W/mqcheck/litestream.errors"; fail "litestream logged errors in the last 24h";;
+  1) ;;
+  *) fail "the litestream log scan failed (status $gs)";;
+esac
+"${COMPOSE[@]}" exec -T litestream \
+  litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
+# b. Restore replica to a scratch file (live files untouched), move off the volume, snapshot live
+"${COMPOSE[@]}" run --rm --no-deps litestream \
+  restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db
+sudo sh -c "mv $D/predeploy-check.db* $W/mqcheck/"
+sudo python3 -I "$W/mqsnap.py" "$D/mealquest.db" "$W/mqcheck/live.db"
+sudo chown -R "$(id -un)" "$W/mqcheck"
+sudo ls -ln "$D"
+# mealquest.db-wal and -shm owned by 10001, never 0 (CTXD "Count check")
+root=$(sudo find "$D" -maxdepth 1 \( -name mealquest.db-wal -o -name mealquest.db-shm \) -uid 0)
+[ -z "$root" ] || fail "owned by root, not 10001 (CTXD RUNBOOKS.MD \"Count check\"): $root"
+# c. Restored replica: integrity ok and identical counts/content-sha256 to the live snapshot
+python3 -I "$W/mqcount.py" "$W/mqcheck/predeploy-check.db" > "$W/mqcheck/restored.txt"
+python3 -I "$W/mqcount.py" "$W/mqcheck/live.db" > "$W/mqcheck/live.txt"
+cat "$W/mqcheck/restored.txt"
+grep -qx 'integrity ok' "$W/mqcheck/restored.txt" || fail "the restored replica failed integrity_check"
+diff "$W/mqcheck/restored.txt" "$W/mqcheck/live.txt" || fail "the restored replica differs from the live snapshot"
+# d. Restore point: after every check, before the restart
+point=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+exec 1>&3
+)
+
+# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 3:
+# boot-required configuration, secrets read the way the service reads them.
+[ -f deploy/.env ] || fail "missing deploy/.env"
+cid=$(grep -m 1 -E '^MEALQUEST_GOOGLE_CLIENT_ID=' deploy/.env | cut -d = -f 2- | tr -d '"' | tr -d "'") \
+  || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env"
+[ -n "$cid" ] || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env"
+[ -e deploy/secrets/google_client_secret ] || fail "missing deploy/secrets/google_client_secret"
+[ -s deploy/secrets/google_client_secret ] || fail "deploy/secrets/google_client_secret is empty"
+id=$("${COMPOSE[@]}" ps --status running --quiet mealquest) || fail "docker compose ps failed"
+[ -n "$id" ] || fail "the mealquest container is not running: start it (the check reads secrets the way the service does)"
+docker exec "$id" sh -c 'test -s /run/secrets/google_client_secret && test -r /run/secrets/google_client_secret' \
+  || fail "google_client_secret is not readable by the service user inside the container"
+echo "boot-required configuration present"
+
+# deploy/RUNBOOK.md "2. Ship and restart".
+"${COMPOSE[@]}" pull litestream searxng
+docker tag "mealquest:$v" mealquest-mealquest
+restarted=yes
+sudo systemctl restart mealquest
+for s in mealquest litestream searxng; do
+  id=$("${COMPOSE[@]}" ps --status running --quiet "$s")
+  [ -n "$id" ] || fail "$s is not running"
+done
+id=$("${COMPOSE[@]}" ps --quiet mealquest)
+health=$(docker inspect -f '{{.State.Health.Status}}' "$id")
+[ "$health" = healthy ] || fail "mealquest is $health, not healthy"
+echo "restarted on: mealquest:$v"
+
+# deploy/RUNBOOK.md "3. Verify the deployed version and a real screen".
+for base in "http://$bind:8090" "$PUBLIC"; do
+  got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
+    || fail "$base/health did not return a version"
+  [ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v"
+  for path in / /plan; do
+    page=$(curl -fsS "$base$path") || fail "$base$path did not return 200"
+    grep -q '<div id="root"></div>' <<< "$page" || fail "$base$path is not the PWA shell"
+    js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true
+    [ -n "$js" ] || fail "$base$path references no built /assets/ script"
+    type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200"
+    case "$type" in
+      text/javascript*) ;;
+      *) fail "$base$js is $type, not JavaScript" ;;
+    esac
+  done
+  echo "$base: version $got, PWA shell and $js served"
+done
+
+deployed=$(git rev-parse HEAD)
+echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) previous=$rollback deployed=$deployed restore_point=$point" >> "$RECORD"
+echo "deployed and verified: $v ($deployed), previous $rollback, restore point $point"

deploy/systemd/mealquest-auto-deploy.service

@@ -0,0 +1,23 @@
+# Deploys origin/main to production when it is new and its CI passed
+# (/usr/local/sbin/mealquest-auto-deploy, from deploy/auto-deploy.sh).
+# Triggered by mealquest-auto-deploy.timer; output goes to the journal.
+# Installing it and the manual override: deploy/RUNBOOK.md "Automated deploys".
+#
+# ExecCondition= skips the run while mealquest.service is not active, so a
+# stack stopped on purpose (a database or media restore) is never redeployed
+# under the operator. It is deliberately not Requires=/Requisite= on
+# mealquest.service: those would start the stack, or stop this run when the
+# deploy itself restarts mealquest.service. Runs as overseer, who owns the
+# checkout and uses sudo for the runbook's privileged steps.
+[Unit]
+Description=MealQuest automated deploy of main
+Wants=network-online.target
+After=network-online.target docker.service mealquest.service
+
+[Service]
+Type=oneshot
+User=overseer
+Group=overseer
+WorkingDirectory=/home/overseer/mealquest
+ExecCondition=/usr/bin/systemctl is-active --quiet mealquest.service
+ExecStart=/usr/local/sbin/mealquest-auto-deploy

deploy/systemd/mealquest-auto-deploy.timer

@@ -0,0 +1,13 @@
+# Checks for a new main at every quarter hour (plus up to 5 minutes' random
+# delay). Persistent= runs a check at once when the host comes back if a
+# scheduled one was missed while it was down.
+[Unit]
+Description=Periodic MealQuest automated deploy
+
+[Timer]
+OnCalendar=*-*-* *:0/15:00
+RandomizedDelaySec=5min
+Persistent=true
+
+[Install]
+WantedBy=timers.target

docs/ARCHITECTURE.md

@@ -2,10 +2,8 @@
 
 This mirrors the architecture section of `CONTEXT.MD` in the CTXD project
 `mealquest`, which is the source of record; if the two disagree, CTXD wins and
-this file should be updated. It describes the **planned** application. The
-repository today is the Feature 01 scaffold: `cmd/server`, `internal/config`,
-`internal/server` (`/health`), `internal/frontend` (embedded PWA with SPA
-fallback), the `web/` skeleton, `Makefile`, `Dockerfile` and CI.
+this file should be updated. It describes the application as built; nothing
+it names is still only planned.
 
 Decision numbers (D-NNN) refer to `DECISIONS.MD` in CTXD.
 
@@ -46,20 +44,22 @@ headers, so sign-in works whichever peer the request arrives from.
 | `cmd/vapidgen` | Prints a new VAPID key pair for Web Push. |
 | `internal/config` | Env-only `MEALQUEST_*` config; secrets from `*_FILE`. |
 | `internal/server` | HTTP server, router, middleware. |
+| `internal/forwarded` | Client IP and scheme from `X-Forwarded-*`, trusted only from `MEALQUEST_TRUSTED_PROXY`. |
+| `internal/secureurl` | The transport rule for URLs that carry credentials (`secureurl.Check`). |
 | `internal/frontend` | Embedded PWA, SPA fallback. |
 | `internal/auth` | Authentik OIDC, sessions, CSRF. |
-| `internal/db`, `internal/store` | SQLite connections, sqlc queries. |
+| `internal/db`, `internal/store` | sqlc-generated queries; SQLite connections, migrations at boot, transactions. |
 | `internal/secretbox` | AES-256-GCM encryption for `_ct` columns. |
-| `internal/household`, `internal/dinners`, `internal/recipes` | Domain. |
-| `internal/ai/llm`, `internal/ai/harness`, `internal/ai/skills` | AI provider adapters, skill harness, skills. |
-| `internal/suggest`, `internal/plan`, `internal/shopping`, `internal/ideas` | Feature services. |
+| `internal/household`, `internal/dinners`, `internal/recipes` | Domain: household settings and local time, dinners (grounding, photo backfill), recipe search, fetch and grounding. |
+| `internal/ai/llm`, `internal/ai/harness`, `internal/ai/skills`, `internal/ai/aisettings` | AI provider adapters, skill harness, skills, AI settings. |
+| `internal/suggest`, `internal/plan`, `internal/shopping`, `internal/ideas`, `internal/home` | Feature services. |
 | `internal/gcal`, `internal/notify`, `internal/push`, `internal/chat` | Integrations. |
 | `internal/vapid` | VAPID key generation and validation. |
 | `internal/jobs`, `internal/scheduler` | Background jobs and schedules. |
-| `migrations/` | SQL migrations (no goose or golang-migrate). |
+| `migrations/` | SQL migrations, embedded and applied at boot by `internal/store` (no goose or golang-migrate); sqlc queries in `migrations/queries/`. |
 | `web/` | React PWA. |
-| `deploy/` | Deployment files. |
-| `docs/` | Documentation. |
+| `deploy/` | Production compose stack, systemd units, the [deploy runbook](../deploy/RUNBOOK.md) and the automated deploy (see Deployment). |
+| `docs/` | Documentation (this file). |
 | `design/` | Requirements and UI handoff (read-only reference). |
 
 ## Backend stack and conventions
@@ -276,6 +276,12 @@ automatic** (D-021).
   `grounded`, `failed` (with its reason) or `skipped`. A dinner whose write
   fails is `failed` with `grounding_error` and left as it was; the others
   still run, then the job fails with `internal_error`, keeping that result.
+  The responses returning a `dinners.Dinner` representation (the list; the
+  create, update, archive, favorite and grounding responses; the idea
+  responses) carry `photo_backfill_eligible`, and
+  `GET /api/v1/dinners/{id}` does not; it says whether the backfill grounds
+  that dinner, computed by the same condition it lists dinners by, so a
+  client offers a search only where one runs.
   `GET /api/v1/dinners/{id}/recipe-candidates`
   searches for the dinner's name and returns `{candidates: [{candidate_id,
   title, site_name, photo_url, minutes}]}`, at most 3, ranked as `Ground`
@@ -805,6 +811,20 @@ purge are fixed and not configurable.
 
 Runs are recorded in `schedule_runs` and executed through `jobs`.
 
+Job kinds (`internal/jobs`; each registered when `cmd/server` builds the
+service named, before the runner starts). `POST /api/v1/jobs` refuses a
+service-only kind; `GET /api/v1/jobs/{id}` reports any job.
+
+| Kind | Registered by | Enqueued by | Service-only |
+| --- | --- | --- | --- |
+| `noop` | `jobs.RegisterBuiltins` | `POST /api/v1/jobs` (does nothing; proves the runner works) | no |
+| `ground_dinner` | `dinners.Service` | Adding or renaming a `pending` dinner | no |
+| `backfill_dinner_photos` | `dinners.Service` | `POST /api/v1/settings/photo-backfill` | yes |
+| `suggestion_batch` | `suggest.Service` | `POST /api/v1/suggestions/batches`, a deck decision (replenish), `suggestions_weekly` | no |
+| `draft_week` | `plan.Service` | `POST /api/v1/plans/{weekStart}/draft`, `plan_draft` | yes |
+| `title_supply_run` | `shopping.Service` | Generating an untitled Supply run | yes |
+| `ideas_from_ingredients`, `quick_ideas` | `ideas.Service` | `POST /api/v1/ideas/from-ingredients`, `POST /api/v1/ideas/quick` | yes |
+
 `internal/scheduler` runs them. A schedule registers a name, a rule read
 from the settings (`scheduler.Daily`, `Weekly` or `WeeklyForNextWeek`, or `DailyAt` for a time read from data) and a
 run function; settings changes reach it through `SettingsService.Subscribe`
@@ -1313,10 +1333,12 @@ fewer the server answers 400 `invalid_settings` naming
 `onboarding_completed` and changes nothing. Step 3 can be reached with
 fewer — by its URL, or after another member removes a Regular — so it then
 shows the server's reason with an "Add Regulars" button back to step 2.
-Notification permission is never asked for on page load; until Feature 19
-wires Web Push, "Turn on notifications" asks for nothing and says plainly
-that notifications are not on yet, offering "Finish setting up" (which
-completes onboarding as above) instead of a silent success. Offline on the
+Notification permission is never asked for on page load. "Turn on
+notifications" subscribes this device to Web Push from the tap and says
+how it ended — on, blocked by the browser, or unsupported (on iPhone and
+iPad, until MealQuest is on the Home Screen) — then offers only "Finish
+setting up" (which completes onboarding as above); a dismissed permission
+prompt can be asked again. Offline on the
 saved pages the settings cannot be read (the service worker never caches `/api/v1/settings`),
 so the saved pages open as they are.
 
@@ -1333,6 +1355,41 @@ are undone if the server refuses them. While any Regular is `pending`, the
 list is read again every 3 seconds (not while a change is in flight), and
 polling stops once none is.
 
+## Deployment
+
+Production is the compose project `mealquest` on 10.60.1.14
+([`deploy/compose.yaml`](../deploy/compose.yaml)): the `mealquest` image built
+from the root `Dockerfile`, `litestream` replicating the database to the
+NAS over SFTP, and `searxng` on the internal network only. Configuration is
+`deploy/.env` and file secrets in `deploy/secrets/`, neither committed. The
+services never restart themselves; systemd units in `deploy/systemd/` own
+the host:
+
+| Unit | Does |
+| --- | --- |
+| `mealquest-firewall.service` | Restricts port 8090 to Traefik in `DOCKER-USER`; required before the stack starts. |
+| `mealquest.service` | Starts and stops the compose stack. |
+| `mealquest-media-backup.timer` / `.service` | Backs up cached photos to the NAS nightly at 03:30. |
+| `mealquest-auto-deploy.timer` / `.service` | Automated deploy of `main` (below). |
+
+Releases: only `overseer` merges `dev` into `main`. The image carries its
+version (`git describe`: a release tag, else the short SHA), which
+`/health` reports. A deploy follows
+[`deploy/RUNBOOK.md`](../deploy/RUNBOOK.md) — build, pre-deploy restore
+point, boot-config check, ship and restart, verify the deployed SHA and a
+real screen — with rollback to the previous release. The automated deploy
+([`deploy/auto-deploy.sh`](../deploy/auto-deploy.sh), installed as
+`/usr/local/sbin/mealquest-auto-deploy`) runs those same steps unattended:
+its timer checks every quarter hour (plus up to 5 minutes' random delay),
+and deploys `refs/remotes/origin/main` only when it differs from the last
+SHA in the deploy record `/home/overseer/mealquest-auto-deploy.log`, its
+Gitea CI status is `success`, its last attempt did not fail, and
+`mealquest.service` is active. It stops at the first failure and never
+retries a failed SHA; `--ref <ref>` is the manual override. Installing,
+seeding the record and enabling the timer are host steps in the runbook's
+"Automated deploys". Database restore and the boot configuration are in
+CTXD `RUNBOOKS.MD`.
+
 ## Boundaries and gotchas
 
 - **Secrets never reach the client** — API keys, OAuth tokens and the Chat

internal/dinners/backfill.go

@@ -25,6 +25,17 @@ const (
 	BackfillSkipped = "skipped"
 )
 
+// PhotoBackfillEligible reports whether a photo backfill grounds r: the
+// condition ListPhotoBackfillDinners lists dinners by, and GroundFailedDinner
+// and RefailDinnerGrounding write under, for one dinner. The dinner is
+// unarchived, failed, with no photo and no pasted recipe_url.
+func PhotoBackfillEligible(r db.Dinner) bool {
+	return r.Status != StatusArchived &&
+		r.GroundingStatus == GroundingFailed &&
+		!r.PhotoID.Valid &&
+		!r.RecipeUrl.Valid
+}
+
 // BackfillJob is the KindBackfillPhotos payload. DinnerID narrows the
 // backfill to that dinner; empty backfills every dinner without a photo.
 type BackfillJob struct {

internal/dinners/backfill_test.go

@@ -192,6 +192,53 @@ func TestBackfillSkipsDinnersNotMissingAPhoto(t *testing.T) {
 	}
 }
 
+// TestPhotoBackfillEligibleIsWhatTheBackfillLists holds the API's
+// photo_backfill_eligible to ListPhotoBackfillDinners for a dinner the
+// backfill grounds and for each kind it never does.
+func TestPhotoBackfillEligibleIsWhatTheBackfillLists(t *testing.T) {
+	t.Parallel()
+	e := newEnv(t)
+	e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
+	grounded, err := e.svc.CreateRegular(t.Context(), dinners.NewRegular{Name: ptr("Chicken curry")}, member)
+	if err != nil {
+		t.Fatal(err)
+	}
+	e.waitJob(t, *grounded.GroundingJobID)
+	linked, err := e.svc.CreateRegular(t.Context(), dinners.NewRegular{Name: ptr("Mystery pie"), RecipeURL: ptr("https://d.example/pie")}, member)
+	if err != nil {
+		t.Fatal(err)
+	}
+	e.waitJob(t, *linked.GroundingJobID)
+	failed := e.failedDinner(t, "Abalone surprise")
+	archived, err := e.svc.Archive(t.Context(), e.failedDinner(t, "Old stew"), member)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	shown := map[string]dinners.Dinner{archived.ID: archived}
+	regulars, err := e.svc.List(t.Context(), dinners.StatusRegular)
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, d := range regulars {
+		shown[d.ID] = d
+	}
+	q := db.New(e.st.Reader)
+	for id, want := range map[string]bool{grounded.ID: false, linked.ID: false, failed: true, archived.ID: false} {
+		listed, err := q.ListPhotoBackfillDinners(t.Context(), sql.NullString{String: id, Valid: true})
+		if err != nil {
+			t.Fatal(err)
+		}
+		d, ok := shown[id]
+		if !ok {
+			t.Fatalf("dinner %s not shown", id)
+		}
+		if got := len(listed) == 1; got != want || d.PhotoBackfillEligible != want {
+			t.Errorf("%s: listed %t, photo_backfill_eligible %t, want both %t", d.Name, got, d.PhotoBackfillEligible, want)
+		}
+	}
+}
+
 func TestBackfillOfOneDinnerGroundsOnlyIt(t *testing.T) {
 	t.Parallel()
 	e := newUnstartedEnv(t)

internal/dinners/dinners.go

@@ -79,6 +79,11 @@ type Dinner struct {
 	UpdatedBy      *string `json:"updated_by"`
 	CreatedAt      string  `json:"created_at"`
 	UpdatedAt      string  `json:"updated_at"`
+
+	// PhotoBackfillEligible is whether a photo backfill grounds the dinner
+	// (PhotoBackfillEligible): false for one with a pasted recipe link,
+	// which a search never replaces.
+	PhotoBackfillEligible bool `json:"photo_backfill_eligible"`
 }
 
 // NewRegular is a Regular to add: its Name, which the household always
@@ -712,6 +717,8 @@ func fromRow(r db.Dinner) Dinner {
 		UpdatedBy:       nullString(r.UpdatedBy),
 		CreatedAt:       r.CreatedAt,
 		UpdatedAt:       r.UpdatedAt,
+
+		PhotoBackfillEligible: PhotoBackfillEligible(r),
 	}
 	if r.MinutesTotal.Valid {
 		d.MinutesTotal = &r.MinutesTotal.Int64

web/src/dinners/DinnerDetailActions.test.tsx

@@ -52,6 +52,8 @@ function listed(is_favorite: boolean) {
     is_favorite,
     photo_url: null,
     grounding_status: 'grounded',
+    grounding_error: null,
+    photo_backfill_eligible: false,
   }
 }
 

web/src/dinners/api.ts

@@ -20,8 +20,12 @@ export type Difficulty = (typeof DIFFICULTIES)[number]
 export type GroundingStatus = (typeof GROUNDING_STATUSES)[number]
 
 // Regular is the part of a dinner the Regulars screens read. photo_id is
-// the stored photo's id, from photo_url. pending is true while an
-// optimistic add waits for the server; its id is not the server's.
+// the stored photo's id, from photo_url. grounding_error is why its
+// grounding failed, a reason code, set only while grounding_status is
+// failed. photo_backfill_eligible is whether a photo backfill (Search
+// again) searches for it: never for a dinner with a pasted recipe link.
+// pending is true while an optimistic add waits for the server;
+// its id is not the server's.
 export type Regular = {
   id: string
   name: string
@@ -31,6 +35,8 @@ export type Regular = {
   is_favorite: boolean
   photo_id: string | undefined
   grounding_status: GroundingStatus
+  grounding_error: string | null
+  photo_backfill_eligible: boolean
   pending?: true
 }
 
@@ -113,6 +119,9 @@ export function parseDinner(where: string, body: unknown): Regular {
     throw new Error(`${where}: response is not a dinner`)
   }
   if (typeof body.is_favorite !== 'boolean') throw new Error(`${where}: is_favorite is not a boolean`)
+  if (typeof body.photo_backfill_eligible !== 'boolean') {
+    throw new Error(`${where}: photo_backfill_eligible is not a boolean`)
+  }
   return {
     id: body.id,
     name: body.name,
@@ -122,6 +131,11 @@ export function parseDinner(where: string, body: unknown): Regular {
     is_favorite: body.is_favorite,
     photo_id: photoId(where, body.photo_url),
     grounding_status: oneOf(where, 'grounding_status', GROUNDING_STATUSES, body.grounding_status),
+    grounding_error: nullable(body.grounding_error, (v) => {
+      if (typeof v !== 'string') throw new Error(`${where}: grounding_error is not a string`)
+      return v
+    }),
+    photo_backfill_eligible: body.photo_backfill_eligible,
   }
 }
 
@@ -225,6 +239,8 @@ export function useAddRegular() {
         is_favorite: false,
         photo_id: undefined,
         grounding_status: 'pending',
+        grounding_error: null,
+        photo_backfill_eligible: false,
         pending: true,
       }
       await changeOptimistically(queryClient, (list) => [...list, optimistic])

web/src/plan/PlanPage.test.tsx

@@ -114,6 +114,8 @@ function dinner(id: string, name: string, difficulty: string | null, minutes: nu
     is_favorite: false,
     photo_url: null,
     grounding_status: 'grounded',
+    grounding_error: null,
+    photo_backfill_eligible: false,
   }
 }
 

web/src/settings/RegularsSettingsPage.module.css

@@ -300,3 +300,27 @@
 .lock {
   display: contents;
 }
+
+/* Find missing photos: its hint, button and progress, padded like a sheet. */
+.card.backfill {
+  display: flex;
+  flex-direction: column;
+  gap: 12px;
+  padding: 16px;
+}
+
+/* Why a Regular's recipe was not found. */
+.reason {
+  margin: 0;
+  padding: 12px 14px;
+  border-radius: 18px;
+  background: var(--color-neutral-100);
+  color: var(--color-neutral-900);
+  font-family: var(--font-body);
+  font-size: 14px;
+  line-height: 1.4;
+}
+
+.reasonLabel {
+  font-weight: 700;
+}

web/src/settings/RegularsSettingsPage.test.tsx

@@ -85,7 +85,11 @@ const TACOS = dinnerBody('d1', 'Beef tacos', {
   photo_url: '/media/photos/p1-400.jpg',
 })
 const CHILI = dinnerBody('d2', 'Chili', { method: 'crockpot', minutes_total: 480, grounding_status: 'pending' })
-const PIZZA = dinnerBody('d3', 'Pizza night', { grounding_status: 'failed', grounding_error: 'no_recipe_found' })
+const PIZZA = dinnerBody('d3', 'Pizza night', {
+  grounding_status: 'failed',
+  grounding_error: 'no_recipe_found',
+  photo_backfill_eligible: true,
+})
 const ROAST = dinnerBody('d4', 'Pot roast', { method: 'oven', grounding_status: 'manual' })
 
 function rows() {
@@ -496,6 +500,204 @@ describe('RegularsSettingsPage', () => {
     })
   })
 
+  describe('Photo backfill', () => {
+    const BACKFILL = 'POST /api/v1/settings/photo-backfill'
+    const JOB = 'GET /api/v1/jobs/j9'
+
+    // backfillJob is a GET /api/v1/jobs/j9 answer for the photo backfill.
+    function backfillJob(status: string, result: unknown = null, error: string | null = null) {
+      return jsonResponse(200, { id: 'j9', kind: 'backfill_dinner_photos', status, result, error })
+    }
+
+    // polls answers each poll of the job with the next answer, then the last
+    // one again; an answer may change the server first.
+    function polls(...answers: (() => Response)[]) {
+      let next = 0
+      return () => answers[Math.min(next++, answers.length - 1)]!()
+    }
+
+    function fixSheet() {
+      return within(screen.getByRole('dialog', { name: 'Fix recipe' }))
+    }
+
+    const NO_CANDIDATES = { 'GET /api/v1/dinners/d3/recipe-candidates': () => jsonResponse(200, { candidates: [] }) }
+
+    it('shows why the recipe was not found in the fix-recipe sheet', async () => {
+      stubServer([PIZZA], NO_CANDIDATES)
+      await renderPage()
+
+      openRow('Pizza night')
+      expect(fixSheet().getByText('No recipe page the search found had a recipe we could use.')).toBeInTheDocument()
+      expect(fixSheet().getByRole('button', { name: 'Search again' })).toBeEnabled()
+    })
+
+    it('offers no search again for a pasted link that had no usable recipe', async () => {
+      const linked = { ...PIZZA, grounding_error: 'recipe_unusable', photo_backfill_eligible: false }
+      stubServer([linked], NO_CANDIDATES)
+      await renderPage()
+
+      openRow('Pizza night')
+      expect(fixSheet().getByText('The pasted recipe link has no recipe we could use.')).toBeInTheDocument()
+      expect(fixSheet().queryByRole('button', { name: 'Search again' })).not.toBeInTheDocument()
+    })
+
+    it('offers no search again for a dinner with a pasted link whatever its reason', async () => {
+      const linked = { ...PIZZA, grounding_error: 'ai_failed', photo_backfill_eligible: false }
+      stubServer([linked], NO_CANDIDATES)
+      await renderPage()
+
+      openRow('Pizza night')
+      expect(fixSheet().getByText("The AI assistant couldn't read the recipe.")).toBeInTheDocument()
+      expect(fixSheet().queryByRole('button', { name: 'Search again' })).not.toBeInTheDocument()
+    })
+
+    it('searches again from the fix-recipe sheet and shows the new reason when it still fails', async () => {
+      const server = stubServer([PIZZA], {
+        ...NO_CANDIDATES,
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: polls(
+          () => backfillJob('running'),
+          () => {
+            server.regulars = [{ ...PIZZA, grounding_error: 'search_failed' }]
+            return backfillJob('succeeded', { dinners: [{ dinner_id: 'd3', outcome: 'failed', reason: 'search_failed' }] })
+          },
+        ),
+      })
+      await renderPage()
+
+      openRow('Pizza night')
+      fireEvent.click(fixSheet().getByRole('button', { name: 'Search again' }))
+      expect(await fixSheet().findByRole('button', { name: 'Searching again…' })).toBeDisabled()
+      expect(fixSheet().getByText('Searching again… this can take a few minutes.')).toBeInTheDocument()
+      expect(sent(server.fetchMock)).toEqual([{ request: BACKFILL, body: undefined }])
+
+      expect(await fixSheet().findByText('Searched again: still no recipe found.', {}, { timeout: 3000 })).toBeInTheDocument()
+      expect(await fixSheet().findByText("The recipe search couldn't be reached.")).toBeInTheDocument()
+      expect(fixSheet().getByRole('button', { name: 'Search again' })).toBeEnabled()
+    })
+
+    it('closes the fix-recipe sheet once searching again finds the recipe and its photo', async () => {
+      const server = stubServer([PIZZA], {
+        ...NO_CANDIDATES,
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: () => {
+          server.regulars = [{ ...PIZZA, grounding_status: 'grounded', grounding_error: null, photo_url: '/media/photos/p8-400.jpg' }]
+          return backfillJob('succeeded', { dinners: [{ dinner_id: 'd3', outcome: 'grounded', reason: null }] })
+        },
+      })
+      await renderPage()
+
+      openRow('Pizza night')
+      fireEvent.click(fixSheet().getByRole('button', { name: 'Search again' }))
+      await waitFor(() => expect(screen.queryByRole('dialog')).not.toBeInTheDocument())
+      expect(within(row('Pizza night')).getByText('Recipe found')).toBeInTheDocument()
+      expect(within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })).toHaveAttribute('src', '/media/photos/p8-400.jpg')
+    })
+
+    it('says when a dinner was not included in the search', async () => {
+      stubServer([PIZZA], {
+        ...NO_CANDIDATES,
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: () => backfillJob('succeeded', { dinners: [] }),
+      })
+      await renderPage()
+
+      openRow('Pizza night')
+      fireEvent.click(fixSheet().getByRole('button', { name: 'Search again' }))
+      expect(await fixSheet().findByText('This dinner was not included in that search.')).toBeInTheDocument()
+    })
+
+    it('finds missing photos: the placeholder shows until the backfill finds a photo', async () => {
+      const found = { ...PIZZA, grounding_status: 'grounded', grounding_error: null, photo_url: '/media/photos/p8-400.jpg' }
+      const server = stubServer([TACOS, PIZZA], {
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: polls(
+          () => backfillJob('queued'),
+          () => {
+            server.regulars = [TACOS, found]
+            return backfillJob('succeeded', { dinners: [{ dinner_id: 'd3', outcome: 'grounded', reason: null }] })
+          },
+        ),
+      })
+      await renderPage()
+
+      // The photoless dinner shows the striped placeholder, not an image.
+      const placeholder = within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })
+      expect(placeholder).not.toHaveAttribute('src')
+      expect(placeholder).toHaveTextContent('dish photo')
+
+      const trigger = screen.getByRole('button', { name: 'Find missing photos' })
+      expectMinHitArea(trigger)
+      fireEvent.click(trigger)
+      expect(await screen.findByRole('button', { name: 'Finding photos…' })).toBeDisabled()
+      expect(await screen.findByText('Finding missing photos… this can take a few minutes.')).toBeInTheDocument()
+      expect(within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })).not.toHaveAttribute('src')
+      expect(sent(server.fetchMock)).toEqual([{ request: BACKFILL, body: undefined }])
+
+      expect(await screen.findByText('Photo search done: 1 photo found.', {}, { timeout: 3000 })).toBeInTheDocument()
+      await waitFor(() =>
+        expect(within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })).toHaveAttribute('src', '/media/photos/p8-400.jpg'),
+      )
+      expect(screen.getByRole('button', { name: 'Find missing photos' })).toBeEnabled()
+      expect(reads(server.fetchMock)).toBe(2)
+    })
+
+    it('sums up what the backfill did', async () => {
+      stubServer([PIZZA], {
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: () =>
+          backfillJob('succeeded', {
+            dinners: [
+              { dinner_id: 'd3', outcome: 'grounded', reason: null },
+              { dinner_id: 'd5', outcome: 'grounded', reason: null },
+              { dinner_id: 'd6', outcome: 'failed', reason: 'no_recipe_found' },
+              { dinner_id: 'd7', outcome: 'skipped', reason: null },
+            ],
+          }),
+      })
+      await renderPage()
+
+      fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
+      expect(
+        await screen.findByText(
+          'Photo search done: 2 photos found · 1 dinner still needs a recipe · 1 dinner changed meanwhile, left as it was.',
+        ),
+      ).toBeInTheDocument()
+    })
+
+    it('says when no dinner was missing a photo', async () => {
+      stubServer([TACOS], {
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: () => backfillJob('succeeded', { dinners: [] }),
+      })
+      await renderPage()
+
+      fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
+      expect(await screen.findByText('No dinners were missing a photo.')).toBeInTheDocument()
+    })
+
+    it('says why the backfill stopped', async () => {
+      stubServer([PIZZA], {
+        [BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
+        [JOB]: () => backfillJob('failed', { dinners: [] }, 'internal_error'),
+      })
+      await renderPage()
+
+      fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
+      expect(await screen.findByRole('alert')).toHaveTextContent('Photo search stopped: Something went wrong on the server.')
+      expect(screen.getByRole('button', { name: 'Find missing photos' })).toBeEnabled()
+    })
+
+    it('says why the backfill could not start', async () => {
+      stubServer([PIZZA], { [BACKFILL]: () => errorResponse(403, 'csrf_failed') })
+      await renderPage()
+
+      fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
+      expect(await screen.findByRole('alert')).toHaveTextContent('Photo search not started: csrf_failed message')
+      expect(screen.getByRole('button', { name: 'Find missing photos' })).toBeEnabled()
+    })
+  })
+
   it('refreshes grounding status while any Regular is finding its recipe, then stops', async () => {
     // Only intervals are faked: TanStack Query polls with setInterval, and
     // Testing Library's waits keep their real timers.

web/src/settings/RegularsSettingsPage.tsx

@@ -34,6 +34,7 @@ import { Sheet } from '../ui/Sheet.tsx'
 import { Skeleton } from '../ui/Skeleton.tsx'
 import { Tag } from '../ui/Tag.tsx'
 import { TextInput } from '../ui/TextInput.tsx'
+import { usePhotoBackfill, type BackfillResult, type PhotoBackfill } from './photoBackfill.ts'
 import styles from './RegularsSettingsPage.module.css'
 
 // The most minutes a dinner may take, as the dinners API allows: a day.
@@ -57,7 +58,8 @@ const LINK = /^https?:\/\//i
 // method and minutes, favorite heart and whether its recipe was found. The
 // field at the top adds one by name, or by pasted recipe link with a name
 // the household gives it; tapping one
-// edits or removes it, or fixes its recipe when none was found.
+// edits or removes it, or fixes its recipe when none was found. Find
+// missing photos searches again for every dinner left without one.
 export function RegularsSettingsPage() {
   const navigate = useNavigate()
   return (
@@ -86,6 +88,7 @@ function RegularsManager() {
   const remove = useRemoveRegular()
   const favorite = useToggleFavorite()
   const { refusals, track } = useRefusals()
+  const backfill = usePhotoBackfill()
   const [draft, setDraft] = useState('')
   const [linkName, setLinkName] = useState('')
   const [open, setOpen] = useState<Open>(null)
@@ -94,8 +97,10 @@ function RegularsManager() {
   const isLink = LINK.test(draft.trim())
   const ready = draft.trim() !== '' && (!isLink || linkName.trim() !== '')
   const list = regulars.data ?? []
-  // A Regular removed meanwhile closes its sheet.
+  // A Regular removed meanwhile closes its sheet, and one whose recipe was
+  // found meanwhile closes its fix-recipe sheet.
   const opened = open && list.find((regular) => regular.id === open.id && !regular.pending)
+  const fixing = opened && open.sheet === 'fix' && opened.grounding_status === 'failed' ? opened : undefined
 
   function onSubmit(event: FormEvent<HTMLFormElement>) {
     event.preventDefault()
@@ -188,6 +193,7 @@ function RegularsManager() {
           )}
         </section>
       )}
+      {regulars.isSuccess && <FindMissingPhotos backfill={backfill} />}
 
       {opened && open.sheet === 'edit' && (
         <EditRegularSheet
@@ -197,12 +203,13 @@ function RegularsManager() {
           onRemove={() => onRemove(opened)}
         />
       )}
-      {opened && open.sheet === 'fix' && (
+      {fixing && (
         <FixRecipeSheet
-          key={opened.id}
-          regular={opened}
+          key={fixing.id}
+          regular={fixing}
+          backfill={backfill}
           onClose={() => setOpen(null)}
-          onEdit={() => setOpen({ sheet: 'edit', id: opened.id })}
+          onEdit={() => setOpen({ sheet: 'edit', id: fixing.id })}
         />
       )}
     </>
@@ -231,6 +238,88 @@ function GroundingBadge({ status }: { status: GroundingStatus }) {
   return <Tag className={`${styles.badge} ${badge.className}`}>{badge.text}</Tag>
 }
 
+// groundingReasons says why a dinner's recipe was not found, by its
+// grounding_error reason code.
+const groundingReasons: Record<string, string> = {
+  no_recipe_found: 'No recipe page the search found had a recipe we could use.',
+  recipe_unusable: 'The pasted recipe link has no recipe we could use.',
+  search_not_configured: 'Recipe search is not set up. An admin can set it up in Settings.',
+  search_failed: "The recipe search couldn't be reached.",
+  ai_failed: "The AI assistant couldn't read the recipe.",
+  interrupted: 'The server stopped while finding the recipe.',
+  grounding_error: 'Something went wrong on the server.',
+}
+
+// groundingReason says why a dinner's recipe was not found; an unknown
+// reason code is shown as it is.
+export function groundingReason(code: string): string {
+  return groundingReasons[code] ?? code
+}
+
+// backfillJobFailures says why a photo backfill job failed, by its failure
+// code; an unknown code is shown as it is.
+const backfillJobFailures: Record<string, string> = {
+  internal_error: 'Something went wrong on the server.',
+}
+
+// backfillProblem says what went wrong with a photo backfill, or null.
+function backfillProblem(backfill: PhotoBackfill): string | null {
+  if (backfill.startError) return `Photo search not started: ${backfill.startError.message}`
+  if (backfill.status === 'failed' && backfill.error) {
+    return `Photo search stopped: ${backfillJobFailures[backfill.error.message] ?? backfill.error.message}`
+  }
+  if (backfill.error) return `Couldn't check on the photo search: ${backfill.error.message}`
+  return null
+}
+
+function count(n: number, one: string, many: string): string {
+  return `${n} ${n === 1 ? one : many}`
+}
+
+// backfillSummary says what a finished photo backfill did.
+function backfillSummary(result: BackfillResult): string {
+  const tally = (outcome: string) => result.dinners.filter((dinner) => dinner.outcome === outcome).length
+  if (result.dinners.length === 0) return 'No dinners were missing a photo.'
+  const parts = [count(tally('grounded'), 'photo found', 'photos found')]
+  const failed = tally('failed')
+  if (failed > 0) parts.push(`${count(failed, 'dinner still needs', 'dinners still need')} a recipe`)
+  const skipped = tally('skipped')
+  if (skipped > 0) parts.push(`${count(skipped, 'dinner', 'dinners')} changed meanwhile, left as it was`)
+  return `Photo search done: ${parts.join(' · ')}.`
+}
+
+// FindMissingPhotos searches again, by name, for a recipe and photo for
+// every dinner left without a photo because its recipe was not found.
+function FindMissingPhotos({ backfill }: { backfill: PhotoBackfill }) {
+  const headingId = useId()
+  const problem = backfillProblem(backfill)
+  return (
+    <section aria-labelledby={headingId} className={styles.section}>
+      <h2 id={headingId} className={styles.groupLabel}>
+        Missing photos
+      </h2>
+      <Card className={`${styles.card} ${styles.backfill}`}>
+        <p className={styles.hint}>
+          Search again for a recipe and photo for every dinner whose recipe wasn&apos;t found. Dinners with a pasted
+          recipe link keep it.
+        </p>
+        <Button disabled={backfill.active} onClick={backfill.start}>
+          {backfill.active ? 'Finding photos…' : 'Find missing photos'}
+        </Button>
+        <p role="status" className={styles.status}>
+          {backfill.active && 'Finding missing photos… this can take a few minutes.'}
+          {backfill.status === 'succeeded' && backfill.result && backfillSummary(backfill.result)}
+        </p>
+        {problem && (
+          <p role="alert" className={styles.problem}>
+            {problem}
+          </p>
+        )}
+      </Card>
+    </section>
+  )
+}
+
 type RegularRowProps = {
   regular: Regular
   onOpen: () => void
@@ -447,6 +536,7 @@ const fixPaths = [
 
 type FixRecipeSheetProps = {
   regular: Regular
+  backfill: PhotoBackfill
   onClose: () => void
   onEdit: () => void
 }
@@ -459,10 +549,20 @@ function lines(text: string): string[] {
     .filter((line) => line !== '')
 }
 
-// FixRecipeSheet grounds a Regular whose recipe was not found: in one of
-// up to three recipe pages found for its name, in a pasted recipe link, or
-// in ingredients and steps typed in.
-function FixRecipeSheet({ regular, onClose, onEdit }: FixRecipeSheetProps) {
+// retryOutcomes says what searching again did for this dinner.
+const retryOutcomes = {
+  grounded: 'Searched again: a recipe and photo were found.',
+  failed: 'Searched again: still no recipe found.',
+  skipped: 'It changed while it was searched again, so it was left as it is.',
+} as const
+
+// FixRecipeSheet shows why a Regular's recipe was not found and grounds it:
+// by searching its name again (a photo backfill), in one of up to three
+// recipe pages found for its name, in a pasted recipe link, or in
+// ingredients and steps typed in. Searching again is offered only for a
+// dinner the backfill searches (photo_backfill_eligible): it never replaces
+// a pasted link.
+function FixRecipeSheet({ regular, backfill, onClose, onEdit }: FixRecipeSheetProps) {
   const [path, setPath] = useState<FixPath>('pick')
   const ground = useGroundRegular()
 
@@ -475,6 +575,15 @@ function FixRecipeSheet({ regular, onClose, onEdit }: FixRecipeSheetProps) {
       <p className={styles.hint}>
         We couldn&apos;t find a recipe for {regular.name}. Pick a match below, paste a recipe link or type it in.
       </p>
+      {regular.grounding_error !== null && (
+        <p className={styles.reason}>
+          <span className={styles.reasonLabel}>Why: </span>
+          {groundingReason(regular.grounding_error)}
+        </p>
+      )}
+      {regular.photo_backfill_eligible && (
+        <RetrySearch id={regular.id} backfill={backfill} disabled={ground.isPending} />
+      )}
       <Segmented
         label="How to fix the recipe"
         options={fixPaths}
@@ -504,6 +613,34 @@ function FixRecipeSheet({ regular, onClose, onEdit }: FixRecipeSheetProps) {
   )
 }
 
+// RetrySearch searches the dinner's name again with a photo backfill, and
+// says what it did for the dinner once it ends. The backfill searches
+// every dinner without a photo, this one with them.
+function RetrySearch({ id, backfill, disabled }: { id: string; backfill: PhotoBackfill; disabled: boolean }) {
+  const problem = backfillProblem(backfill)
+  const tried = backfill.result?.dinners.find((dinner) => dinner.dinner_id === id)
+  let status: string | null = null
+  if (backfill.active) status = 'Searching again… this can take a few minutes.'
+  else if (backfill.status === 'succeeded') {
+    status = tried ? retryOutcomes[tried.outcome] : 'This dinner was not included in that search.'
+  }
+  return (
+    <>
+      <Button variant="outline" disabled={disabled || backfill.active} onClick={backfill.start}>
+        {backfill.active ? 'Searching again…' : 'Search again'}
+      </Button>
+      <p role="status" className={styles.status}>
+        {status}
+      </p>
+      {problem && (
+        <p role="alert" className={styles.problem}>
+          {problem}
+        </p>
+      )}
+    </>
+  )
+}
+
 function PickCandidate({ id, onPick }: { id: string; onPick: (candidateId: string) => void }) {
   const candidates = useRecipeCandidates(id)
   if (candidates.isPending) return <Skeleton className={styles.candidatesLoading} />

web/src/settings/photoBackfill.test.tsx

@@ -0,0 +1,63 @@
+import { QueryClientProvider } from '@tanstack/react-query'
+import { act, renderHook, waitFor } from '@testing-library/react'
+import type { ReactNode } from 'react'
+import { afterEach, describe, expect, it, vi } from 'vitest'
+import { ApiProvider, createApiClient, createQueryClient, createVerifiedSession } from '../api/index.ts'
+import { HOME_KEY } from '../home/api.ts'
+import { planKey } from '../plan/api.ts'
+import { createNetworkStatus } from '../pwa/networkStatus.ts'
+import { fakeLocation, jsonResponse, ME, stubFetch } from '../test/fetchStub.ts'
+import { parseBackfillResult, usePhotoBackfill } from './photoBackfill.ts'
+
+afterEach(() => {
+  vi.unstubAllGlobals()
+})
+
+const RESULT = { dinners: [{ dinner_id: 'd3', outcome: 'grounded', reason: null }] }
+
+function backfillJob(status: string, result: unknown) {
+  return jsonResponse(200, { id: 'j9', kind: 'backfill_dinner_photos', status, result, error: null })
+}
+
+describe('usePhotoBackfill', () => {
+  it('reads the plans, home and dinners again once the backfill ends, so a found photo shows everywhere', async () => {
+    const fetchMock = stubFetch(
+      jsonResponse(200, ME),
+      jsonResponse(202, { id: 'j9' }),
+      backfillJob('running', null),
+      backfillJob('succeeded', RESULT),
+    )
+    const api = createApiClient(fakeLocation(), createNetworkStatus(window), createVerifiedSession(localStorage))
+    await api.loadMe()
+    const queryClient = createQueryClient()
+    const plan = planKey('2026-10-05')
+    for (const key of [plan, HOME_KEY, ['dinners', 'quest_log']]) queryClient.setQueryData(key, {})
+    const wrapper = ({ children }: { children: ReactNode }) => (
+      <ApiProvider client={api}>
+        <QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
+      </ApiProvider>
+    )
+    const { result } = renderHook(() => usePhotoBackfill(), { wrapper })
+
+    act(() => result.current.start())
+    await waitFor(() => expect(result.current.status).toBe('running'))
+    expect(result.current.active).toBe(true)
+    expect(queryClient.getQueryState(plan)?.isInvalidated).toBe(false)
+
+    await waitFor(() => expect(result.current.status).toBe('succeeded'), { timeout: 3000 })
+    expect(result.current).toMatchObject({ active: false, error: null, result: RESULT })
+    expect(fetchMock).toHaveBeenCalledTimes(4)
+    for (const key of [plan, HOME_KEY, ['dinners', 'quest_log']]) {
+      expect(queryClient.getQueryState(key)?.isInvalidated).toBe(true)
+    }
+  })
+})
+
+describe('parseBackfillResult', () => {
+  it('refuses a result that is not a photo backfill', () => {
+    expect(() => parseBackfillResult('job', { dinners: [{ dinner_id: 'd3', outcome: 'lost', reason: null }] })).toThrow(
+      'job: outcome "lost" is not one of grounded, failed, skipped',
+    )
+    expect(() => parseBackfillResult('job', { stage: 'x' })).toThrow('job: result is not a photo backfill')
+  })
+})

web/src/settings/photoBackfill.ts

@@ -0,0 +1,110 @@
+import { useMutation, useQueryClient } from '@tanstack/react-query'
+import { useEffect, useMemo, useState } from 'react'
+import { useApi, useJob, type JobStatus } from '../api/index.ts'
+import { isRecord } from '../api/errors.ts'
+import { oneOf } from '../dinners/api.ts'
+import { HOME_KEY } from '../home/api.ts'
+
+export const PHOTO_BACKFILL_PATH = '/api/v1/settings/photo-backfill'
+
+// BACKFILL_JOB_KIND is the kind of the job a photo backfill runs as.
+export const BACKFILL_JOB_KIND = 'backfill_dinner_photos'
+
+export const BACKFILL_OUTCOMES = ['grounded', 'failed', 'skipped'] as const
+export type BackfillOutcome = (typeof BACKFILL_OUTCOMES)[number]
+
+// BackfillDinner is what a photo backfill did for one dinner. reason is
+// set only when it failed: the reason code now its grounding_error.
+export type BackfillDinner = { dinner_id: string; outcome: BackfillOutcome; reason: string | null }
+
+// BackfillResult is the photo backfill job's result: each dinner it tried,
+// in the order it tried them.
+export type BackfillResult = { dinners: BackfillDinner[] }
+
+// Every query showing a dinner's photo, read again once a backfill ends:
+// the dinners lists and details, the plans and the home dashboard.
+const PHOTO_KEYS = [['dinners'], ['plans'], HOME_KEY] as const
+
+// parseBackfillResult checks a photo backfill job's result, which useJob
+// passes on unread.
+export function parseBackfillResult(where: string, body: unknown): BackfillResult {
+  if (!isRecord(body) || !Array.isArray(body.dinners)) throw new Error(`${where}: result is not a photo backfill`)
+  return {
+    dinners: body.dinners.map((dinner): BackfillDinner => {
+      if (!isRecord(dinner) || typeof dinner.dinner_id !== 'string' || (dinner.reason !== null && typeof dinner.reason !== 'string')) {
+        throw new Error(`${where}: result has a dinner that is not a backfill outcome`)
+      }
+      return {
+        dinner_id: dinner.dinner_id,
+        outcome: oneOf(where, 'outcome', BACKFILL_OUTCOMES, dinner.outcome),
+        reason: dinner.reason,
+      }
+    }),
+  }
+}
+
+// PhotoBackfill is a photo backfill started from this screen, as it goes.
+export type PhotoBackfill = {
+  // start asks the server for a backfill over every dinner without a
+  // photo. While one is already queued or running the server answers with
+  // that job, so starting never runs two at once.
+  start: () => void
+  // active is true from start until the job ends, except while a poll's
+  // failure leaves its progress unknown: starting again then asks the
+  // server for the job it is running and polls it anew.
+  active: boolean
+  // Why the backfill could not be started.
+  startError: Error | null
+  // The job's status, undefined until it is started and first polled.
+  status: JobStatus | undefined
+  // The job's result, once it has one.
+  result: BackfillResult | null
+  // The job's failure, a poll's failure, or a result that is not a backfill.
+  error: Error | null
+}
+
+// usePhotoBackfill starts a photo backfill (POST
+// /api/v1/settings/photo-backfill, 202 {id}) and follows its job with
+// useJob. When the job ends, every query showing a dinner's photo is read
+// again, so dinners it grounded show their photo everywhere.
+export function usePhotoBackfill(): PhotoBackfill {
+  const { apiFetch } = useApi()
+  const queryClient = useQueryClient()
+  const [jobId, setJobId] = useState<string | null>(null)
+  const job = useJob<unknown>(BACKFILL_JOB_KIND, jobId)
+  const begin = useMutation({
+    mutationFn: async (): Promise<string> => {
+      const body = await apiFetch<unknown>(PHOTO_BACKFILL_PATH, { method: 'POST' })
+      if (!isRecord(body) || typeof body.id !== 'string') throw new Error(`POST ${PHOTO_BACKFILL_PATH}: response has no job id`)
+      return body.id
+    },
+    onSuccess: (id) => {
+      setJobId(id)
+      job.poll()
+    },
+  })
+
+  const ended = job.status === 'succeeded' || job.status === 'failed'
+  useEffect(() => {
+    if (!ended) return
+    for (const queryKey of PHOTO_KEYS) void queryClient.invalidateQueries({ queryKey })
+  }, [ended, jobId, queryClient])
+
+  const parsed = useMemo(() => {
+    if (job.result === null) return { result: null, error: null }
+    try {
+      return { result: parseBackfillResult(`GET /api/v1/jobs/${jobId}`, job.result), error: null }
+    } catch (error) {
+      return { result: null, error: error as Error }
+    }
+  }, [job.result, jobId])
+
+  return {
+    start: () => begin.mutate(),
+    active: begin.isPending || (jobId !== null && !ended && job.error === null),
+    startError: begin.error,
+    status: job.status,
+    result: parsed.result,
+    error: job.error ?? parsed.error,
+  }
+}

web/src/test/dinners.ts

@@ -13,6 +13,7 @@ export type DinnerBody = {
   photo_url: string | null
   grounding_status: string
   grounding_error: string | null
+  photo_backfill_eligible: boolean
   updated_by: string | null
   created_at: string
   updated_at: string
@@ -35,6 +36,7 @@ export function dinnerBody(id: string, name: string, overrides: Partial<DinnerBo
     photo_url: null,
     grounding_status: 'grounded',
     grounding_error: null,
+    photo_backfill_eligible: false,
     updated_by: null,
     created_at: '2026-10-01T15:00:00.000Z',
     updated_at: '2026-10-01T15:00:00.000Z',