Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2613e969ff | ||
|
|
884d16638f | ||
|
|
34e84964a3 |
@@ -78,17 +78,18 @@ behaviour, exercise it (hit the endpoint, render the page).
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
| `cmd/server` | Entry point. Wiring only — no business logic. |
|
||||
| `cmd/vapidgen` | Prints a new VAPID key pair for Web Push. |
|
||||
| `internal/config` | `MEALQUEST_*` environment config; secrets from `*_FILE`. |
|
||||
| `internal/server` | HTTP server, routes, `/health`. |
|
||||
| `internal/frontend` | `go:embed` of the built PWA with SPA fallback. |
|
||||
| `internal/*` (planned) | `auth`, `db`, `store`, `secretbox`, `household`, `dinners`, `recipes`, `ai/llm`, `ai/harness`, `ai/skills`, `suggest`, `plan`, `shopping`, `ideas`, `gcal`, `notify`, `push`, `chat`, `jobs`, `scheduler`. |
|
||||
| `migrations/` | SQL migrations (planned). |
|
||||
| `internal/*` | `auth`, `forwarded`, `secureurl`, `db`, `store`, `secretbox`, `household`, `dinners`, `recipes`, `ai/llm`, `ai/harness`, `ai/skills`, `ai/aisettings`, `suggest`, `plan`, `shopping`, `ideas`, `home`, `gcal`, `notify`, `push`, `chat`, `vapid`, `jobs`, `scheduler` — one line each in [ARCHITECTURE.md](docs/ARCHITECTURE.md#repository-layout). |
|
||||
| `migrations/` | SQL migrations, embedded and applied at boot by `internal/store`; sqlc queries in `migrations/queries/`. |
|
||||
| `web/` | React PWA source. |
|
||||
| `deploy/` | Deployment files (planned). |
|
||||
| `deploy/` | Deployment files: compose stack, systemd units, [deploy runbook](deploy/RUNBOOK.md) and the automated deploy (`auto-deploy.sh` with `systemd/mealquest-auto-deploy.*`). |
|
||||
| `docs/` | Project documentation, including [ARCHITECTURE.md](docs/ARCHITECTURE.md). |
|
||||
| `design/` | Requirements and UI handoff. **Read-only reference** — do not edit. |
|
||||
| `Dockerfile` | Multi-stage image build, at the repository root. |
|
||||
| `Makefile` | `dev`, `test`, `build`, `docker`. |
|
||||
| `Makefile` | `dev`, `test`, `build`, `docker`; `sqlc` and `sqlc-check` regenerate and check `internal/db`. |
|
||||
| `.gitea/workflows/ci.yml` | CI quality floor. |
|
||||
|
||||
Design references:
|
||||
@@ -104,7 +105,10 @@ Design references:
|
||||
- **Task branches:** `task/<slug>_<task-uuid>`, branched from the feature branch.
|
||||
- Task pull requests **squash-merge into the feature branch**.
|
||||
- The feature pull request merges into `dev`.
|
||||
- **Only `overseer` merges `dev` → `main`** (a release).
|
||||
- **Only `overseer` merges `dev` → `main`** (a release). On the production
|
||||
host, `mealquest-auto-deploy.timer` then deploys the new `main` once its
|
||||
CI passes ([deploy/RUNBOOK.md](deploy/RUNBOOK.md) "Automated deploys");
|
||||
nothing deploys `dev` or a feature branch.
|
||||
- Never push directly to `dev` or `main`.
|
||||
- Authors never review, approve, merge or close their own work.
|
||||
- Never store credentials in the repository; environment or ignored files only.
|
||||
|
||||
@@ -7,9 +7,27 @@ builds a weekly plan, pushes it to Google Calendar, and generates a shopping
|
||||
list grouped by dinner, kept as a browsable weekly history. It ships as a
|
||||
single Go service with the React PWA embedded in the binary.
|
||||
|
||||
> Status: this repository is the Feature 01 scaffold (Go service with
|
||||
> `/health`, embedded PWA skeleton, Makefile, Dockerfile, CI). The planned
|
||||
> application is described in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
|
||||
Production runs at <https://mealquest.cubecraftlabs.com> for one household,
|
||||
signed in through Authentik.
|
||||
|
||||
## Stack
|
||||
|
||||
- **Backend:** Go 1.25 (`CGO_ENABLED=0`, one static binary), chi router,
|
||||
SQLite through `modernc.org/sqlite` with sqlc queries and embedded
|
||||
migrations, Authentik OIDC, `log/slog`. AI calls go to an
|
||||
OpenAI-compatible, Anthropic or Ollama endpoint over plain HTTP; recipes
|
||||
come from SearXNG or Brave search and the recipe page's schema.org data.
|
||||
Google Calendar, Google Chat and Web Push are the integrations.
|
||||
- **Frontend:** React 19, TypeScript, Vite, React Router 7, TanStack Query 5,
|
||||
`vite-plugin-pwa`, CSS Modules over the design tokens; tests with Vitest,
|
||||
Testing Library and jsdom.
|
||||
- **Deployment:** a Docker Compose stack (the service, Litestream
|
||||
replication to the NAS, a SearXNG sidecar) behind Traefik, with systemd
|
||||
units for the firewall, nightly media backup and the automated deploy of
|
||||
`main`.
|
||||
|
||||
The approved dependency list is in [AGENTS.md](AGENTS.md); nothing else is
|
||||
added without the owner's approval.
|
||||
|
||||
## Running it locally
|
||||
|
||||
@@ -28,6 +46,15 @@ invalid variable.
|
||||
| `make test` | `gofmt` check, `go vet`, `go test -race`, web typecheck and tests. |
|
||||
| `make build` | Build the PWA, embed it, build the static binary `bin/mealquest`. |
|
||||
| `make docker` | Build the container image from the root `Dockerfile`. |
|
||||
| `make sqlc-check` | Regenerate `internal/db` with sqlc and fail if it drifted. |
|
||||
|
||||
## Deploying and operating
|
||||
|
||||
[deploy/RUNBOOK.md](deploy/RUNBOOK.md) is the deploy runbook: build, ship and
|
||||
restart, verifying the deployed version, rollback and the automated deploy.
|
||||
Database restore, the boot configuration and other host procedures are in
|
||||
CTXD: project `mealquest`, file `RUNBOOKS.MD`. Only `overseer` releases, by
|
||||
merging `dev` into `main`.
|
||||
|
||||
## Where to read next
|
||||
|
||||
|
||||
@@ -0,0 +1,357 @@
|
||||
# MealQuest deploy runbook
|
||||
|
||||
The one way to deploy MealQuest to production: the docker compose project
|
||||
`mealquest` on office-srvr `10.60.1.14` (see
|
||||
[ARCHITECTURE.md](../docs/ARCHITECTURE.md#topology)). Production runs `main`
|
||||
or a release tag; only `overseer` merges `dev` → `main`. Every deploy builds
|
||||
on the host, from the checked-out commit, with `make docker`.
|
||||
|
||||
Host setup (clone, firewall, the systemd unit, Traefik, DNS, Litestream) and
|
||||
the database restore are in CTXD project `mealquest`, `RUNBOOKS.MD`. This
|
||||
file is the build, restart, verification and rollback procedure; where it
|
||||
uses a CTXD section, it names it.
|
||||
|
||||
All commands run on `10.60.1.14` as `overseer`, in bash, from the checkout
|
||||
`/home/overseer/mealquest`. A block that ends with `FAILED` or a non-zero
|
||||
`$?` stops the deploy: do not go on to the next step.
|
||||
|
||||
## How the image carries its version
|
||||
|
||||
`make docker` runs `docker build --build-arg VERSION=<version>` and tags the
|
||||
image `mealquest:<version>`, where `<version>` is
|
||||
`git describe --tags --always` of the checkout (a release tag such as
|
||||
`v1.0.0`, otherwise the short SHA), with `-dirty` added when a tracked file
|
||||
is modified. The [Dockerfile](../Dockerfile) links it into the binary, and
|
||||
`/health` reports it:
|
||||
|
||||
```json
|
||||
{"status":"ok","version":"v1.0.0"}
|
||||
```
|
||||
|
||||
Never pass `--build-arg VERSION` by hand, and never build with
|
||||
`docker compose build` or `docker compose up --build`: compose passes no
|
||||
version, so that image reports `dev`.
|
||||
|
||||
The compose service `mealquest` runs the image named `mealquest-mealquest`
|
||||
(compose's name for the image it would build; check with
|
||||
`docker compose -f deploy/compose.yaml config --images mealquest`). It
|
||||
already exists, so `docker compose up` uses it and never builds. A deploy
|
||||
points that name at the `make docker` image with `docker tag`. Every
|
||||
`mealquest:<version>` image is kept for rollback; do not prune them.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `make`, `git`, `curl`, `python3` and Docker Engine with Compose on the
|
||||
host (`make` is not in the base install: `sudo apt-get install make`).
|
||||
- `deploy/.env` (git-ignored, owner `overseer`, mode 0600) sets every
|
||||
variable [`deploy/.env.example`](.env.example) sets, with
|
||||
`MEALQUEST_BIND_ADDRESS=10.60.1.14`. It must not set the variables
|
||||
`deploy/compose.yaml` fixes (`MEALQUEST_HTTP_ADDR`, `MEALQUEST_DATA_DIR`,
|
||||
`MEALQUEST_SEARXNG_URL` and the `*_FILE` variables). Creating it: CTXD
|
||||
`RUNBOOKS.MD` "Stack and first deploy" > "3. Environment file".
|
||||
- `deploy/secrets/` (git-ignored, mode 0700) holds `secret_key`,
|
||||
`oidc_client_secret`, `google_client_secret`, `vapid_private_key` and
|
||||
`litestream_key`, each owned by `10001:10001` with mode 0600, and
|
||||
`searxng.env`, owned by `overseer` with mode 0600. Creating them: CTXD
|
||||
`RUNBOOKS.MD` "Stack and first deploy" > "4. Secrets".
|
||||
- The compose file is the checkout's [`deploy/compose.yaml`](compose.yaml);
|
||||
the systemd unit [`mealquest.service`](systemd/mealquest.service) starts
|
||||
and stops it and is the only way the stack starts.
|
||||
- DNS and Traefik already route `https://mealquest.cubecraftlabs.com` to
|
||||
`10.60.1.14:8090` (CTXD `RUNBOOKS.MD` "Reverse proxy (Traefik)", "DNS").
|
||||
|
||||
Check `deploy/.env` and the secrets before every deploy (a new release may
|
||||
add a variable; the service refuses to boot without it). Both commands must
|
||||
print nothing:
|
||||
|
||||
```sh
|
||||
cd /home/overseer/mealquest
|
||||
comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \
|
||||
<(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u) # variables missing from deploy/.env
|
||||
sudo find deploy/secrets -type f ! -perm 600 # secrets not mode 0600
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
### 1. Build
|
||||
|
||||
Record the release production is running now, check out the new release
|
||||
and build its image. Nothing restarts.
|
||||
|
||||
The rollback target is the version the running service reports on
|
||||
`/health`, read before the checkout changes, never the checkout's `HEAD`:
|
||||
after a failed or deferred deploy the checkout can be ahead of production.
|
||||
The block resolves that version to a commit and prints it as the rollback
|
||||
SHA. A running version of `dev` or `*-dirty` names no commit: the block
|
||||
fails, and the rollback target must be established by hand before going on.
|
||||
|
||||
The checkout must be clean (the same test as the Makefile's `GIT_DIRTY`:
|
||||
no modified tracked file outside the generated `internal/frontend/dist`).
|
||||
The block computes the image tag exactly as `make docker` does, `-dirty`
|
||||
included, and checks that image exists after the build.
|
||||
|
||||
```sh
|
||||
cd /home/overseer/mealquest && (
|
||||
set -euo pipefail
|
||||
fail() { echo "$*" >&2; exit 1; }
|
||||
built=
|
||||
trap 'st=$?
|
||||
if [ "$st" -eq 0 ] && [ -n "$built" ]; then echo "built: $built"
|
||||
else [ "$st" -ne 0 ] || st=1; echo "build FAILED (status $st)" >&2; fi
|
||||
exit "$st"' EXIT
|
||||
bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-)
|
||||
[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
|
||||
running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
|
||||
|| fail "http://$bind:8090/health did not return a version: no running release to roll back to"
|
||||
case "$running" in
|
||||
dev|*-dirty) fail "the running release reports $running, which names no commit" ;;
|
||||
esac
|
||||
git fetch --tags origin
|
||||
rollback=$(git rev-parse --verify --quiet "$running^{commit}") \
|
||||
|| fail "running version $running is not a commit in this repository"
|
||||
echo "rollback SHA: $rollback (running version $running)" # RECORD it for rollback
|
||||
git checkout --detach "refs/remotes/origin/main^{commit}" # or the release tag, e.g. v1.0.0
|
||||
dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist')
|
||||
[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first:
|
||||
$dirty"
|
||||
v=$(git describe --tags --always)${dirty:+-dirty} # the Makefile's VERSION
|
||||
make docker
|
||||
docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed"
|
||||
built="mealquest:$v"
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Ship and restart
|
||||
|
||||
1. **Pre-restart gates.** Run CTXD `RUNBOOKS.MD` "Routine deploy and
|
||||
rollback" > "Deploy" step 2 (a verified pre-deploy restore point; record
|
||||
it with the rollback SHA) and step 3 (boot-required configuration). The
|
||||
restart applies migrations the previous release refuses, so without that
|
||||
restore point there is no rollback. If either fails, do not restart:
|
||||
production stays on the previous release.
|
||||
2. **Point the stack at the new image and restart.** `docker compose pull`
|
||||
fetches the pinned `litestream` and `searxng` images before anything
|
||||
stops; `mealquest` is never pulled. The restart applies the firewall,
|
||||
then runs `down` and `up -d`, and returns once `mealquest` passes its
|
||||
healthcheck (`/health` inside the container).
|
||||
|
||||
```sh
|
||||
cd /home/overseer/mealquest && (
|
||||
set -euo pipefail
|
||||
fail() { echo "$*" >&2; exit 1; }
|
||||
up=
|
||||
trap 'st=$?
|
||||
if [ "$st" -eq 0 ] && [ -n "$up" ]; then echo "restarted on: $up"
|
||||
else [ "$st" -ne 0 ] || st=1; echo "restart FAILED (status $st)" >&2; fi
|
||||
exit "$st"' EXIT
|
||||
v=$(git describe --tags --always)
|
||||
docker compose -f deploy/compose.yaml pull litestream searxng
|
||||
docker tag "mealquest:$v" mealquest-mealquest
|
||||
sudo systemctl restart mealquest
|
||||
for s in mealquest litestream searxng; do
|
||||
id=$(docker compose -f deploy/compose.yaml ps --status running --quiet "$s")
|
||||
[ -n "$id" ] || fail "$s is not running"
|
||||
done
|
||||
id=$(docker compose -f deploy/compose.yaml ps --quiet mealquest)
|
||||
health=$(docker inspect -f '{{.State.Health.Status}}' "$id")
|
||||
[ "$health" = healthy ] || fail "mealquest is $health, not healthy"
|
||||
up="mealquest:$v"
|
||||
)
|
||||
```
|
||||
|
||||
On failure read `journalctl -u mealquest` and
|
||||
`docker compose -f deploy/compose.yaml logs mealquest`: a configuration
|
||||
error names the variable. Fix it or roll back.
|
||||
|
||||
### 3. Verify the deployed version and a real screen
|
||||
|
||||
A `/health` 200 alone proves only that some build is up. This step checks,
|
||||
on the LAN address and through Traefik, that:
|
||||
|
||||
- `/health` reports exactly the checkout's `git describe --tags --always`;
|
||||
- `/` and the Plan screen's route `/plan` serve the built PWA shell (the
|
||||
React root and a hashed `/assets/` script, not the placeholder page an
|
||||
unbuilt embed serves);
|
||||
- that script is served as JavaScript.
|
||||
|
||||
```sh
|
||||
cd /home/overseer/mealquest && (
|
||||
set -euo pipefail
|
||||
fail() { echo "$*" >&2; exit 1; }
|
||||
want=
|
||||
trap 'st=$?
|
||||
if [ "$st" -eq 0 ] && [ -n "$want" ]; then echo "deployed and verified: $want ($(git rev-parse HEAD))"
|
||||
else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): the deploy is not verified" >&2; fi
|
||||
exit "$st"' EXIT
|
||||
v=$(git describe --tags --always)
|
||||
for base in http://10.60.1.14:8090 https://mealquest.cubecraftlabs.com; do
|
||||
got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
|
||||
|| fail "$base/health did not return a version"
|
||||
[ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v"
|
||||
for path in / /plan; do
|
||||
page=$(curl -fsS "$base$path") || fail "$base$path did not return 200"
|
||||
grep -q '<div id="root"></div>' <<< "$page" || fail "$base$path is not the PWA shell"
|
||||
js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true
|
||||
[ -n "$js" ] || fail "$base$path references no built /assets/ script"
|
||||
type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200"
|
||||
case "$type" in
|
||||
text/javascript*) ;;
|
||||
*) fail "$base$js is $type, not JavaScript" ;;
|
||||
esac
|
||||
done
|
||||
echo "$base: version $got, PWA shell and $js served"
|
||||
done
|
||||
want=$v
|
||||
)
|
||||
```
|
||||
|
||||
- `deployed and verified: <version> (<SHA>)` and `$?` = `0` → open
|
||||
`https://mealquest.cubecraftlabs.com` in a browser: sign in and the Home
|
||||
screen renders with the household's data. Then record the deployed
|
||||
version and SHA, the rollback SHA, the pre-deploy restore point and the
|
||||
date in the release's Maestro task.
|
||||
- `verification FAILED`, any error or a non-zero `$?` → the deploy is not
|
||||
done. A version mismatch means the stack runs a different image than the
|
||||
checkout: repeat step 2, or roll back.
|
||||
|
||||
## Rollback to a previous release
|
||||
|
||||
Rolling back points `mealquest-mealquest` at the previous release's
|
||||
`mealquest:<version>` image again; that release's checkout keeps the
|
||||
compose file and the version check consistent with it.
|
||||
|
||||
1. **Check whether the rollback crosses a migration** with CTXD
|
||||
`RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a previous
|
||||
SHA" (its first block), run against the rollback SHA that deploy step 1
|
||||
recorded. If it prints versions unknown to the rollback SHA, restore the
|
||||
database to the pre-deploy restore point first (CTXD `RUNBOOKS.MD`
|
||||
"Restore the database", with `-timestamp <restore point>`); changes
|
||||
since then are lost, so tell the household.
|
||||
2. **Check out the rollback SHA and retag its image.** An image built before
|
||||
this runbook was not tagged `mealquest:<version>`; `make docker` builds
|
||||
it from the same commit.
|
||||
|
||||
```sh
|
||||
cd /home/overseer/mealquest && (
|
||||
set -euo pipefail
|
||||
fail() { echo "$*" >&2; exit 1; }
|
||||
P=<rollback SHA> # recorded by step 1, from the running release's /health
|
||||
back=
|
||||
trap 'st=$?
|
||||
if [ "$st" -eq 0 ] && [ -n "$back" ]; then echo "rolled back to: $back"
|
||||
else [ "$st" -ne 0 ] || st=1; echo "rollback FAILED (status $st)" >&2; fi
|
||||
exit "$st"' EXIT
|
||||
git checkout --detach "$P"
|
||||
v=$(git describe --tags --always)
|
||||
docker image inspect "mealquest:$v" > /dev/null 2>&1 || make docker
|
||||
docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v"
|
||||
docker tag "mealquest:$v" mealquest-mealquest
|
||||
sudo systemctl restart mealquest
|
||||
back="mealquest:$v"
|
||||
)
|
||||
```
|
||||
|
||||
3. **Run step 3 above** unchanged. The rollback is done only when it prints
|
||||
`deployed and verified:` with the previous release's version.
|
||||
|
||||
## Automated deploys
|
||||
|
||||
[`auto-deploy.sh`](auto-deploy.sh), installed as
|
||||
`/usr/local/sbin/mealquest-auto-deploy`, runs this runbook's deploy unattended:
|
||||
step 1 (build), the prerequisite `deploy/.env` and secrets check, CTXD
|
||||
`RUNBOOKS.MD` "Deploy" steps 2 (pre-deploy restore point) and 3
|
||||
(boot-required configuration), step 2 (ship and restart) and step 3
|
||||
(verify). No check is skipped or weakened, and it stops at the first
|
||||
failure. Every failure before the restart leaves production on the previous
|
||||
release.
|
||||
|
||||
### What runs, and when
|
||||
|
||||
[`mealquest-auto-deploy.timer`](systemd/mealquest-auto-deploy.timer) starts
|
||||
[`mealquest-auto-deploy.service`](systemd/mealquest-auto-deploy.service)
|
||||
at every quarter hour (`OnCalendar=*-*-* *:0/15:00`), plus up to 5 minutes'
|
||||
random delay; a check missed while the host was down runs as soon as it is
|
||||
back (`Persistent=true`). The service runs the script as `overseer` from the
|
||||
checkout, and only while `mealquest.service` is active: a stack stopped on
|
||||
purpose (a database or media restore) is skipped, not redeployed. Output goes
|
||||
to the journal (`journalctl -u mealquest-auto-deploy`).
|
||||
|
||||
Each run fetches `origin` and compares `origin/main` with the last
|
||||
`deployed=` SHA in the deploy record `/home/overseer/mealquest-auto-deploy.log`:
|
||||
|
||||
| `origin/main` | Result |
|
||||
| --- | --- |
|
||||
| is the last deployed SHA | No-op: exit 0, no output. |
|
||||
| CI (Gitea commit status) `pending` | Exit 0, no output; the next run checks again. |
|
||||
| CI `failure`, `error`, `warning`, or no status yet | Error in the journal, exit 1, nothing deployed. |
|
||||
| the last record line is `FAILED` for this SHA | Error in the journal, exit 1, nothing deployed: a failed deploy is never retried by the timer. |
|
||||
| CI `success` | Deploy. |
|
||||
|
||||
The CI status is read from
|
||||
`https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits/<SHA>/status`
|
||||
(public; no credential). A commit merged moments before a run can briefly
|
||||
report no status, which fails that run; the next run reads it again.
|
||||
|
||||
A deploy appends one line to the record:
|
||||
|
||||
```text
|
||||
2026-10-12T15:04:05Z previous=<rollback SHA> deployed=<SHA> restore_point=2026-10-12T15:01:59Z
|
||||
```
|
||||
|
||||
`previous` is the rollback SHA (from the running release's `/health`,
|
||||
never the checkout) and `restore_point` the verified pre-deploy restore
|
||||
point: record both in the release's Maestro task. A failed deploy appends
|
||||
`<date> FAILED target=<SHA> status=<exit status>` and logs
|
||||
`auto-deploy FAILED (status <n>)` to the journal, adding whether it failed
|
||||
before the restart (production untouched) or after it (read
|
||||
`journalctl -u mealquest`, then fix or roll back). Either way the checkout is
|
||||
left at the target commit; the running release on `/health` stays the
|
||||
rollback target.
|
||||
|
||||
### Install on the host
|
||||
|
||||
The script and units are installed copies: repeat this whenever a release
|
||||
changes `deploy/auto-deploy.sh` or the units. From the checkout, at a release
|
||||
that contains them:
|
||||
|
||||
```sh
|
||||
cd /home/overseer/mealquest
|
||||
sudo install -m 0755 deploy/auto-deploy.sh /usr/local/sbin/mealquest-auto-deploy
|
||||
sudo install -m 0644 deploy/systemd/mealquest-auto-deploy.service \
|
||||
deploy/systemd/mealquest-auto-deploy.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
The timer refuses to deploy until the record holds a deploy, so seed it with
|
||||
one manual deploy of the release production should run (below), then start
|
||||
the timer:
|
||||
|
||||
```sh
|
||||
sudo /usr/local/sbin/mealquest-auto-deploy --ref refs/remotes/origin/main
|
||||
sudo systemctl enable --now mealquest-auto-deploy.timer
|
||||
systemctl list-timers mealquest-auto-deploy.timer
|
||||
```
|
||||
|
||||
To stop automated deploys: `sudo systemctl disable --now mealquest-auto-deploy.timer`.
|
||||
|
||||
### Manual override
|
||||
|
||||
```sh
|
||||
sudo /usr/local/sbin/mealquest-auto-deploy --ref <ref>
|
||||
```
|
||||
|
||||
deploys an explicit ref (a SHA, tag or `origin/<branch>`) through the same
|
||||
blocks, bypassing the CI and no-op gates: for a roll-forward fix, a re-run
|
||||
after a `FAILED` record, or seeding the record. Run as root, the script
|
||||
re-executes itself as `overseer`; it holds a lock, so it never runs alongside
|
||||
the timer's run. It prints `deployed and verified:` and exits 0 only when
|
||||
step 3 passed.
|
||||
|
||||
### Rollback
|
||||
|
||||
Rollback is not automated: follow "Rollback to a previous release" above
|
||||
and CTXD `RUNBOOKS.MD` "Routine deploy and rollback" > "Rollback to a
|
||||
previous SHA", using the `previous` SHA and `restore_point` from the record.
|
||||
The timer does not undo a manual rollback: `origin/main` is still the last
|
||||
recorded deploy, so its runs are no-ops until `main` moves, and a new `main`
|
||||
then deploys as usual. Disable the timer if the rollback must hold.
|
||||
Executable
+251
@@ -0,0 +1,251 @@
|
||||
#!/usr/bin/env bash
|
||||
# MealQuest automated deploy: deploy/RUNBOOK.md, run unattended.
|
||||
#
|
||||
# mealquest-auto-deploy deploy origin/main if it is new and CI passed
|
||||
# mealquest-auto-deploy --ref <ref> deploy <ref> now, bypassing the CI and no-op gates
|
||||
#
|
||||
# Installed as /usr/local/sbin/mealquest-auto-deploy and run as overseer by
|
||||
# mealquest-auto-deploy.service (deploy/systemd); install, semantics and the
|
||||
# manual override: deploy/RUNBOOK.md "Automated deploys". Run as root (sudo)
|
||||
# it re-executes itself as overseer.
|
||||
#
|
||||
# Without --ref: exit 0 with no output when origin/main is the last deployed
|
||||
# SHA in the deploy record or its CI is pending; exit 1, deploying nothing,
|
||||
# when its CI failed or reports no status, or when the last attempt to deploy
|
||||
# that SHA failed. Every deploy runs the runbook's blocks in order and stops
|
||||
# at the first failure; nothing before the restart touches production. A
|
||||
# verified deploy appends one line to the deploy record:
|
||||
# <UTC date> previous=<SHA> deployed=<SHA> restore_point=<UTC date>
|
||||
# a failed one appends:
|
||||
# <UTC date> FAILED target=<SHA> status=<exit status>
|
||||
set -euo pipefail
|
||||
|
||||
CHECKOUT=/home/overseer/mealquest
|
||||
RECORD=/home/overseer/mealquest-auto-deploy.log
|
||||
LOCK=/home/overseer/mealquest-auto-deploy.lock
|
||||
STATUS_API=https://code.cubecraftlabs.com/api/v1/repos/Maniacs_Mansion/MealQuest/commits
|
||||
PUBLIC=https://mealquest.cubecraftlabs.com
|
||||
D=/var/lib/docker/volumes/mealquest-data/_data
|
||||
COMPOSE=(docker compose -f deploy/compose.yaml)
|
||||
|
||||
fail() { echo "$*" >&2; exit 1; }
|
||||
|
||||
if [ "$(id -u)" -eq 0 ]; then
|
||||
exec sudo -u overseer -H -- "$0" "$@"
|
||||
fi
|
||||
[ "$(id -un)" = overseer ] || fail "run as overseer (or root), not $(id -un)"
|
||||
|
||||
ref=
|
||||
case "$#" in
|
||||
0) ;;
|
||||
2) [ "$1" = --ref ] && [ -n "$2" ] || fail "usage: $0 [--ref <ref>]"; ref=$2 ;;
|
||||
*) fail "usage: $0 [--ref <ref>]" ;;
|
||||
esac
|
||||
|
||||
exec 9>> "$LOCK"
|
||||
flock -n 9 || fail "another mealquest-auto-deploy run holds $LOCK"
|
||||
|
||||
cd "$CHECKOUT"
|
||||
git fetch --quiet --tags origin || fail "git fetch origin failed"
|
||||
|
||||
if [ -n "$ref" ]; then
|
||||
target=$(git rev-parse --verify --quiet "$ref^{commit}") || fail "$ref is not a commit in $CHECKOUT"
|
||||
else
|
||||
# The full remote-tracking ref: a bare origin/main resolves a tag or local
|
||||
# branch of that name first, which could deploy a commit not on main.
|
||||
target=$(git rev-parse --verify --quiet "refs/remotes/origin/main^{commit}") \
|
||||
|| fail "refs/remotes/origin/main is not a commit in $CHECKOUT"
|
||||
[ -f "$RECORD" ] || fail "no deploy record at $RECORD: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")"
|
||||
last=$(grep -oE 'deployed=[0-9a-f]{40}' "$RECORD" | tail -n 1 | cut -d = -f 2) || true
|
||||
[ -n "$last" ] || fail "$RECORD records no deploy: seed it with one manual deploy (deploy/RUNBOOK.md \"Automated deploys\")"
|
||||
[ "$target" != "$last" ] || exit 0
|
||||
if tail -n 1 "$RECORD" | grep -q " FAILED target=$target status="; then
|
||||
fail "auto-deploy of $target already failed (see $RECORD and the journal): fix it, then deploy with --ref"
|
||||
fi
|
||||
state=$(curl -fsS --max-time 30 "$STATUS_API/$target/status" \
|
||||
| python3 -c 'import json, sys; d = json.load(sys.stdin); print(d["state"] if d["total_count"] else "absent")') \
|
||||
|| fail "could not read the CI status of $target from $STATUS_API"
|
||||
case "$state" in
|
||||
success) ;;
|
||||
pending) exit 0 ;;
|
||||
*) fail "CI for origin/main $target is $state, not success: not deploying" ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
echo "auto-deploy: deploying $target${ref:+ (--ref $ref)}"
|
||||
restarted=
|
||||
W=
|
||||
on_exit() {
|
||||
local st=$?
|
||||
[ -z "$W" ] || rm -rf "$W" || st=1
|
||||
if [ "$st" -ne 0 ]; then
|
||||
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) FAILED target=$target status=$st" >> "$RECORD"
|
||||
if [ -n "$restarted" ]; then
|
||||
echo "auto-deploy FAILED (status $st) after the restart: production may be down or unverified; read journalctl -u mealquest, then fix or roll back (deploy/RUNBOOK.md)" >&2
|
||||
else
|
||||
echo "auto-deploy FAILED (status $st): production untouched" >&2
|
||||
fi
|
||||
fi
|
||||
exit "$st"
|
||||
}
|
||||
trap on_exit EXIT
|
||||
|
||||
# deploy/RUNBOOK.md "Deploy" > "1. Build": record the rollback SHA from the
|
||||
# RUNNING release, check out the target, reject a dirty checkout, build.
|
||||
bind=$(grep -E '^MEALQUEST_BIND_ADDRESS=' deploy/.env | cut -d = -f 2-) || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
|
||||
[ -n "$bind" ] || fail "deploy/.env sets no MEALQUEST_BIND_ADDRESS"
|
||||
running=$(curl -fsS "http://$bind:8090/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
|
||||
|| fail "http://$bind:8090/health did not return a version: no running release to roll back to"
|
||||
case "$running" in
|
||||
dev|*-dirty) fail "the running release reports $running, which names no commit" ;;
|
||||
esac
|
||||
rollback=$(git rev-parse --verify --quiet "$running^{commit}") \
|
||||
|| fail "running version $running is not a commit in this repository"
|
||||
echo "rollback SHA: $rollback (running version $running)"
|
||||
git checkout --quiet --detach "$target"
|
||||
dirty=$(git status --porcelain --untracked-files=no -- . ':(exclude)internal/frontend/dist')
|
||||
[ -z "$dirty" ] || fail "the checkout has modified tracked files; clean it first:
|
||||
$dirty"
|
||||
v=$(git describe --tags --always)${dirty:+-dirty} # the Makefile's VERSION
|
||||
make docker
|
||||
docker image inspect "mealquest:$v" > /dev/null 2>&1 || fail "no image mealquest:$v: the build failed"
|
||||
echo "built: mealquest:$v"
|
||||
|
||||
# deploy/RUNBOOK.md "Prerequisites": deploy/.env sets every variable of the
|
||||
# target's .env.example, and every secret is mode 0600.
|
||||
missing=$(comm -23 <(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env.example | sort -u) \
|
||||
<(grep -oE '^MEALQUEST_[A-Z_]+' deploy/.env | sort -u))
|
||||
[ -z "$missing" ] || fail "deploy/.env is missing: $missing"
|
||||
loose=$(sudo find deploy/secrets -type f ! -perm 600)
|
||||
[ -z "$loose" ] || fail "secrets not mode 0600: $loose"
|
||||
|
||||
# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 2: a verified
|
||||
# pre-deploy restore point. The scripts are CTXD "Count check". They and the
|
||||
# check files live in a private mktemp directory (mode 0700), not at shared
|
||||
# /tmp paths, and python runs with -I: the snapshot runs as root, and a
|
||||
# shared directory on its import path would let any local user run code as
|
||||
# root.
|
||||
W=$(mktemp -d)
|
||||
cat > "$W/mqcount.py" <<'EOF'
|
||||
import sqlite3, sys, hashlib
|
||||
c = sqlite3.connect(sys.argv[1])
|
||||
tabs = [r[0] for r in c.execute("select name from sqlite_master where type=\"table\" and name not like \"sqlite_%\" and name not like \"_litestream%\" order by name")]
|
||||
for t in ("dinners", "week_plans", "schema_migrations"):
|
||||
print(t, c.execute(f"select count(*) from {t}").fetchone()[0])
|
||||
h = hashlib.sha256()
|
||||
for t in tabs:
|
||||
for row in c.execute(f"select * from \"{t}\" order by rowid"):
|
||||
h.update(repr(row).encode())
|
||||
print("tables", len(tabs), "content-sha256", h.hexdigest()[:16])
|
||||
print("integrity", c.execute("pragma integrity_check").fetchone()[0])
|
||||
EOF
|
||||
cat > "$W/mqsnap.py" <<'EOF'
|
||||
import os, sqlite3, sys
|
||||
src, dst = sys.argv[1], sys.argv[2]
|
||||
if os.path.exists(dst):
|
||||
sys.exit(f"{dst} already exists")
|
||||
s = sqlite3.connect(f"file:{src}?mode=ro", uri=True)
|
||||
d = sqlite3.connect(dst)
|
||||
s.backup(d)
|
||||
s.close()
|
||||
result = d.execute("pragma integrity_check").fetchall()
|
||||
d.close()
|
||||
if result != [("ok",)]:
|
||||
sys.exit(f"snapshot integrity_check failed: {result}")
|
||||
print("snapshot", dst, "integrity ok")
|
||||
EOF
|
||||
point=$(
|
||||
set -eu
|
||||
point=
|
||||
trap 'st=$?
|
||||
rm -rf "$W" || st=1
|
||||
sudo sh -c "rm -f $D/predeploy-check.db*" || st=1
|
||||
if [ "$st" -eq 0 ] && [ -n "$point" ]; then echo "pre-deploy restore point: $point" >&2; echo "$point"
|
||||
else [ "$st" -ne 0 ] || st=1; echo "verification FAILED (status $st): do not restart" >&2; fi
|
||||
exit "$st"' EXIT
|
||||
exec 3>&1 1>&2 # block output to the journal; only the restore point to $point
|
||||
mkdir "$W/mqcheck"
|
||||
# a. Replication running, no log errors (grep: 0 match, 1 none, 2+ error), newest LTX listed
|
||||
id=$("${COMPOSE[@]}" ps --status running --quiet litestream) || fail "docker compose ps failed"
|
||||
[ -n "$id" ] || fail "litestream is not running"
|
||||
"${COMPOSE[@]}" logs --since 24h litestream > "$W/mqcheck/litestream.log" || fail "docker compose logs failed"
|
||||
gs=0
|
||||
grep -i error "$W/mqcheck/litestream.log" > "$W/mqcheck/litestream.errors" || gs=$?
|
||||
case "$gs" in
|
||||
0) cat "$W/mqcheck/litestream.errors"; fail "litestream logged errors in the last 24h";;
|
||||
1) ;;
|
||||
*) fail "the litestream log scan failed (status $gs)";;
|
||||
esac
|
||||
"${COMPOSE[@]}" exec -T litestream \
|
||||
litestream ltx -config /etc/litestream.yml -level all /data/mealquest.db
|
||||
# b. Restore replica to a scratch file (live files untouched), move off the volume, snapshot live
|
||||
"${COMPOSE[@]}" run --rm --no-deps litestream \
|
||||
restore -config /etc/litestream.yml -o /data/predeploy-check.db /data/mealquest.db
|
||||
sudo sh -c "mv $D/predeploy-check.db* $W/mqcheck/"
|
||||
sudo python3 -I "$W/mqsnap.py" "$D/mealquest.db" "$W/mqcheck/live.db"
|
||||
sudo chown -R "$(id -un)" "$W/mqcheck"
|
||||
sudo ls -ln "$D"
|
||||
# mealquest.db-wal and -shm owned by 10001, never 0 (CTXD "Count check")
|
||||
root=$(sudo find "$D" -maxdepth 1 \( -name mealquest.db-wal -o -name mealquest.db-shm \) -uid 0)
|
||||
[ -z "$root" ] || fail "owned by root, not 10001 (CTXD RUNBOOKS.MD \"Count check\"): $root"
|
||||
# c. Restored replica: integrity ok and identical counts/content-sha256 to the live snapshot
|
||||
python3 -I "$W/mqcount.py" "$W/mqcheck/predeploy-check.db" > "$W/mqcheck/restored.txt"
|
||||
python3 -I "$W/mqcount.py" "$W/mqcheck/live.db" > "$W/mqcheck/live.txt"
|
||||
cat "$W/mqcheck/restored.txt"
|
||||
grep -qx 'integrity ok' "$W/mqcheck/restored.txt" || fail "the restored replica failed integrity_check"
|
||||
diff "$W/mqcheck/restored.txt" "$W/mqcheck/live.txt" || fail "the restored replica differs from the live snapshot"
|
||||
# d. Restore point: after every check, before the restart
|
||||
point=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
exec 1>&3
|
||||
)
|
||||
|
||||
# CTXD RUNBOOKS.MD "Routine deploy and rollback" > "Deploy" step 3:
|
||||
# boot-required configuration, secrets read the way the service reads them.
|
||||
[ -f deploy/.env ] || fail "missing deploy/.env"
|
||||
cid=$(grep -m 1 -E '^MEALQUEST_GOOGLE_CLIENT_ID=' deploy/.env | cut -d = -f 2- | tr -d '"' | tr -d "'") \
|
||||
|| fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env"
|
||||
[ -n "$cid" ] || fail "missing MEALQUEST_GOOGLE_CLIENT_ID in deploy/.env"
|
||||
[ -e deploy/secrets/google_client_secret ] || fail "missing deploy/secrets/google_client_secret"
|
||||
[ -s deploy/secrets/google_client_secret ] || fail "deploy/secrets/google_client_secret is empty"
|
||||
id=$("${COMPOSE[@]}" ps --status running --quiet mealquest) || fail "docker compose ps failed"
|
||||
[ -n "$id" ] || fail "the mealquest container is not running: start it (the check reads secrets the way the service does)"
|
||||
docker exec "$id" sh -c 'test -s /run/secrets/google_client_secret && test -r /run/secrets/google_client_secret' \
|
||||
|| fail "google_client_secret is not readable by the service user inside the container"
|
||||
echo "boot-required configuration present"
|
||||
|
||||
# deploy/RUNBOOK.md "2. Ship and restart".
|
||||
"${COMPOSE[@]}" pull litestream searxng
|
||||
docker tag "mealquest:$v" mealquest-mealquest
|
||||
restarted=yes
|
||||
sudo systemctl restart mealquest
|
||||
for s in mealquest litestream searxng; do
|
||||
id=$("${COMPOSE[@]}" ps --status running --quiet "$s")
|
||||
[ -n "$id" ] || fail "$s is not running"
|
||||
done
|
||||
id=$("${COMPOSE[@]}" ps --quiet mealquest)
|
||||
health=$(docker inspect -f '{{.State.Health.Status}}' "$id")
|
||||
[ "$health" = healthy ] || fail "mealquest is $health, not healthy"
|
||||
echo "restarted on: mealquest:$v"
|
||||
|
||||
# deploy/RUNBOOK.md "3. Verify the deployed version and a real screen".
|
||||
for base in "http://$bind:8090" "$PUBLIC"; do
|
||||
got=$(curl -fsS "$base/health" | python3 -c 'import json, sys; print(json.load(sys.stdin)["version"])') \
|
||||
|| fail "$base/health did not return a version"
|
||||
[ "$got" = "$v" ] || fail "$base/health reports version $got, the checkout is $v"
|
||||
for path in / /plan; do
|
||||
page=$(curl -fsS "$base$path") || fail "$base$path did not return 200"
|
||||
grep -q '<div id="root"></div>' <<< "$page" || fail "$base$path is not the PWA shell"
|
||||
js=$(grep -oE 'src="/assets/[^"]+\.js"' <<< "$page" | head -n 1 | cut -d '"' -f 2) || true
|
||||
[ -n "$js" ] || fail "$base$path references no built /assets/ script"
|
||||
type=$(curl -fsS -o /dev/null -w '%{content_type}' "$base$js") || fail "$base$js did not return 200"
|
||||
case "$type" in
|
||||
text/javascript*) ;;
|
||||
*) fail "$base$js is $type, not JavaScript" ;;
|
||||
esac
|
||||
done
|
||||
echo "$base: version $got, PWA shell and $js served"
|
||||
done
|
||||
|
||||
deployed=$(git rev-parse HEAD)
|
||||
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) previous=$rollback deployed=$deployed restore_point=$point" >> "$RECORD"
|
||||
echo "deployed and verified: $v ($deployed), previous $rollback, restore point $point"
|
||||
@@ -0,0 +1,23 @@
|
||||
# Deploys origin/main to production when it is new and its CI passed
|
||||
# (/usr/local/sbin/mealquest-auto-deploy, from deploy/auto-deploy.sh).
|
||||
# Triggered by mealquest-auto-deploy.timer; output goes to the journal.
|
||||
# Installing it and the manual override: deploy/RUNBOOK.md "Automated deploys".
|
||||
#
|
||||
# ExecCondition= skips the run while mealquest.service is not active, so a
|
||||
# stack stopped on purpose (a database or media restore) is never redeployed
|
||||
# under the operator. It is deliberately not Requires=/Requisite= on
|
||||
# mealquest.service: those would start the stack, or stop this run when the
|
||||
# deploy itself restarts mealquest.service. Runs as overseer, who owns the
|
||||
# checkout and uses sudo for the runbook's privileged steps.
|
||||
[Unit]
|
||||
Description=MealQuest automated deploy of main
|
||||
Wants=network-online.target
|
||||
After=network-online.target docker.service mealquest.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=overseer
|
||||
Group=overseer
|
||||
WorkingDirectory=/home/overseer/mealquest
|
||||
ExecCondition=/usr/bin/systemctl is-active --quiet mealquest.service
|
||||
ExecStart=/usr/local/sbin/mealquest-auto-deploy
|
||||
@@ -0,0 +1,13 @@
|
||||
# Checks for a new main at every quarter hour (plus up to 5 minutes' random
|
||||
# delay). Persistent= runs a check at once when the host comes back if a
|
||||
# scheduled one was missed while it was down.
|
||||
[Unit]
|
||||
Description=Periodic MealQuest automated deploy
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* *:0/15:00
|
||||
RandomizedDelaySec=5min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
+89
-15
@@ -2,10 +2,8 @@
|
||||
|
||||
This mirrors the architecture section of `CONTEXT.MD` in the CTXD project
|
||||
`mealquest`, which is the source of record; if the two disagree, CTXD wins and
|
||||
this file should be updated. It describes the **planned** application. The
|
||||
repository today is the Feature 01 scaffold: `cmd/server`, `internal/config`,
|
||||
`internal/server` (`/health`), `internal/frontend` (embedded PWA with SPA
|
||||
fallback), the `web/` skeleton, `Makefile`, `Dockerfile` and CI.
|
||||
this file should be updated. It describes the application as built; nothing
|
||||
it names is still only planned.
|
||||
|
||||
Decision numbers (D-NNN) refer to `DECISIONS.MD` in CTXD.
|
||||
|
||||
@@ -46,20 +44,22 @@ headers, so sign-in works whichever peer the request arrives from.
|
||||
| `cmd/vapidgen` | Prints a new VAPID key pair for Web Push. |
|
||||
| `internal/config` | Env-only `MEALQUEST_*` config; secrets from `*_FILE`. |
|
||||
| `internal/server` | HTTP server, router, middleware. |
|
||||
| `internal/forwarded` | Client IP and scheme from `X-Forwarded-*`, trusted only from `MEALQUEST_TRUSTED_PROXY`. |
|
||||
| `internal/secureurl` | The transport rule for URLs that carry credentials (`secureurl.Check`). |
|
||||
| `internal/frontend` | Embedded PWA, SPA fallback. |
|
||||
| `internal/auth` | Authentik OIDC, sessions, CSRF. |
|
||||
| `internal/db`, `internal/store` | SQLite connections, sqlc queries. |
|
||||
| `internal/db`, `internal/store` | sqlc-generated queries; SQLite connections, migrations at boot, transactions. |
|
||||
| `internal/secretbox` | AES-256-GCM encryption for `_ct` columns. |
|
||||
| `internal/household`, `internal/dinners`, `internal/recipes` | Domain. |
|
||||
| `internal/ai/llm`, `internal/ai/harness`, `internal/ai/skills` | AI provider adapters, skill harness, skills. |
|
||||
| `internal/suggest`, `internal/plan`, `internal/shopping`, `internal/ideas` | Feature services. |
|
||||
| `internal/household`, `internal/dinners`, `internal/recipes` | Domain: household settings and local time, dinners (grounding, photo backfill), recipe search, fetch and grounding. |
|
||||
| `internal/ai/llm`, `internal/ai/harness`, `internal/ai/skills`, `internal/ai/aisettings` | AI provider adapters, skill harness, skills, AI settings. |
|
||||
| `internal/suggest`, `internal/plan`, `internal/shopping`, `internal/ideas`, `internal/home` | Feature services. |
|
||||
| `internal/gcal`, `internal/notify`, `internal/push`, `internal/chat` | Integrations. |
|
||||
| `internal/vapid` | VAPID key generation and validation. |
|
||||
| `internal/jobs`, `internal/scheduler` | Background jobs and schedules. |
|
||||
| `migrations/` | SQL migrations (no goose or golang-migrate). |
|
||||
| `migrations/` | SQL migrations, embedded and applied at boot by `internal/store` (no goose or golang-migrate); sqlc queries in `migrations/queries/`. |
|
||||
| `web/` | React PWA. |
|
||||
| `deploy/` | Deployment files. |
|
||||
| `docs/` | Documentation. |
|
||||
| `deploy/` | Production compose stack, systemd units, the [deploy runbook](../deploy/RUNBOOK.md) and the automated deploy (see Deployment). |
|
||||
| `docs/` | Documentation (this file). |
|
||||
| `design/` | Requirements and UI handoff (read-only reference). |
|
||||
|
||||
## Backend stack and conventions
|
||||
@@ -259,6 +259,29 @@ automatic** (D-021).
|
||||
unarchived dinner still `pending` with no `ground_dinner` job for its revision
|
||||
queued or running is set `failed` with `interrupted`, so no dinner stays
|
||||
pending with nothing left to ground it.
|
||||
Photo backfill. `POST /api/v1/settings/photo-backfill` (CSRF-checked)
|
||||
enqueues a `backfill_dinner_photos` job and answers 202 `{id}` without
|
||||
waiting for it; while one is queued or running it enqueues nothing and
|
||||
answers with that job's id, so two passes never overlap, across restarts
|
||||
too. The kind is refused by `POST /api/v1/jobs`. The job
|
||||
grounds, one at a time with `Ground` on its name, every unarchived dinner
|
||||
that is `failed` with no photo and no pasted `recipe_url` — grounded,
|
||||
`manual` and `pending` dinners are never listed, and a pasted link is
|
||||
never replaced by a search. Success sets grounding_status `grounded` as
|
||||
the `ground_dinner` job does (the recipe keeps its source URL and site);
|
||||
failure leaves the dinner `failed` with the new `grounding_error`, for
|
||||
the fix-recipe flow. One dinner's failure never stops the others; a
|
||||
dinner renamed, archived or grounded by hand meanwhile is left as it is.
|
||||
Its result is `{dinners: [{dinner_id, outcome, reason}]}`, outcome
|
||||
`grounded`, `failed` (with its reason) or `skipped`. A dinner whose write
|
||||
fails is `failed` with `grounding_error` and left as it was; the others
|
||||
still run, then the job fails with `internal_error`, keeping that result.
|
||||
The responses returning a `dinners.Dinner` representation (the list; the
|
||||
create, update, archive, favorite and grounding responses; the idea
|
||||
responses) carry `photo_backfill_eligible`, and
|
||||
`GET /api/v1/dinners/{id}` does not; it says whether the backfill grounds
|
||||
that dinner, computed by the same condition it lists dinners by, so a
|
||||
client offers a search only where one runs.
|
||||
`GET /api/v1/dinners/{id}/recipe-candidates`
|
||||
searches for the dinner's name and returns `{candidates: [{candidate_id,
|
||||
title, site_name, photo_url, minutes}]}`, at most 3, ranked as `Ground`
|
||||
@@ -788,6 +811,20 @@ purge are fixed and not configurable.
|
||||
|
||||
Runs are recorded in `schedule_runs` and executed through `jobs`.
|
||||
|
||||
Job kinds (`internal/jobs`; each registered when `cmd/server` builds the
|
||||
service named, before the runner starts). `POST /api/v1/jobs` refuses a
|
||||
service-only kind; `GET /api/v1/jobs/{id}` reports any job.
|
||||
|
||||
| Kind | Registered by | Enqueued by | Service-only |
|
||||
| --- | --- | --- | --- |
|
||||
| `noop` | `jobs.RegisterBuiltins` | `POST /api/v1/jobs` (does nothing; proves the runner works) | no |
|
||||
| `ground_dinner` | `dinners.Service` | Adding or renaming a `pending` dinner | no |
|
||||
| `backfill_dinner_photos` | `dinners.Service` | `POST /api/v1/settings/photo-backfill` | yes |
|
||||
| `suggestion_batch` | `suggest.Service` | `POST /api/v1/suggestions/batches`, a deck decision (replenish), `suggestions_weekly` | no |
|
||||
| `draft_week` | `plan.Service` | `POST /api/v1/plans/{weekStart}/draft`, `plan_draft` | yes |
|
||||
| `title_supply_run` | `shopping.Service` | Generating an untitled Supply run | yes |
|
||||
| `ideas_from_ingredients`, `quick_ideas` | `ideas.Service` | `POST /api/v1/ideas/from-ingredients`, `POST /api/v1/ideas/quick` | yes |
|
||||
|
||||
`internal/scheduler` runs them. A schedule registers a name, a rule read
|
||||
from the settings (`scheduler.Daily`, `Weekly` or `WeeklyForNextWeek`, or `DailyAt` for a time read from data) and a
|
||||
run function; settings changes reach it through `SettingsService.Subscribe`
|
||||
@@ -1296,10 +1333,12 @@ fewer the server answers 400 `invalid_settings` naming
|
||||
`onboarding_completed` and changes nothing. Step 3 can be reached with
|
||||
fewer — by its URL, or after another member removes a Regular — so it then
|
||||
shows the server's reason with an "Add Regulars" button back to step 2.
|
||||
Notification permission is never asked for on page load; until Feature 19
|
||||
wires Web Push, "Turn on notifications" asks for nothing and says plainly
|
||||
that notifications are not on yet, offering "Finish setting up" (which
|
||||
completes onboarding as above) instead of a silent success. Offline on the
|
||||
Notification permission is never asked for on page load. "Turn on
|
||||
notifications" subscribes this device to Web Push from the tap and says
|
||||
how it ended — on, blocked by the browser, or unsupported (on iPhone and
|
||||
iPad, until MealQuest is on the Home Screen) — then offers only "Finish
|
||||
setting up" (which completes onboarding as above); a dismissed permission
|
||||
prompt can be asked again. Offline on the
|
||||
saved pages the settings cannot be read (the service worker never caches `/api/v1/settings`),
|
||||
so the saved pages open as they are.
|
||||
|
||||
@@ -1316,6 +1355,41 @@ are undone if the server refuses them. While any Regular is `pending`, the
|
||||
list is read again every 3 seconds (not while a change is in flight), and
|
||||
polling stops once none is.
|
||||
|
||||
## Deployment
|
||||
|
||||
Production is the compose project `mealquest` on 10.60.1.14
|
||||
([`deploy/compose.yaml`](../deploy/compose.yaml)): the `mealquest` image built
|
||||
from the root `Dockerfile`, `litestream` replicating the database to the
|
||||
NAS over SFTP, and `searxng` on the internal network only. Configuration is
|
||||
`deploy/.env` and file secrets in `deploy/secrets/`, neither committed. The
|
||||
services never restart themselves; systemd units in `deploy/systemd/` own
|
||||
the host:
|
||||
|
||||
| Unit | Does |
|
||||
| --- | --- |
|
||||
| `mealquest-firewall.service` | Restricts port 8090 to Traefik in `DOCKER-USER`; required before the stack starts. |
|
||||
| `mealquest.service` | Starts and stops the compose stack. |
|
||||
| `mealquest-media-backup.timer` / `.service` | Backs up cached photos to the NAS nightly at 03:30. |
|
||||
| `mealquest-auto-deploy.timer` / `.service` | Automated deploy of `main` (below). |
|
||||
|
||||
Releases: only `overseer` merges `dev` into `main`. The image carries its
|
||||
version (`git describe`: a release tag, else the short SHA), which
|
||||
`/health` reports. A deploy follows
|
||||
[`deploy/RUNBOOK.md`](../deploy/RUNBOOK.md) — build, pre-deploy restore
|
||||
point, boot-config check, ship and restart, verify the deployed SHA and a
|
||||
real screen — with rollback to the previous release. The automated deploy
|
||||
([`deploy/auto-deploy.sh`](../deploy/auto-deploy.sh), installed as
|
||||
`/usr/local/sbin/mealquest-auto-deploy`) runs those same steps unattended:
|
||||
its timer checks every quarter hour (plus up to 5 minutes' random delay),
|
||||
and deploys `refs/remotes/origin/main` only when it differs from the last
|
||||
SHA in the deploy record `/home/overseer/mealquest-auto-deploy.log`, its
|
||||
Gitea CI status is `success`, its last attempt did not fail, and
|
||||
`mealquest.service` is active. It stops at the first failure and never
|
||||
retries a failed SHA; `--ref <ref>` is the manual override. Installing,
|
||||
seeding the record and enabling the timer are host steps in the runbook's
|
||||
"Automated deploys". Database restore and the boot configuration are in
|
||||
CTXD `RUNBOOKS.MD`.
|
||||
|
||||
## Boundaries and gotchas
|
||||
|
||||
- **Secrets never reach the client** — API keys, OAuth tokens and the Chat
|
||||
|
||||
@@ -214,6 +214,26 @@ func (q *Queries) FailPendingDinnerGrounding(ctx context.Context, arg FailPendin
|
||||
return result.RowsAffected()
|
||||
}
|
||||
|
||||
const getActivePhotoBackfillJob = `-- name: GetActivePhotoBackfillJob :one
|
||||
SELECT id
|
||||
FROM jobs
|
||||
WHERE kind = ?1
|
||||
AND status IN ('queued', 'running')
|
||||
AND json_extract(payload, '$.dinner_id') IS NULL
|
||||
ORDER BY id
|
||||
LIMIT 1
|
||||
`
|
||||
|
||||
// The id of the full photo backfill job (no dinner_id) of kind queued or
|
||||
// running. StartPhotoBackfill enqueues one only when there is none, so
|
||||
// there is at most one. Returns sql.ErrNoRows when there is none.
|
||||
func (q *Queries) GetActivePhotoBackfillJob(ctx context.Context, kind string) (string, error) {
|
||||
row := q.db.QueryRowContext(ctx, getActivePhotoBackfillJob, kind)
|
||||
var id string
|
||||
err := row.Scan(&id)
|
||||
return id, err
|
||||
}
|
||||
|
||||
const getDinner = `-- name: GetDinner :one
|
||||
SELECT id, name, normalized_name, status, origin, method, minutes_total, difficulty, cuisine, is_favorite, recipe_id, photo_id, grounding_status, updated_by, created_at, updated_at, recipe_url, grounding_error, grounding_revision FROM dinners
|
||||
WHERE id = ?
|
||||
@@ -344,6 +364,58 @@ func (q *Queries) GroundDinner(ctx context.Context, arg GroundDinnerParams) (Din
|
||||
return i, err
|
||||
}
|
||||
|
||||
const groundFailedDinner = `-- name: GroundFailedDinner :execrows
|
||||
UPDATE dinners
|
||||
SET recipe_id = ?1,
|
||||
photo_id = ?2,
|
||||
method = COALESCE(method, ?3),
|
||||
minutes_total = COALESCE(minutes_total, ?4),
|
||||
difficulty = COALESCE(difficulty, ?5),
|
||||
grounding_status = 'grounded',
|
||||
grounding_error = NULL,
|
||||
updated_at = ?6
|
||||
WHERE id = ?7
|
||||
AND status <> 'archived'
|
||||
AND grounding_status = 'failed'
|
||||
AND photo_id IS NULL
|
||||
AND recipe_url IS NULL
|
||||
AND grounding_revision = ?8
|
||||
`
|
||||
|
||||
type GroundFailedDinnerParams struct {
|
||||
RecipeID sql.NullString
|
||||
PhotoID sql.NullString
|
||||
Method sql.NullString
|
||||
MinutesTotal sql.NullInt64
|
||||
Difficulty sql.NullString
|
||||
UpdatedAt string
|
||||
ID string
|
||||
GroundingRevision int64
|
||||
}
|
||||
|
||||
// Records a photo backfill's grounding of a dinner it listed, while the
|
||||
// dinner is still unarchived, failed, photoless, without a recipe_url and at
|
||||
// the grounding_revision it was listed at; a dinner grounded by hand,
|
||||
// renamed or archived meanwhile is left as it is (0 rows). Like
|
||||
// GroundPendingDinner it keeps a method, minutes_total or difficulty a
|
||||
// member already set.
|
||||
func (q *Queries) GroundFailedDinner(ctx context.Context, arg GroundFailedDinnerParams) (int64, error) {
|
||||
result, err := q.db.ExecContext(ctx, groundFailedDinner,
|
||||
arg.RecipeID,
|
||||
arg.PhotoID,
|
||||
arg.Method,
|
||||
arg.MinutesTotal,
|
||||
arg.Difficulty,
|
||||
arg.UpdatedAt,
|
||||
arg.ID,
|
||||
arg.GroundingRevision,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return result.RowsAffected()
|
||||
}
|
||||
|
||||
const groundPendingDinner = `-- name: GroundPendingDinner :execrows
|
||||
UPDATE dinners
|
||||
SET recipe_id = ?1,
|
||||
@@ -444,6 +516,64 @@ func (q *Queries) ListDinnersByStatus(ctx context.Context, status string) ([]Din
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const listPhotoBackfillDinners = `-- name: ListPhotoBackfillDinners :many
|
||||
SELECT id, name, normalized_name, status, origin, method, minutes_total, difficulty, cuisine, is_favorite, recipe_id, photo_id, grounding_status, updated_by, created_at, updated_at, recipe_url, grounding_error, grounding_revision FROM dinners
|
||||
WHERE status <> 'archived'
|
||||
AND grounding_status = 'failed'
|
||||
AND photo_id IS NULL
|
||||
AND recipe_url IS NULL
|
||||
AND id = COALESCE(?1, id)
|
||||
ORDER BY normalized_name, id
|
||||
`
|
||||
|
||||
// The dinners a photo backfill grounds by name: unarchived, failed, with no
|
||||
// photo and no pasted recipe_url (the household's own link is never
|
||||
// replaced by a search), ordered by normalized name. A non-NULL id narrows
|
||||
// the list to that dinner. Grounded, manual and pending dinners are never
|
||||
// listed: they have a recipe, or a ground_dinner job grounding them.
|
||||
func (q *Queries) ListPhotoBackfillDinners(ctx context.Context, id sql.NullString) ([]Dinner, error) {
|
||||
rows, err := q.db.QueryContext(ctx, listPhotoBackfillDinners, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var items []Dinner
|
||||
for rows.Next() {
|
||||
var i Dinner
|
||||
if err := rows.Scan(
|
||||
&i.ID,
|
||||
&i.Name,
|
||||
&i.NormalizedName,
|
||||
&i.Status,
|
||||
&i.Origin,
|
||||
&i.Method,
|
||||
&i.MinutesTotal,
|
||||
&i.Difficulty,
|
||||
&i.Cuisine,
|
||||
&i.IsFavorite,
|
||||
&i.RecipeID,
|
||||
&i.PhotoID,
|
||||
&i.GroundingStatus,
|
||||
&i.UpdatedBy,
|
||||
&i.CreatedAt,
|
||||
&i.UpdatedAt,
|
||||
&i.RecipeUrl,
|
||||
&i.GroundingError,
|
||||
&i.GroundingRevision,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, i)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const listUnarchivedDinners = `-- name: ListUnarchivedDinners :many
|
||||
SELECT dinners.id, dinners.name, dinners.normalized_name, dinners.status, dinners.origin, dinners.method, dinners.minutes_total, dinners.difficulty, dinners.cuisine, dinners.is_favorite, dinners.recipe_id, dinners.photo_id, dinners.grounding_status, dinners.updated_by, dinners.created_at, dinners.updated_at, dinners.recipe_url, dinners.grounding_error, dinners.grounding_revision, recipes.source_url AS recipe_source_url
|
||||
FROM dinners
|
||||
@@ -560,6 +690,40 @@ func (q *Queries) ListUnplannedQuestLogDinners(ctx context.Context, fromWeekStar
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const refailDinnerGrounding = `-- name: RefailDinnerGrounding :execrows
|
||||
UPDATE dinners
|
||||
SET grounding_error = ?1,
|
||||
updated_at = ?2
|
||||
WHERE id = ?3
|
||||
AND status <> 'archived'
|
||||
AND grounding_status = 'failed'
|
||||
AND photo_id IS NULL
|
||||
AND recipe_url IS NULL
|
||||
AND grounding_revision = ?4
|
||||
`
|
||||
|
||||
type RefailDinnerGroundingParams struct {
|
||||
GroundingError sql.NullString
|
||||
UpdatedAt string
|
||||
ID string
|
||||
GroundingRevision int64
|
||||
}
|
||||
|
||||
// Records why a photo backfill could not ground a dinner it listed, under
|
||||
// the same conditions as GroundFailedDinner (0 rows otherwise).
|
||||
func (q *Queries) RefailDinnerGrounding(ctx context.Context, arg RefailDinnerGroundingParams) (int64, error) {
|
||||
result, err := q.db.ExecContext(ctx, refailDinnerGrounding,
|
||||
arg.GroundingError,
|
||||
arg.UpdatedAt,
|
||||
arg.ID,
|
||||
arg.GroundingRevision,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return result.RowsAffected()
|
||||
}
|
||||
|
||||
const setDinnerFavorite = `-- name: SetDinnerFavorite :one
|
||||
UPDATE dinners
|
||||
SET is_favorite = ?,
|
||||
|
||||
@@ -117,6 +117,10 @@ type Querier interface {
|
||||
// when there is none, so there is at most one. Returns sql.ErrNoRows when
|
||||
// there is none.
|
||||
GetActiveDraftJob(ctx context.Context, arg GetActiveDraftJobParams) (GetActiveDraftJobRow, error)
|
||||
// The id of the full photo backfill job (no dinner_id) of kind queued or
|
||||
// running. StartPhotoBackfill enqueues one only when there is none, so
|
||||
// there is at most one. Returns sql.ErrNoRows when there is none.
|
||||
GetActivePhotoBackfillJob(ctx context.Context, kind string) (string, error)
|
||||
// The batch still pending or running, oldest first. Returns sql.ErrNoRows
|
||||
// when there is none.
|
||||
GetActiveSuggestionBatch(ctx context.Context) (SuggestionBatch, error)
|
||||
@@ -175,6 +179,13 @@ type Querier interface {
|
||||
// unknown id, returns sql.ErrNoRows. A method, minutes_total or difficulty
|
||||
// no longer as observed is kept: a member edited it while it was grounding.
|
||||
GroundDinner(ctx context.Context, arg GroundDinnerParams) (Dinner, error)
|
||||
// Records a photo backfill's grounding of a dinner it listed, while the
|
||||
// dinner is still unarchived, failed, photoless, without a recipe_url and at
|
||||
// the grounding_revision it was listed at; a dinner grounded by hand,
|
||||
// renamed or archived meanwhile is left as it is (0 rows). Like
|
||||
// GroundPendingDinner it keeps a method, minutes_total or difficulty a
|
||||
// member already set.
|
||||
GroundFailedDinner(ctx context.Context, arg GroundFailedDinnerParams) (int64, error)
|
||||
// Records the background grounding of an unarchived dinner still pending at
|
||||
// the grounding_revision it was grounded for; a dinner grounded some other
|
||||
// way, renamed or archived meanwhile is left as it is (0 rows). A method,
|
||||
@@ -206,6 +217,12 @@ type Querier interface {
|
||||
// The pending Rotation checks whose dinner is still tried, oldest plan
|
||||
// date first, with their dinners.
|
||||
ListPendingRotationReviews(ctx context.Context) ([]ListPendingRotationReviewsRow, error)
|
||||
// The dinners a photo backfill grounds by name: unarchived, failed, with no
|
||||
// photo and no pasted recipe_url (the household's own link is never
|
||||
// replaced by a search), ordered by normalized name. A non-NULL id narrows
|
||||
// the list to that dinner. Grounded, manual and pending dinners are never
|
||||
// listed: they have a recipe, or a ground_dinner job grounding them.
|
||||
ListPhotoBackfillDinners(ctx context.Context, id sql.NullString) ([]Dinner, error)
|
||||
// The days of plan plan_id with their dinners, Monday first. A day with
|
||||
// no dinner is left out.
|
||||
ListPlanDays(ctx context.Context, planID string) ([]ListPlanDaysRow, error)
|
||||
@@ -307,6 +324,9 @@ type Querier interface {
|
||||
// Records a new draft over plan id: its status and note, raising its
|
||||
// revision.
|
||||
RedraftWeekPlan(ctx context.Context, arg RedraftWeekPlanParams) error
|
||||
// Records why a photo backfill could not ground a dinner it listed, under
|
||||
// the same conditions as GroundFailedDinner (0 rows otherwise).
|
||||
RefailDinnerGrounding(ctx context.Context, arg RefailDinnerGroundingParams) (int64, error)
|
||||
// Records a generation of list id from plan plan_id.
|
||||
RegenerateShoppingList(ctx context.Context, arg RegenerateShoppingListParams) error
|
||||
// Records a failed delivery and when to try again. The lease must still be
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
package dinners
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/db"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
|
||||
)
|
||||
|
||||
// KindBackfillPhotos is the job that grounds, by name, the dinners left
|
||||
// without a photo: those whose grounding failed and that have no pasted
|
||||
// recipe_url (ListPhotoBackfillDinners).
|
||||
const KindBackfillPhotos = "backfill_dinner_photos"
|
||||
|
||||
// Photo backfill outcomes for one dinner.
|
||||
const (
|
||||
BackfillGrounded = "grounded"
|
||||
BackfillFailed = "failed"
|
||||
// BackfillSkipped: the dinner was grounded by hand, renamed or archived
|
||||
// while the backfill grounded it, so the backfill left it as it is.
|
||||
BackfillSkipped = "skipped"
|
||||
)
|
||||
|
||||
// PhotoBackfillEligible reports whether a photo backfill grounds r: the
|
||||
// condition ListPhotoBackfillDinners lists dinners by, and GroundFailedDinner
|
||||
// and RefailDinnerGrounding write under, for one dinner. The dinner is
|
||||
// unarchived, failed, with no photo and no pasted recipe_url.
|
||||
func PhotoBackfillEligible(r db.Dinner) bool {
|
||||
return r.Status != StatusArchived &&
|
||||
r.GroundingStatus == GroundingFailed &&
|
||||
!r.PhotoID.Valid &&
|
||||
!r.RecipeUrl.Valid
|
||||
}
|
||||
|
||||
// BackfillJob is the KindBackfillPhotos payload. DinnerID narrows the
|
||||
// backfill to that dinner; empty backfills every dinner without a photo.
|
||||
type BackfillJob struct {
|
||||
DinnerID string `json:"dinner_id,omitempty"`
|
||||
}
|
||||
|
||||
// BackfillOutcome is what a photo backfill did for one dinner. Reason is
|
||||
// set only for BackfillFailed: the Reason recorded as its grounding_error.
|
||||
type BackfillOutcome struct {
|
||||
DinnerID string `json:"dinner_id"`
|
||||
Outcome string `json:"outcome"`
|
||||
Reason *string `json:"reason"`
|
||||
}
|
||||
|
||||
// backfillResult is the KindBackfillPhotos result: the outcome of each
|
||||
// dinner the backfill tried, in the order it tried them.
|
||||
type backfillResult struct {
|
||||
Dinners []BackfillOutcome `json:"dinners"`
|
||||
}
|
||||
|
||||
func (r backfillResult) PublicJSON() ([]byte, error) { return json.Marshal(r) }
|
||||
|
||||
// StartPhotoBackfill enqueues a KindBackfillPhotos job over every dinner
|
||||
// without a photo and returns its id, without waiting for it. While a full
|
||||
// backfill is queued or running it enqueues nothing and returns that job's
|
||||
// id instead, so two passes never ground the same dinner at once; the jobs
|
||||
// table holds the claim, so it holds across restarts too.
|
||||
func (s *Service) StartPhotoBackfill(ctx context.Context) (string, error) {
|
||||
var id string
|
||||
created := false
|
||||
err := s.st.WithTx(ctx, func(tx *sql.Tx) error {
|
||||
active, err := db.New(tx).GetActivePhotoBackfillJob(ctx, KindBackfillPhotos)
|
||||
switch {
|
||||
case errors.Is(err, sql.ErrNoRows):
|
||||
case err != nil:
|
||||
return fmt.Errorf("dinners: read active photo backfill: %w", err)
|
||||
default:
|
||||
id = active
|
||||
return nil
|
||||
}
|
||||
id, err = s.backfillJob.EnqueueTx(ctx, tx, BackfillJob{})
|
||||
if err != nil {
|
||||
return fmt.Errorf("dinners: enqueue photo backfill: %w", err)
|
||||
}
|
||||
created = true
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if created {
|
||||
s.backfillJob.Wake()
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// errBackfillNotRecorded fails a photo backfill that could not record the
|
||||
// outcome of one or more of its dinners.
|
||||
var errBackfillNotRecorded = errors.New("dinners: photo backfill: dinner outcomes not recorded")
|
||||
|
||||
// runBackfillJob grounds each dinner ListPhotoBackfillDinners lists for p,
|
||||
// one at a time, with recipes.Grounder.Ground on its name: the first ranked
|
||||
// candidate whose photo caches and whose ingredients normalize grounds it,
|
||||
// with the recipe's source URL and site kept as any grounding keeps them.
|
||||
// Success records the recipe and photo with grounding_status grounded;
|
||||
// failure leaves the dinner failed with the new Reason as its
|
||||
// grounding_error, for the household to fix its recipe by hand. One
|
||||
// dinner's failure, its search's, model's or database write's, never stops
|
||||
// the others. A failure to list the dinners, or the job's context ending,
|
||||
// fails the job at once. A write that failed fails the job once every
|
||||
// dinner has been tried, as a grounding job's failed write does, with the
|
||||
// outcomes published as its result, so the job never reports done for a
|
||||
// dinner whose outcome is not stored.
|
||||
func (s *Service) runBackfillJob(ctx context.Context, p BackfillJob, publish func(backfillResult) error) (backfillResult, error) {
|
||||
q := db.New(s.st.Reader)
|
||||
rows, err := q.ListPhotoBackfillDinners(ctx, sql.NullString{String: p.DinnerID, Valid: p.DinnerID != ""})
|
||||
if err != nil {
|
||||
return backfillResult{}, fmt.Errorf("dinners: photo backfill: list: %w", err)
|
||||
}
|
||||
res := backfillResult{Dinners: make([]BackfillOutcome, 0, len(rows))}
|
||||
grounded, unrecorded := false, false
|
||||
for _, row := range rows {
|
||||
out, recorded, err := s.backfillDinner(ctx, row)
|
||||
if err != nil {
|
||||
return backfillResult{}, err
|
||||
}
|
||||
grounded = grounded || out.Outcome == BackfillGrounded
|
||||
unrecorded = unrecorded || !recorded
|
||||
res.Dinners = append(res.Dinners, out)
|
||||
}
|
||||
if grounded {
|
||||
s.wake()
|
||||
}
|
||||
if unrecorded {
|
||||
if err := publish(res); err != nil {
|
||||
return backfillResult{}, errors.Join(errBackfillNotRecorded, err)
|
||||
}
|
||||
return backfillResult{}, errBackfillNotRecorded
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// backfillDinner grounds row, a dinner ListPhotoBackfillDinners listed, as
|
||||
// runBackfillJob describes, and reports whether its outcome was recorded.
|
||||
// Its error is only the job's context ending.
|
||||
func (s *Service) backfillDinner(ctx context.Context, row db.Dinner) (BackfillOutcome, bool, error) {
|
||||
out := BackfillOutcome{DinnerID: row.ID}
|
||||
w := db.New(s.st.Writer)
|
||||
gr, err := s.grounder.Ground(ctx, row.Name, row.Name)
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
return BackfillOutcome{}, false, ctxErr
|
||||
}
|
||||
if err != nil {
|
||||
reason := reasonFor(err, false)
|
||||
// Only the reason is logged: the error may quote a dependency's
|
||||
// response, which jobs never store, return or log.
|
||||
s.logger.InfoContext(ctx, "dinner photo backfill failed", "dinner_id", row.ID, "reason", reason)
|
||||
n, err := w.RefailDinnerGrounding(ctx, db.RefailDinnerGroundingParams{
|
||||
GroundingError: sql.NullString{String: reason, Valid: true},
|
||||
UpdatedAt: store.Now().String(),
|
||||
ID: row.ID,
|
||||
GroundingRevision: row.GroundingRevision,
|
||||
})
|
||||
out, recorded := s.backfillRecorded(ctx, out, BackfillFailed, reason, n, err)
|
||||
return out, recorded, nil
|
||||
}
|
||||
n, err := w.GroundFailedDinner(ctx, db.GroundFailedDinnerParams{
|
||||
RecipeID: sql.NullString{String: gr.RecipeID, Valid: true},
|
||||
PhotoID: sql.NullString{String: gr.PhotoID, Valid: true},
|
||||
Method: sql.NullString{String: gr.Method, Valid: true},
|
||||
MinutesTotal: nullInt(gr.MinutesTotal),
|
||||
Difficulty: sql.NullString{String: gr.Difficulty, Valid: true},
|
||||
UpdatedAt: store.Now().String(),
|
||||
ID: row.ID,
|
||||
GroundingRevision: row.GroundingRevision,
|
||||
})
|
||||
out, recorded := s.backfillRecorded(ctx, out, BackfillGrounded, "", n, err)
|
||||
return out, recorded, nil
|
||||
}
|
||||
|
||||
// backfillRecorded is out once the backfill recorded outcome (with reason,
|
||||
// for BackfillFailed) on the dinner, writing n rows with err: a dinner that
|
||||
// changed meanwhile (no row) is BackfillSkipped, and one whose write failed
|
||||
// is BackfillFailed with ReasonGroundingError, the dinner left as it was,
|
||||
// and not recorded.
|
||||
func (s *Service) backfillRecorded(ctx context.Context, out BackfillOutcome, outcome, reason string, n int64, err error) (BackfillOutcome, bool) {
|
||||
switch {
|
||||
case err != nil:
|
||||
// The error is a database error, never a dependency's text.
|
||||
s.logger.ErrorContext(ctx, "dinner photo backfill not recorded", "dinner_id", out.DinnerID, "error", err)
|
||||
out.Outcome, reason = BackfillFailed, ReasonGroundingError
|
||||
case n == 0:
|
||||
out.Outcome, reason = BackfillSkipped, ""
|
||||
default:
|
||||
out.Outcome = outcome
|
||||
}
|
||||
if reason != "" {
|
||||
out.Reason = &reason
|
||||
}
|
||||
return out, err == nil
|
||||
}
|
||||
@@ -0,0 +1,425 @@
|
||||
package dinners_test
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"slices"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/ai/harness"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/db"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/dinners"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/jobs"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/recipes"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/recipes/recipestest"
|
||||
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
|
||||
)
|
||||
|
||||
// failedDinner stores a Regular named name whose grounding failed with
|
||||
// no_recipe_found, without a photo or recipe_url, and returns its id.
|
||||
func (e *env) failedDinner(t *testing.T, name string) string {
|
||||
t.Helper()
|
||||
q := db.New(e.st.Writer)
|
||||
now := store.Now().String()
|
||||
row, err := q.CreateDinner(t.Context(), db.CreateDinnerParams{
|
||||
ID: store.NewID(),
|
||||
Name: name,
|
||||
NormalizedName: harness.Normalize(name),
|
||||
Status: dinners.StatusRegular,
|
||||
Origin: "regular",
|
||||
GroundingStatus: dinners.GroundingPending,
|
||||
UpdatedBy: sql.NullString{String: member, Valid: true},
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := q.FailPendingDinnerGrounding(t.Context(), db.FailPendingDinnerGroundingParams{
|
||||
GroundingError: sql.NullString{String: dinners.ReasonNoRecipe, Valid: true},
|
||||
UpdatedAt: now,
|
||||
ID: row.ID,
|
||||
GroundingRevision: row.GroundingRevision,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return row.ID
|
||||
}
|
||||
|
||||
func outcome(id, out string, reason *string) dinners.BackfillOutcome {
|
||||
return dinners.BackfillOutcome{DinnerID: id, Outcome: out, Reason: reason}
|
||||
}
|
||||
|
||||
func checkOutcomes(t *testing.T, got, want []dinners.BackfillOutcome) {
|
||||
t.Helper()
|
||||
gj, _ := json.Marshal(got)
|
||||
wj, _ := json.Marshal(want)
|
||||
if string(gj) != string(wj) {
|
||||
t.Fatalf("outcomes = %s, want %s", gj, wj)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackfillGroundsFirstCandidateWithAPhoto(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newUnstartedEnv(t)
|
||||
// curryA ranks first but its photo is gone; curryB's photo caches.
|
||||
delete(e.images.JPEGs, "https://a.example/curry.jpg")
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 25))
|
||||
id := e.failedDinner(t, "Chicken curry")
|
||||
|
||||
got, err := e.svc.RunBackfillJob(t.Context(), dinners.BackfillJob{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkOutcomes(t, got, []dinners.BackfillOutcome{outcome(id, dinners.BackfillGrounded, nil)})
|
||||
|
||||
row := e.dinner(t, id)
|
||||
if row.GroundingStatus != dinners.GroundingGrounded || !row.PhotoID.Valid || !row.RecipeID.Valid || row.GroundingError.Valid {
|
||||
t.Fatalf("dinner = %+v, want grounded with a photo and no error", row)
|
||||
}
|
||||
rec, err := db.New(e.st.Reader).GetRecipe(t.Context(), row.RecipeID.String)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rec.SourceUrl != curryB || rec.SiteName.String != "B Kitchen" || rec.PhotoID != row.PhotoID {
|
||||
t.Errorf("recipe = %s from %q photo %s, want %s from B Kitchen photo %s", rec.SourceUrl, rec.SiteName.String, rec.PhotoID.String, curryB, row.PhotoID.String)
|
||||
}
|
||||
if got := e.search.Queries(); !slices.Equal(got, []string{"Chicken curry"}) {
|
||||
t.Errorf("searched %q, want the dinner's name", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackfillReusesCachedRecipeAndPhoto(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newUnstartedEnv(t)
|
||||
e.search.Results["Curry chicken"] = []recipes.Candidate{{URL: curryA}}
|
||||
// One reply: the second dinner must reuse the recipe the first stored.
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
first := e.failedDinner(t, "Chicken curry")
|
||||
second := e.failedDinner(t, "Curry chicken")
|
||||
|
||||
got, err := e.svc.RunBackfillJob(t.Context(), dinners.BackfillJob{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkOutcomes(t, got, []dinners.BackfillOutcome{
|
||||
outcome(first, dinners.BackfillGrounded, nil),
|
||||
outcome(second, dinners.BackfillGrounded, nil),
|
||||
})
|
||||
a, b := e.dinner(t, first), e.dinner(t, second)
|
||||
if a.RecipeID != b.RecipeID || a.PhotoID != b.PhotoID || !a.PhotoID.Valid {
|
||||
t.Errorf("dinners grounded in recipe %s photo %s and recipe %s photo %s, want the same cached ones",
|
||||
a.RecipeID.String, a.PhotoID.String, b.RecipeID.String, b.PhotoID.String)
|
||||
}
|
||||
if n := len(e.model.Requests()); n != 1 {
|
||||
t.Errorf("normalize_recipe ran %d times, want 1", n)
|
||||
}
|
||||
if n := slices.Index(e.pages.Fetched(), curryA); n < 0 || slices.Contains(e.pages.Fetched()[n+1:], curryA) {
|
||||
t.Errorf("fetched %q, want %s fetched once", e.pages.Fetched(), curryA)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackfillIsolatesEachDinnersFailure(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newUnstartedEnv(t)
|
||||
e.search.Fail = map[string]error{"Broken search": errors.New("search provider down")}
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
// Listed by normalized name: both failures come before the curry.
|
||||
broken := e.failedDinner(t, "Broken search")
|
||||
curry := e.failedDinner(t, "Chicken curry")
|
||||
absent := e.failedDinner(t, "Abalone surprise")
|
||||
|
||||
got, err := e.svc.RunBackfillJob(t.Context(), dinners.BackfillJob{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkOutcomes(t, got, []dinners.BackfillOutcome{
|
||||
outcome(absent, dinners.BackfillFailed, ptr(dinners.ReasonNoRecipe)),
|
||||
outcome(broken, dinners.BackfillFailed, ptr(dinners.ReasonSearchFailed)),
|
||||
outcome(curry, dinners.BackfillGrounded, nil),
|
||||
})
|
||||
for id, reason := range map[string]string{absent: dinners.ReasonNoRecipe, broken: dinners.ReasonSearchFailed} {
|
||||
row := e.dinner(t, id)
|
||||
if row.GroundingStatus != dinners.GroundingFailed || row.GroundingError.String != reason || row.PhotoID.Valid {
|
||||
t.Errorf("dinner %s = %+v, want failed with %s and no photo", row.Name, row, reason)
|
||||
}
|
||||
}
|
||||
if row := e.dinner(t, curry); row.GroundingStatus != dinners.GroundingGrounded || !row.PhotoID.Valid {
|
||||
t.Errorf("curry = %+v, want grounded with a photo", row)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackfillSkipsDinnersNotMissingAPhoto(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newEnv(t)
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
c, err := e.svc.CreateRegular(t.Context(), dinners.NewRegular{Name: ptr("Chicken curry")}, member)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.waitJob(t, *c.GroundingJobID)
|
||||
if row := e.dinner(t, c.ID); row.GroundingStatus != dinners.GroundingGrounded {
|
||||
t.Fatalf("dinner = %+v, want grounded", row)
|
||||
}
|
||||
// A failed dinner with a pasted link keeps that link for a fix by hand.
|
||||
linked, err := e.svc.CreateRegular(t.Context(), dinners.NewRegular{Name: ptr("Mystery pie"), RecipeURL: ptr("https://d.example/pie")}, member)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.waitJob(t, *linked.GroundingJobID)
|
||||
if row := e.dinner(t, linked.ID); row.GroundingStatus != dinners.GroundingFailed {
|
||||
t.Fatalf("linked dinner = %+v, want failed", row)
|
||||
}
|
||||
archived := e.failedDinner(t, "Old stew")
|
||||
if _, err := e.svc.Archive(t.Context(), archived, member); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
searched := len(e.search.Queries())
|
||||
|
||||
id, err := e.svc.StartPhotoBackfill(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
job := e.waitJob(t, id)
|
||||
if job.Kind != dinners.KindBackfillPhotos || string(job.Result) != `{"dinners":[]}` {
|
||||
t.Fatalf("job = %+v (result %s), want a backfill with no dinners", job, job.Result)
|
||||
}
|
||||
if got := e.search.Queries()[searched:]; len(got) != 0 {
|
||||
t.Errorf("backfill searched %q, want nothing", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPhotoBackfillEligibleIsWhatTheBackfillLists holds the API's
|
||||
// photo_backfill_eligible to ListPhotoBackfillDinners for a dinner the
|
||||
// backfill grounds and for each kind it never does.
|
||||
func TestPhotoBackfillEligibleIsWhatTheBackfillLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newEnv(t)
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
grounded, err := e.svc.CreateRegular(t.Context(), dinners.NewRegular{Name: ptr("Chicken curry")}, member)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.waitJob(t, *grounded.GroundingJobID)
|
||||
linked, err := e.svc.CreateRegular(t.Context(), dinners.NewRegular{Name: ptr("Mystery pie"), RecipeURL: ptr("https://d.example/pie")}, member)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.waitJob(t, *linked.GroundingJobID)
|
||||
failed := e.failedDinner(t, "Abalone surprise")
|
||||
archived, err := e.svc.Archive(t.Context(), e.failedDinner(t, "Old stew"), member)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
shown := map[string]dinners.Dinner{archived.ID: archived}
|
||||
regulars, err := e.svc.List(t.Context(), dinners.StatusRegular)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, d := range regulars {
|
||||
shown[d.ID] = d
|
||||
}
|
||||
q := db.New(e.st.Reader)
|
||||
for id, want := range map[string]bool{grounded.ID: false, linked.ID: false, failed: true, archived.ID: false} {
|
||||
listed, err := q.ListPhotoBackfillDinners(t.Context(), sql.NullString{String: id, Valid: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d, ok := shown[id]
|
||||
if !ok {
|
||||
t.Fatalf("dinner %s not shown", id)
|
||||
}
|
||||
if got := len(listed) == 1; got != want || d.PhotoBackfillEligible != want {
|
||||
t.Errorf("%s: listed %t, photo_backfill_eligible %t, want both %t", d.Name, got, d.PhotoBackfillEligible, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackfillOfOneDinnerGroundsOnlyIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newUnstartedEnv(t)
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
curry := e.failedDinner(t, "Chicken curry")
|
||||
other := e.failedDinner(t, "Abalone surprise")
|
||||
|
||||
got, err := e.svc.RunBackfillJob(t.Context(), dinners.BackfillJob{DinnerID: curry})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkOutcomes(t, got, []dinners.BackfillOutcome{outcome(curry, dinners.BackfillGrounded, nil)})
|
||||
if row := e.dinner(t, other); row.GroundingStatus != dinners.GroundingFailed || row.PhotoID.Valid {
|
||||
t.Errorf("other dinner = %+v, want left failed", row)
|
||||
}
|
||||
if got := e.search.Queries(); !slices.Equal(got, []string{"Chicken curry"}) {
|
||||
t.Errorf("searched %q, want only the dinner's name", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackfillLeavesDinnerChangedWhileGrounding(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newUnstartedEnv(t)
|
||||
id := e.failedDinner(t, "Chicken curry")
|
||||
gate := make(chan struct{})
|
||||
e.search.Gate = gate
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
|
||||
done := make(chan []dinners.BackfillOutcome)
|
||||
go func() {
|
||||
got, err := e.svc.RunBackfillJob(t.Context(), dinners.BackfillJob{DinnerID: id})
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
done <- got
|
||||
}()
|
||||
// The backfill listed the dinner and waits at its search.
|
||||
for len(e.search.Queries()) == 0 {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
if _, err := e.svc.Update(t.Context(), id, dinners.Patch{Name: ptr("Chicken korma")}, member); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
close(gate)
|
||||
checkOutcomes(t, <-done, []dinners.BackfillOutcome{outcome(id, dinners.BackfillSkipped, nil)})
|
||||
if row := e.dinner(t, id); row.GroundingStatus != dinners.GroundingFailed || row.PhotoID.Valid {
|
||||
t.Errorf("dinner = %+v, want left failed without a photo", row)
|
||||
}
|
||||
}
|
||||
|
||||
// triggerAtOnce calls svc.StartPhotoBackfill from n goroutines released
|
||||
// together and returns the job ids they got.
|
||||
func triggerAtOnce(t *testing.T, svc *dinners.Service, n int) []string {
|
||||
t.Helper()
|
||||
ids := make([]string, n)
|
||||
errs := make([]error, n)
|
||||
release := make(chan struct{})
|
||||
var wg sync.WaitGroup
|
||||
for i := range n {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
<-release
|
||||
ids[i], errs[i] = svc.StartPhotoBackfill(t.Context())
|
||||
}()
|
||||
}
|
||||
close(release)
|
||||
wg.Wait()
|
||||
if err := errors.Join(errs...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// backfillJobs counts the KindBackfillPhotos jobs in st.
|
||||
func backfillJobs(t *testing.T, st *store.Store) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
if err := st.Reader.QueryRowContext(t.Context(), `SELECT count(*) FROM jobs WHERE kind = ?`, dinners.KindBackfillPhotos).Scan(&n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func TestStartPhotoBackfillIsSingleFlight(t *testing.T) {
|
||||
t.Parallel()
|
||||
const triggers = 8
|
||||
dataDir := t.TempDir()
|
||||
st := openStore(t, dataDir)
|
||||
e := newUnstartedEnvOn(t, st, dataDir)
|
||||
e.failedDinner(t, "Chicken curry")
|
||||
gate := make(chan struct{})
|
||||
e.search.Gate = gate
|
||||
|
||||
// Queued: simultaneous triggers all get the one pass they enqueued.
|
||||
ids := triggerAtOnce(t, e.svc, triggers)
|
||||
first := ids[0]
|
||||
for _, id := range ids {
|
||||
if id != first {
|
||||
t.Fatalf("simultaneous triggers = %q, want one job id", ids)
|
||||
}
|
||||
}
|
||||
if n := backfillJobs(t, st); n != 1 {
|
||||
t.Fatalf("%d backfill jobs after simultaneous triggers, want 1", n)
|
||||
}
|
||||
|
||||
// After a restart the queued pass is still the one: the claim is the
|
||||
// stored job, not the process's memory.
|
||||
if err := st.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st = openStore(t, dataDir)
|
||||
restarted := newUnstartedEnvOn(t, st, dataDir)
|
||||
restarted.search.Gate = gate
|
||||
restarted.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
if again, err := restarted.svc.StartPhotoBackfill(t.Context()); err != nil || again != first {
|
||||
t.Fatalf("trigger after restart = %q, %v; want the queued %q", again, err, first)
|
||||
}
|
||||
|
||||
// Running: the pass waits at its search, and simultaneous triggers all
|
||||
// return it.
|
||||
restarted.start(t)
|
||||
for len(restarted.search.Queries()) == 0 {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
for _, id := range triggerAtOnce(t, restarted.svc, triggers) {
|
||||
if id != first {
|
||||
t.Fatalf("trigger while running = %q, want the running %q", id, first)
|
||||
}
|
||||
}
|
||||
if n := backfillJobs(t, st); n != 1 {
|
||||
t.Fatalf("%d backfill jobs, want 1", n)
|
||||
}
|
||||
|
||||
// Finished: the next trigger starts a new pass.
|
||||
close(gate)
|
||||
if job := restarted.waitJob(t, first); job.Status != jobs.StatusSucceeded {
|
||||
t.Fatalf("job = %+v (result %s), want succeeded", job, job.Result)
|
||||
}
|
||||
next, err := restarted.svc.StartPhotoBackfill(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if next == first {
|
||||
t.Fatalf("trigger after the pass finished = %q, want a new job", next)
|
||||
}
|
||||
restarted.waitJob(t, next)
|
||||
}
|
||||
|
||||
func TestBackfillFailsJobWhenAWriteFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newEnv(t)
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
absent := e.failedDinner(t, "Abalone surprise")
|
||||
curry := e.failedDinner(t, "Chicken curry")
|
||||
if _, err := e.st.Writer.ExecContext(t.Context(), `CREATE TRIGGER refuse_grounded BEFORE UPDATE OF grounding_status ON dinners
|
||||
WHEN NEW.id = '`+curry+`' AND NEW.grounding_status = 'grounded'
|
||||
BEGIN SELECT RAISE(ABORT, 'grounded refused'); END`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
id, err := e.svc.StartPhotoBackfill(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
job := e.waitJob(t, id)
|
||||
if job.Status != jobs.StatusFailed || job.Error == nil || *job.Error != jobs.CodeInternal {
|
||||
t.Fatalf("job = %+v, want failed with %s", job, jobs.CodeInternal)
|
||||
}
|
||||
var got struct {
|
||||
Dinners []dinners.BackfillOutcome `json:"dinners"`
|
||||
}
|
||||
if err := json.Unmarshal(job.Result, &got); err != nil {
|
||||
t.Fatalf("result %s: %v", job.Result, err)
|
||||
}
|
||||
// The other dinner was still tried and recorded.
|
||||
checkOutcomes(t, got.Dinners, []dinners.BackfillOutcome{
|
||||
outcome(absent, dinners.BackfillFailed, ptr(dinners.ReasonNoRecipe)),
|
||||
outcome(curry, dinners.BackfillFailed, ptr(dinners.ReasonGroundingError)),
|
||||
})
|
||||
if row := e.dinner(t, curry); row.GroundingStatus != dinners.GroundingFailed || row.PhotoID.Valid {
|
||||
t.Errorf("curry = %+v, want left failed without a photo", row)
|
||||
}
|
||||
}
|
||||
@@ -79,6 +79,11 @@ type Dinner struct {
|
||||
UpdatedBy *string `json:"updated_by"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
|
||||
// PhotoBackfillEligible is whether a photo backfill grounds the dinner
|
||||
// (PhotoBackfillEligible): false for one with a pasted recipe link,
|
||||
// which a search never replaces.
|
||||
PhotoBackfillEligible bool `json:"photo_backfill_eligible"`
|
||||
}
|
||||
|
||||
// NewRegular is a Regular to add: its Name, which the household always
|
||||
@@ -138,6 +143,8 @@ type Service struct {
|
||||
grounder *recipes.Grounder
|
||||
logger *slog.Logger
|
||||
groundJob *jobs.Kind[GroundJob]
|
||||
// backfillJob enqueues KindBackfillPhotos jobs.
|
||||
backfillJob *jobs.Kind[BackfillJob]
|
||||
// wake is called after every committed change to a dinner's method or
|
||||
// minutes, which can move its crock-pot start reminder.
|
||||
wake func()
|
||||
@@ -162,8 +169,8 @@ type Service struct {
|
||||
|
||||
// NewService returns a service over st that grounds dinners with grounder,
|
||||
// calls wake (the scheduler's Wake) after every committed change to a
|
||||
// dinner's method or minutes, and registers the KindGroundDinner job on
|
||||
// runner, which must not have started yet.
|
||||
// dinner's method or minutes, and registers the KindGroundDinner and
|
||||
// KindBackfillPhotos jobs on runner, which must not have started yet.
|
||||
func NewService(st *store.Store, grounder *recipes.Grounder, runner *jobs.Runner, wake func(), logger *slog.Logger) (*Service, error) {
|
||||
switch {
|
||||
case st == nil:
|
||||
@@ -184,6 +191,14 @@ func NewService(st *store.Store, grounder *recipes.Grounder, runner *jobs.Runner
|
||||
return nil, err
|
||||
}
|
||||
s.groundJob = kind
|
||||
backfill, err := jobs.Register(runner, KindBackfillPhotos, s.runBackfillJob)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := backfill.ServiceOnly(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.backfillJob = backfill
|
||||
return s, nil
|
||||
}
|
||||
|
||||
@@ -702,6 +717,8 @@ func fromRow(r db.Dinner) Dinner {
|
||||
UpdatedBy: nullString(r.UpdatedBy),
|
||||
CreatedAt: r.CreatedAt,
|
||||
UpdatedAt: r.UpdatedAt,
|
||||
|
||||
PhotoBackfillEligible: PhotoBackfillEligible(r),
|
||||
}
|
||||
if r.MinutesTotal.Valid {
|
||||
d.MinutesTotal = &r.MinutesTotal.Int64
|
||||
|
||||
@@ -74,3 +74,18 @@ func (s *Service) RunGroundJob(ctx context.Context, p GroundJob) (string, error)
|
||||
|
||||
// SafeSourceURL is safeSourceURL.
|
||||
var SafeSourceURL = safeSourceURL
|
||||
|
||||
// RunBackfillJob runs the KindBackfillPhotos handler for p in the caller's
|
||||
// goroutine, as the job runner would, and returns its outcomes: on an
|
||||
// error, those it last published.
|
||||
func (s *Service) RunBackfillJob(ctx context.Context, p BackfillJob) ([]BackfillOutcome, error) {
|
||||
var published backfillResult
|
||||
r, err := s.runBackfillJob(ctx, p, func(partial backfillResult) error {
|
||||
published = partial
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return published.Dinners, err
|
||||
}
|
||||
return r.Dinners, nil
|
||||
}
|
||||
|
||||
@@ -39,6 +39,7 @@ type env struct {
|
||||
runner *jobs.Runner
|
||||
search *recipestest.Search
|
||||
pages *recipestest.Pages
|
||||
images *recipestest.Images
|
||||
model *llm.Fake
|
||||
}
|
||||
|
||||
@@ -108,7 +109,7 @@ func newUnstartedEnvOn(t *testing.T, st *store.Store, dataDir string) *env {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return &env{svc: svc, st: st, runner: runner, search: search, pages: pages, model: model}
|
||||
return &env{svc: svc, st: st, runner: runner, search: search, pages: pages, images: images, model: model}
|
||||
}
|
||||
|
||||
// start starts the env's job runner, stopping it when the test ends.
|
||||
|
||||
@@ -112,6 +112,20 @@ func postDinner(logger *slog.Logger, svc *dinners.Service) http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// postPhotoBackfill starts a photo backfill over every dinner without a
|
||||
// photo and answers 202 with its job id, which GET /jobs/{id} reports on,
|
||||
// without waiting for the job.
|
||||
func postPhotoBackfill(logger *slog.Logger, svc *dinners.Service) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
id, err := svc.StartPhotoBackfill(r.Context())
|
||||
if err != nil {
|
||||
writeDinnerError(w, r, logger, "start photo backfill", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, r, logger, http.StatusAccepted, jobCreated{ID: id})
|
||||
}
|
||||
}
|
||||
|
||||
// patchDinner edits a dinner, recording the session's actor as updated_by.
|
||||
func patchDinner(logger *slog.Logger, svc *dinners.Service) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
@@ -556,3 +556,43 @@ func TestPostDinnerEnqueueFailureAnswersFailedDinner(t *testing.T) {
|
||||
t.Errorf("listed = %v, want failed", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostPhotoBackfillGroundsFailedDinners(t *testing.T) {
|
||||
t.Parallel()
|
||||
e := newDinnersEnv(t).withRecipes(t)
|
||||
e.start(t)
|
||||
|
||||
// No model reply is scripted, so grounding on creation fails.
|
||||
rec, created := callDinners(t, e.h, http.MethodPost, dinnersPath, `{"name":"Chicken curry"}`)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("POST = %d: %s", rec.Code, rec.Body)
|
||||
}
|
||||
id := created["id"].(string)
|
||||
waitJob(t, e.h, created["grounding_job_id"].(string))
|
||||
if d := listedDinner(t, e.h, id); d["grounding_status"] != "failed" || d["photo_url"] != nil {
|
||||
t.Fatalf("dinner = %v, want failed without a photo", d)
|
||||
}
|
||||
|
||||
e.model.QueueGenerate(recipestest.NormalizedReply("stovetop", 40))
|
||||
rec, started := callDinners(t, e.h, http.MethodPost, "/api/v1/settings/photo-backfill", "")
|
||||
jobID, _ := started["id"].(string)
|
||||
if rec.Code != http.StatusAccepted || jobID == "" {
|
||||
t.Fatalf("POST photo-backfill = %d: %s, want 202 with a job id", rec.Code, rec.Body)
|
||||
}
|
||||
job := waitJob(t, e.h, jobID)
|
||||
want := fmt.Sprintf(`map[dinners:[map[dinner_id:%s outcome:grounded reason:<nil>]]]`, id)
|
||||
if job["status"] != "succeeded" || job["kind"] != "backfill_dinner_photos" || fmt.Sprint(job["result"]) != want {
|
||||
t.Fatalf("job = %v, want backfill_dinner_photos succeeded with %s", job, want)
|
||||
}
|
||||
d := listedDinner(t, e.h, id)
|
||||
photo, _ := d["photo_url"].(string)
|
||||
if d["grounding_status"] != "grounded" || d["grounding_error"] != nil || !strings.HasSuffix(photo, "-400.jpg") {
|
||||
t.Errorf("dinner = %v, want grounded with a photo", d)
|
||||
}
|
||||
|
||||
// The backfill is started only by its own endpoint.
|
||||
rec, _ = callDinners(t, e.h, http.MethodPost, "/api/v1/jobs", `{"kind":"backfill_dinner_photos","payload":{}}`)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "service_only_kind") {
|
||||
t.Errorf("POST /jobs backfill = %d: %s, want 400 service_only_kind", rec.Code, rec.Body)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -162,6 +162,7 @@ func NewRouter(logger *slog.Logger, version string, trusted forwarded.Trusted, a
|
||||
apiV1.Get("/settings/chat-webhook", getChatWebhook(logger, chatSvc))
|
||||
apiV1.Put("/settings/chat-webhook", putChatWebhook(logger, chatSvc))
|
||||
apiV1.Post("/settings/chat-webhook/test", postChatWebhookTest(logger, chatSvc))
|
||||
apiV1.Post("/settings/photo-backfill", postPhotoBackfill(logger, dinnerSvc))
|
||||
apiV1.Get("/dinners", getDinners(logger, dinnerSvc))
|
||||
apiV1.Post("/dinners", postDinner(logger, dinnerSvc))
|
||||
apiV1.Get("/dinners/{id}", getDinner(logger, dinnerSvc))
|
||||
|
||||
@@ -181,3 +181,65 @@ WHERE grounding_status = 'pending'
|
||||
AND json_extract(jobs.payload, '$.dinner_id') = dinners.id
|
||||
AND json_extract(jobs.payload, '$.revision') = dinners.grounding_revision
|
||||
);
|
||||
|
||||
-- name: ListPhotoBackfillDinners :many
|
||||
-- The dinners a photo backfill grounds by name: unarchived, failed, with no
|
||||
-- photo and no pasted recipe_url (the household's own link is never
|
||||
-- replaced by a search), ordered by normalized name. A non-NULL id narrows
|
||||
-- the list to that dinner. Grounded, manual and pending dinners are never
|
||||
-- listed: they have a recipe, or a ground_dinner job grounding them.
|
||||
SELECT * FROM dinners
|
||||
WHERE status <> 'archived'
|
||||
AND grounding_status = 'failed'
|
||||
AND photo_id IS NULL
|
||||
AND recipe_url IS NULL
|
||||
AND id = COALESCE(sqlc.narg(id), id)
|
||||
ORDER BY normalized_name, id;
|
||||
|
||||
-- name: GetActivePhotoBackfillJob :one
|
||||
-- The id of the full photo backfill job (no dinner_id) of kind queued or
|
||||
-- running. StartPhotoBackfill enqueues one only when there is none, so
|
||||
-- there is at most one. Returns sql.ErrNoRows when there is none.
|
||||
SELECT id
|
||||
FROM jobs
|
||||
WHERE kind = sqlc.arg(kind)
|
||||
AND status IN ('queued', 'running')
|
||||
AND json_extract(payload, '$.dinner_id') IS NULL
|
||||
ORDER BY id
|
||||
LIMIT 1;
|
||||
|
||||
-- name: GroundFailedDinner :execrows
|
||||
-- Records a photo backfill's grounding of a dinner it listed, while the
|
||||
-- dinner is still unarchived, failed, photoless, without a recipe_url and at
|
||||
-- the grounding_revision it was listed at; a dinner grounded by hand,
|
||||
-- renamed or archived meanwhile is left as it is (0 rows). Like
|
||||
-- GroundPendingDinner it keeps a method, minutes_total or difficulty a
|
||||
-- member already set.
|
||||
UPDATE dinners
|
||||
SET recipe_id = sqlc.arg(recipe_id),
|
||||
photo_id = sqlc.arg(photo_id),
|
||||
method = COALESCE(method, sqlc.arg(method)),
|
||||
minutes_total = COALESCE(minutes_total, sqlc.arg(minutes_total)),
|
||||
difficulty = COALESCE(difficulty, sqlc.arg(difficulty)),
|
||||
grounding_status = 'grounded',
|
||||
grounding_error = NULL,
|
||||
updated_at = sqlc.arg(updated_at)
|
||||
WHERE id = sqlc.arg(id)
|
||||
AND status <> 'archived'
|
||||
AND grounding_status = 'failed'
|
||||
AND photo_id IS NULL
|
||||
AND recipe_url IS NULL
|
||||
AND grounding_revision = sqlc.arg(grounding_revision);
|
||||
|
||||
-- name: RefailDinnerGrounding :execrows
|
||||
-- Records why a photo backfill could not ground a dinner it listed, under
|
||||
-- the same conditions as GroundFailedDinner (0 rows otherwise).
|
||||
UPDATE dinners
|
||||
SET grounding_error = sqlc.arg(grounding_error),
|
||||
updated_at = sqlc.arg(updated_at)
|
||||
WHERE id = sqlc.arg(id)
|
||||
AND status <> 'archived'
|
||||
AND grounding_status = 'failed'
|
||||
AND photo_id IS NULL
|
||||
AND recipe_url IS NULL
|
||||
AND grounding_revision = sqlc.arg(grounding_revision);
|
||||
|
||||
@@ -52,6 +52,8 @@ function listed(is_favorite: boolean) {
|
||||
is_favorite,
|
||||
photo_url: null,
|
||||
grounding_status: 'grounded',
|
||||
grounding_error: null,
|
||||
photo_backfill_eligible: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+18
-2
@@ -20,8 +20,12 @@ export type Difficulty = (typeof DIFFICULTIES)[number]
|
||||
export type GroundingStatus = (typeof GROUNDING_STATUSES)[number]
|
||||
|
||||
// Regular is the part of a dinner the Regulars screens read. photo_id is
|
||||
// the stored photo's id, from photo_url. pending is true while an
|
||||
// optimistic add waits for the server; its id is not the server's.
|
||||
// the stored photo's id, from photo_url. grounding_error is why its
|
||||
// grounding failed, a reason code, set only while grounding_status is
|
||||
// failed. photo_backfill_eligible is whether a photo backfill (Search
|
||||
// again) searches for it: never for a dinner with a pasted recipe link.
|
||||
// pending is true while an optimistic add waits for the server;
|
||||
// its id is not the server's.
|
||||
export type Regular = {
|
||||
id: string
|
||||
name: string
|
||||
@@ -31,6 +35,8 @@ export type Regular = {
|
||||
is_favorite: boolean
|
||||
photo_id: string | undefined
|
||||
grounding_status: GroundingStatus
|
||||
grounding_error: string | null
|
||||
photo_backfill_eligible: boolean
|
||||
pending?: true
|
||||
}
|
||||
|
||||
@@ -113,6 +119,9 @@ export function parseDinner(where: string, body: unknown): Regular {
|
||||
throw new Error(`${where}: response is not a dinner`)
|
||||
}
|
||||
if (typeof body.is_favorite !== 'boolean') throw new Error(`${where}: is_favorite is not a boolean`)
|
||||
if (typeof body.photo_backfill_eligible !== 'boolean') {
|
||||
throw new Error(`${where}: photo_backfill_eligible is not a boolean`)
|
||||
}
|
||||
return {
|
||||
id: body.id,
|
||||
name: body.name,
|
||||
@@ -122,6 +131,11 @@ export function parseDinner(where: string, body: unknown): Regular {
|
||||
is_favorite: body.is_favorite,
|
||||
photo_id: photoId(where, body.photo_url),
|
||||
grounding_status: oneOf(where, 'grounding_status', GROUNDING_STATUSES, body.grounding_status),
|
||||
grounding_error: nullable(body.grounding_error, (v) => {
|
||||
if (typeof v !== 'string') throw new Error(`${where}: grounding_error is not a string`)
|
||||
return v
|
||||
}),
|
||||
photo_backfill_eligible: body.photo_backfill_eligible,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -225,6 +239,8 @@ export function useAddRegular() {
|
||||
is_favorite: false,
|
||||
photo_id: undefined,
|
||||
grounding_status: 'pending',
|
||||
grounding_error: null,
|
||||
photo_backfill_eligible: false,
|
||||
pending: true,
|
||||
}
|
||||
await changeOptimistically(queryClient, (list) => [...list, optimistic])
|
||||
|
||||
@@ -114,6 +114,8 @@ function dinner(id: string, name: string, difficulty: string | null, minutes: nu
|
||||
is_favorite: false,
|
||||
photo_url: null,
|
||||
grounding_status: 'grounded',
|
||||
grounding_error: null,
|
||||
photo_backfill_eligible: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -300,3 +300,27 @@
|
||||
.lock {
|
||||
display: contents;
|
||||
}
|
||||
|
||||
/* Find missing photos: its hint, button and progress, padded like a sheet. */
|
||||
.card.backfill {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 12px;
|
||||
padding: 16px;
|
||||
}
|
||||
|
||||
/* Why a Regular's recipe was not found. */
|
||||
.reason {
|
||||
margin: 0;
|
||||
padding: 12px 14px;
|
||||
border-radius: 18px;
|
||||
background: var(--color-neutral-100);
|
||||
color: var(--color-neutral-900);
|
||||
font-family: var(--font-body);
|
||||
font-size: 14px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
|
||||
.reasonLabel {
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
@@ -85,7 +85,11 @@ const TACOS = dinnerBody('d1', 'Beef tacos', {
|
||||
photo_url: '/media/photos/p1-400.jpg',
|
||||
})
|
||||
const CHILI = dinnerBody('d2', 'Chili', { method: 'crockpot', minutes_total: 480, grounding_status: 'pending' })
|
||||
const PIZZA = dinnerBody('d3', 'Pizza night', { grounding_status: 'failed', grounding_error: 'no_recipe_found' })
|
||||
const PIZZA = dinnerBody('d3', 'Pizza night', {
|
||||
grounding_status: 'failed',
|
||||
grounding_error: 'no_recipe_found',
|
||||
photo_backfill_eligible: true,
|
||||
})
|
||||
const ROAST = dinnerBody('d4', 'Pot roast', { method: 'oven', grounding_status: 'manual' })
|
||||
|
||||
function rows() {
|
||||
@@ -496,6 +500,204 @@ describe('RegularsSettingsPage', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('Photo backfill', () => {
|
||||
const BACKFILL = 'POST /api/v1/settings/photo-backfill'
|
||||
const JOB = 'GET /api/v1/jobs/j9'
|
||||
|
||||
// backfillJob is a GET /api/v1/jobs/j9 answer for the photo backfill.
|
||||
function backfillJob(status: string, result: unknown = null, error: string | null = null) {
|
||||
return jsonResponse(200, { id: 'j9', kind: 'backfill_dinner_photos', status, result, error })
|
||||
}
|
||||
|
||||
// polls answers each poll of the job with the next answer, then the last
|
||||
// one again; an answer may change the server first.
|
||||
function polls(...answers: (() => Response)[]) {
|
||||
let next = 0
|
||||
return () => answers[Math.min(next++, answers.length - 1)]!()
|
||||
}
|
||||
|
||||
function fixSheet() {
|
||||
return within(screen.getByRole('dialog', { name: 'Fix recipe' }))
|
||||
}
|
||||
|
||||
const NO_CANDIDATES = { 'GET /api/v1/dinners/d3/recipe-candidates': () => jsonResponse(200, { candidates: [] }) }
|
||||
|
||||
it('shows why the recipe was not found in the fix-recipe sheet', async () => {
|
||||
stubServer([PIZZA], NO_CANDIDATES)
|
||||
await renderPage()
|
||||
|
||||
openRow('Pizza night')
|
||||
expect(fixSheet().getByText('No recipe page the search found had a recipe we could use.')).toBeInTheDocument()
|
||||
expect(fixSheet().getByRole('button', { name: 'Search again' })).toBeEnabled()
|
||||
})
|
||||
|
||||
it('offers no search again for a pasted link that had no usable recipe', async () => {
|
||||
const linked = { ...PIZZA, grounding_error: 'recipe_unusable', photo_backfill_eligible: false }
|
||||
stubServer([linked], NO_CANDIDATES)
|
||||
await renderPage()
|
||||
|
||||
openRow('Pizza night')
|
||||
expect(fixSheet().getByText('The pasted recipe link has no recipe we could use.')).toBeInTheDocument()
|
||||
expect(fixSheet().queryByRole('button', { name: 'Search again' })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('offers no search again for a dinner with a pasted link whatever its reason', async () => {
|
||||
const linked = { ...PIZZA, grounding_error: 'ai_failed', photo_backfill_eligible: false }
|
||||
stubServer([linked], NO_CANDIDATES)
|
||||
await renderPage()
|
||||
|
||||
openRow('Pizza night')
|
||||
expect(fixSheet().getByText("The AI assistant couldn't read the recipe.")).toBeInTheDocument()
|
||||
expect(fixSheet().queryByRole('button', { name: 'Search again' })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('searches again from the fix-recipe sheet and shows the new reason when it still fails', async () => {
|
||||
const server = stubServer([PIZZA], {
|
||||
...NO_CANDIDATES,
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: polls(
|
||||
() => backfillJob('running'),
|
||||
() => {
|
||||
server.regulars = [{ ...PIZZA, grounding_error: 'search_failed' }]
|
||||
return backfillJob('succeeded', { dinners: [{ dinner_id: 'd3', outcome: 'failed', reason: 'search_failed' }] })
|
||||
},
|
||||
),
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
openRow('Pizza night')
|
||||
fireEvent.click(fixSheet().getByRole('button', { name: 'Search again' }))
|
||||
expect(await fixSheet().findByRole('button', { name: 'Searching again…' })).toBeDisabled()
|
||||
expect(fixSheet().getByText('Searching again… this can take a few minutes.')).toBeInTheDocument()
|
||||
expect(sent(server.fetchMock)).toEqual([{ request: BACKFILL, body: undefined }])
|
||||
|
||||
expect(await fixSheet().findByText('Searched again: still no recipe found.', {}, { timeout: 3000 })).toBeInTheDocument()
|
||||
expect(await fixSheet().findByText("The recipe search couldn't be reached.")).toBeInTheDocument()
|
||||
expect(fixSheet().getByRole('button', { name: 'Search again' })).toBeEnabled()
|
||||
})
|
||||
|
||||
it('closes the fix-recipe sheet once searching again finds the recipe and its photo', async () => {
|
||||
const server = stubServer([PIZZA], {
|
||||
...NO_CANDIDATES,
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: () => {
|
||||
server.regulars = [{ ...PIZZA, grounding_status: 'grounded', grounding_error: null, photo_url: '/media/photos/p8-400.jpg' }]
|
||||
return backfillJob('succeeded', { dinners: [{ dinner_id: 'd3', outcome: 'grounded', reason: null }] })
|
||||
},
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
openRow('Pizza night')
|
||||
fireEvent.click(fixSheet().getByRole('button', { name: 'Search again' }))
|
||||
await waitFor(() => expect(screen.queryByRole('dialog')).not.toBeInTheDocument())
|
||||
expect(within(row('Pizza night')).getByText('Recipe found')).toBeInTheDocument()
|
||||
expect(within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })).toHaveAttribute('src', '/media/photos/p8-400.jpg')
|
||||
})
|
||||
|
||||
it('says when a dinner was not included in the search', async () => {
|
||||
stubServer([PIZZA], {
|
||||
...NO_CANDIDATES,
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: () => backfillJob('succeeded', { dinners: [] }),
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
openRow('Pizza night')
|
||||
fireEvent.click(fixSheet().getByRole('button', { name: 'Search again' }))
|
||||
expect(await fixSheet().findByText('This dinner was not included in that search.')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('finds missing photos: the placeholder shows until the backfill finds a photo', async () => {
|
||||
const found = { ...PIZZA, grounding_status: 'grounded', grounding_error: null, photo_url: '/media/photos/p8-400.jpg' }
|
||||
const server = stubServer([TACOS, PIZZA], {
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: polls(
|
||||
() => backfillJob('queued'),
|
||||
() => {
|
||||
server.regulars = [TACOS, found]
|
||||
return backfillJob('succeeded', { dinners: [{ dinner_id: 'd3', outcome: 'grounded', reason: null }] })
|
||||
},
|
||||
),
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
// The photoless dinner shows the striped placeholder, not an image.
|
||||
const placeholder = within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })
|
||||
expect(placeholder).not.toHaveAttribute('src')
|
||||
expect(placeholder).toHaveTextContent('dish photo')
|
||||
|
||||
const trigger = screen.getByRole('button', { name: 'Find missing photos' })
|
||||
expectMinHitArea(trigger)
|
||||
fireEvent.click(trigger)
|
||||
expect(await screen.findByRole('button', { name: 'Finding photos…' })).toBeDisabled()
|
||||
expect(await screen.findByText('Finding missing photos… this can take a few minutes.')).toBeInTheDocument()
|
||||
expect(within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })).not.toHaveAttribute('src')
|
||||
expect(sent(server.fetchMock)).toEqual([{ request: BACKFILL, body: undefined }])
|
||||
|
||||
expect(await screen.findByText('Photo search done: 1 photo found.', {}, { timeout: 3000 })).toBeInTheDocument()
|
||||
await waitFor(() =>
|
||||
expect(within(row('Pizza night')).getByRole('img', { name: 'Pizza night' })).toHaveAttribute('src', '/media/photos/p8-400.jpg'),
|
||||
)
|
||||
expect(screen.getByRole('button', { name: 'Find missing photos' })).toBeEnabled()
|
||||
expect(reads(server.fetchMock)).toBe(2)
|
||||
})
|
||||
|
||||
it('sums up what the backfill did', async () => {
|
||||
stubServer([PIZZA], {
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: () =>
|
||||
backfillJob('succeeded', {
|
||||
dinners: [
|
||||
{ dinner_id: 'd3', outcome: 'grounded', reason: null },
|
||||
{ dinner_id: 'd5', outcome: 'grounded', reason: null },
|
||||
{ dinner_id: 'd6', outcome: 'failed', reason: 'no_recipe_found' },
|
||||
{ dinner_id: 'd7', outcome: 'skipped', reason: null },
|
||||
],
|
||||
}),
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
|
||||
expect(
|
||||
await screen.findByText(
|
||||
'Photo search done: 2 photos found · 1 dinner still needs a recipe · 1 dinner changed meanwhile, left as it was.',
|
||||
),
|
||||
).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('says when no dinner was missing a photo', async () => {
|
||||
stubServer([TACOS], {
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: () => backfillJob('succeeded', { dinners: [] }),
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
|
||||
expect(await screen.findByText('No dinners were missing a photo.')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('says why the backfill stopped', async () => {
|
||||
stubServer([PIZZA], {
|
||||
[BACKFILL]: () => jsonResponse(202, { id: 'j9' }),
|
||||
[JOB]: () => backfillJob('failed', { dinners: [] }, 'internal_error'),
|
||||
})
|
||||
await renderPage()
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
|
||||
expect(await screen.findByRole('alert')).toHaveTextContent('Photo search stopped: Something went wrong on the server.')
|
||||
expect(screen.getByRole('button', { name: 'Find missing photos' })).toBeEnabled()
|
||||
})
|
||||
|
||||
it('says why the backfill could not start', async () => {
|
||||
stubServer([PIZZA], { [BACKFILL]: () => errorResponse(403, 'csrf_failed') })
|
||||
await renderPage()
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: 'Find missing photos' }))
|
||||
expect(await screen.findByRole('alert')).toHaveTextContent('Photo search not started: csrf_failed message')
|
||||
expect(screen.getByRole('button', { name: 'Find missing photos' })).toBeEnabled()
|
||||
})
|
||||
})
|
||||
|
||||
it('refreshes grounding status while any Regular is finding its recipe, then stops', async () => {
|
||||
// Only intervals are faked: TanStack Query polls with setInterval, and
|
||||
// Testing Library's waits keep their real timers.
|
||||
|
||||
@@ -34,6 +34,7 @@ import { Sheet } from '../ui/Sheet.tsx'
|
||||
import { Skeleton } from '../ui/Skeleton.tsx'
|
||||
import { Tag } from '../ui/Tag.tsx'
|
||||
import { TextInput } from '../ui/TextInput.tsx'
|
||||
import { usePhotoBackfill, type BackfillResult, type PhotoBackfill } from './photoBackfill.ts'
|
||||
import styles from './RegularsSettingsPage.module.css'
|
||||
|
||||
// The most minutes a dinner may take, as the dinners API allows: a day.
|
||||
@@ -57,7 +58,8 @@ const LINK = /^https?:\/\//i
|
||||
// method and minutes, favorite heart and whether its recipe was found. The
|
||||
// field at the top adds one by name, or by pasted recipe link with a name
|
||||
// the household gives it; tapping one
|
||||
// edits or removes it, or fixes its recipe when none was found.
|
||||
// edits or removes it, or fixes its recipe when none was found. Find
|
||||
// missing photos searches again for every dinner left without one.
|
||||
export function RegularsSettingsPage() {
|
||||
const navigate = useNavigate()
|
||||
return (
|
||||
@@ -86,6 +88,7 @@ function RegularsManager() {
|
||||
const remove = useRemoveRegular()
|
||||
const favorite = useToggleFavorite()
|
||||
const { refusals, track } = useRefusals()
|
||||
const backfill = usePhotoBackfill()
|
||||
const [draft, setDraft] = useState('')
|
||||
const [linkName, setLinkName] = useState('')
|
||||
const [open, setOpen] = useState<Open>(null)
|
||||
@@ -94,8 +97,10 @@ function RegularsManager() {
|
||||
const isLink = LINK.test(draft.trim())
|
||||
const ready = draft.trim() !== '' && (!isLink || linkName.trim() !== '')
|
||||
const list = regulars.data ?? []
|
||||
// A Regular removed meanwhile closes its sheet.
|
||||
// A Regular removed meanwhile closes its sheet, and one whose recipe was
|
||||
// found meanwhile closes its fix-recipe sheet.
|
||||
const opened = open && list.find((regular) => regular.id === open.id && !regular.pending)
|
||||
const fixing = opened && open.sheet === 'fix' && opened.grounding_status === 'failed' ? opened : undefined
|
||||
|
||||
function onSubmit(event: FormEvent<HTMLFormElement>) {
|
||||
event.preventDefault()
|
||||
@@ -188,6 +193,7 @@ function RegularsManager() {
|
||||
)}
|
||||
</section>
|
||||
)}
|
||||
{regulars.isSuccess && <FindMissingPhotos backfill={backfill} />}
|
||||
|
||||
{opened && open.sheet === 'edit' && (
|
||||
<EditRegularSheet
|
||||
@@ -197,12 +203,13 @@ function RegularsManager() {
|
||||
onRemove={() => onRemove(opened)}
|
||||
/>
|
||||
)}
|
||||
{opened && open.sheet === 'fix' && (
|
||||
{fixing && (
|
||||
<FixRecipeSheet
|
||||
key={opened.id}
|
||||
regular={opened}
|
||||
key={fixing.id}
|
||||
regular={fixing}
|
||||
backfill={backfill}
|
||||
onClose={() => setOpen(null)}
|
||||
onEdit={() => setOpen({ sheet: 'edit', id: opened.id })}
|
||||
onEdit={() => setOpen({ sheet: 'edit', id: fixing.id })}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
@@ -231,6 +238,88 @@ function GroundingBadge({ status }: { status: GroundingStatus }) {
|
||||
return <Tag className={`${styles.badge} ${badge.className}`}>{badge.text}</Tag>
|
||||
}
|
||||
|
||||
// groundingReasons says why a dinner's recipe was not found, by its
|
||||
// grounding_error reason code.
|
||||
const groundingReasons: Record<string, string> = {
|
||||
no_recipe_found: 'No recipe page the search found had a recipe we could use.',
|
||||
recipe_unusable: 'The pasted recipe link has no recipe we could use.',
|
||||
search_not_configured: 'Recipe search is not set up. An admin can set it up in Settings.',
|
||||
search_failed: "The recipe search couldn't be reached.",
|
||||
ai_failed: "The AI assistant couldn't read the recipe.",
|
||||
interrupted: 'The server stopped while finding the recipe.',
|
||||
grounding_error: 'Something went wrong on the server.',
|
||||
}
|
||||
|
||||
// groundingReason says why a dinner's recipe was not found; an unknown
|
||||
// reason code is shown as it is.
|
||||
export function groundingReason(code: string): string {
|
||||
return groundingReasons[code] ?? code
|
||||
}
|
||||
|
||||
// backfillJobFailures says why a photo backfill job failed, by its failure
|
||||
// code; an unknown code is shown as it is.
|
||||
const backfillJobFailures: Record<string, string> = {
|
||||
internal_error: 'Something went wrong on the server.',
|
||||
}
|
||||
|
||||
// backfillProblem says what went wrong with a photo backfill, or null.
|
||||
function backfillProblem(backfill: PhotoBackfill): string | null {
|
||||
if (backfill.startError) return `Photo search not started: ${backfill.startError.message}`
|
||||
if (backfill.status === 'failed' && backfill.error) {
|
||||
return `Photo search stopped: ${backfillJobFailures[backfill.error.message] ?? backfill.error.message}`
|
||||
}
|
||||
if (backfill.error) return `Couldn't check on the photo search: ${backfill.error.message}`
|
||||
return null
|
||||
}
|
||||
|
||||
function count(n: number, one: string, many: string): string {
|
||||
return `${n} ${n === 1 ? one : many}`
|
||||
}
|
||||
|
||||
// backfillSummary says what a finished photo backfill did.
|
||||
function backfillSummary(result: BackfillResult): string {
|
||||
const tally = (outcome: string) => result.dinners.filter((dinner) => dinner.outcome === outcome).length
|
||||
if (result.dinners.length === 0) return 'No dinners were missing a photo.'
|
||||
const parts = [count(tally('grounded'), 'photo found', 'photos found')]
|
||||
const failed = tally('failed')
|
||||
if (failed > 0) parts.push(`${count(failed, 'dinner still needs', 'dinners still need')} a recipe`)
|
||||
const skipped = tally('skipped')
|
||||
if (skipped > 0) parts.push(`${count(skipped, 'dinner', 'dinners')} changed meanwhile, left as it was`)
|
||||
return `Photo search done: ${parts.join(' · ')}.`
|
||||
}
|
||||
|
||||
// FindMissingPhotos searches again, by name, for a recipe and photo for
|
||||
// every dinner left without a photo because its recipe was not found.
|
||||
function FindMissingPhotos({ backfill }: { backfill: PhotoBackfill }) {
|
||||
const headingId = useId()
|
||||
const problem = backfillProblem(backfill)
|
||||
return (
|
||||
<section aria-labelledby={headingId} className={styles.section}>
|
||||
<h2 id={headingId} className={styles.groupLabel}>
|
||||
Missing photos
|
||||
</h2>
|
||||
<Card className={`${styles.card} ${styles.backfill}`}>
|
||||
<p className={styles.hint}>
|
||||
Search again for a recipe and photo for every dinner whose recipe wasn't found. Dinners with a pasted
|
||||
recipe link keep it.
|
||||
</p>
|
||||
<Button disabled={backfill.active} onClick={backfill.start}>
|
||||
{backfill.active ? 'Finding photos…' : 'Find missing photos'}
|
||||
</Button>
|
||||
<p role="status" className={styles.status}>
|
||||
{backfill.active && 'Finding missing photos… this can take a few minutes.'}
|
||||
{backfill.status === 'succeeded' && backfill.result && backfillSummary(backfill.result)}
|
||||
</p>
|
||||
{problem && (
|
||||
<p role="alert" className={styles.problem}>
|
||||
{problem}
|
||||
</p>
|
||||
)}
|
||||
</Card>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
type RegularRowProps = {
|
||||
regular: Regular
|
||||
onOpen: () => void
|
||||
@@ -447,6 +536,7 @@ const fixPaths = [
|
||||
|
||||
type FixRecipeSheetProps = {
|
||||
regular: Regular
|
||||
backfill: PhotoBackfill
|
||||
onClose: () => void
|
||||
onEdit: () => void
|
||||
}
|
||||
@@ -459,10 +549,20 @@ function lines(text: string): string[] {
|
||||
.filter((line) => line !== '')
|
||||
}
|
||||
|
||||
// FixRecipeSheet grounds a Regular whose recipe was not found: in one of
|
||||
// up to three recipe pages found for its name, in a pasted recipe link, or
|
||||
// in ingredients and steps typed in.
|
||||
function FixRecipeSheet({ regular, onClose, onEdit }: FixRecipeSheetProps) {
|
||||
// retryOutcomes says what searching again did for this dinner.
|
||||
const retryOutcomes = {
|
||||
grounded: 'Searched again: a recipe and photo were found.',
|
||||
failed: 'Searched again: still no recipe found.',
|
||||
skipped: 'It changed while it was searched again, so it was left as it is.',
|
||||
} as const
|
||||
|
||||
// FixRecipeSheet shows why a Regular's recipe was not found and grounds it:
|
||||
// by searching its name again (a photo backfill), in one of up to three
|
||||
// recipe pages found for its name, in a pasted recipe link, or in
|
||||
// ingredients and steps typed in. Searching again is offered only for a
|
||||
// dinner the backfill searches (photo_backfill_eligible): it never replaces
|
||||
// a pasted link.
|
||||
function FixRecipeSheet({ regular, backfill, onClose, onEdit }: FixRecipeSheetProps) {
|
||||
const [path, setPath] = useState<FixPath>('pick')
|
||||
const ground = useGroundRegular()
|
||||
|
||||
@@ -475,6 +575,15 @@ function FixRecipeSheet({ regular, onClose, onEdit }: FixRecipeSheetProps) {
|
||||
<p className={styles.hint}>
|
||||
We couldn't find a recipe for {regular.name}. Pick a match below, paste a recipe link or type it in.
|
||||
</p>
|
||||
{regular.grounding_error !== null && (
|
||||
<p className={styles.reason}>
|
||||
<span className={styles.reasonLabel}>Why: </span>
|
||||
{groundingReason(regular.grounding_error)}
|
||||
</p>
|
||||
)}
|
||||
{regular.photo_backfill_eligible && (
|
||||
<RetrySearch id={regular.id} backfill={backfill} disabled={ground.isPending} />
|
||||
)}
|
||||
<Segmented
|
||||
label="How to fix the recipe"
|
||||
options={fixPaths}
|
||||
@@ -504,6 +613,34 @@ function FixRecipeSheet({ regular, onClose, onEdit }: FixRecipeSheetProps) {
|
||||
)
|
||||
}
|
||||
|
||||
// RetrySearch searches the dinner's name again with a photo backfill, and
|
||||
// says what it did for the dinner once it ends. The backfill searches
|
||||
// every dinner without a photo, this one with them.
|
||||
function RetrySearch({ id, backfill, disabled }: { id: string; backfill: PhotoBackfill; disabled: boolean }) {
|
||||
const problem = backfillProblem(backfill)
|
||||
const tried = backfill.result?.dinners.find((dinner) => dinner.dinner_id === id)
|
||||
let status: string | null = null
|
||||
if (backfill.active) status = 'Searching again… this can take a few minutes.'
|
||||
else if (backfill.status === 'succeeded') {
|
||||
status = tried ? retryOutcomes[tried.outcome] : 'This dinner was not included in that search.'
|
||||
}
|
||||
return (
|
||||
<>
|
||||
<Button variant="outline" disabled={disabled || backfill.active} onClick={backfill.start}>
|
||||
{backfill.active ? 'Searching again…' : 'Search again'}
|
||||
</Button>
|
||||
<p role="status" className={styles.status}>
|
||||
{status}
|
||||
</p>
|
||||
{problem && (
|
||||
<p role="alert" className={styles.problem}>
|
||||
{problem}
|
||||
</p>
|
||||
)}
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
function PickCandidate({ id, onPick }: { id: string; onPick: (candidateId: string) => void }) {
|
||||
const candidates = useRecipeCandidates(id)
|
||||
if (candidates.isPending) return <Skeleton className={styles.candidatesLoading} />
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import { QueryClientProvider } from '@tanstack/react-query'
|
||||
import { act, renderHook, waitFor } from '@testing-library/react'
|
||||
import type { ReactNode } from 'react'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { ApiProvider, createApiClient, createQueryClient, createVerifiedSession } from '../api/index.ts'
|
||||
import { HOME_KEY } from '../home/api.ts'
|
||||
import { planKey } from '../plan/api.ts'
|
||||
import { createNetworkStatus } from '../pwa/networkStatus.ts'
|
||||
import { fakeLocation, jsonResponse, ME, stubFetch } from '../test/fetchStub.ts'
|
||||
import { parseBackfillResult, usePhotoBackfill } from './photoBackfill.ts'
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
const RESULT = { dinners: [{ dinner_id: 'd3', outcome: 'grounded', reason: null }] }
|
||||
|
||||
function backfillJob(status: string, result: unknown) {
|
||||
return jsonResponse(200, { id: 'j9', kind: 'backfill_dinner_photos', status, result, error: null })
|
||||
}
|
||||
|
||||
describe('usePhotoBackfill', () => {
|
||||
it('reads the plans, home and dinners again once the backfill ends, so a found photo shows everywhere', async () => {
|
||||
const fetchMock = stubFetch(
|
||||
jsonResponse(200, ME),
|
||||
jsonResponse(202, { id: 'j9' }),
|
||||
backfillJob('running', null),
|
||||
backfillJob('succeeded', RESULT),
|
||||
)
|
||||
const api = createApiClient(fakeLocation(), createNetworkStatus(window), createVerifiedSession(localStorage))
|
||||
await api.loadMe()
|
||||
const queryClient = createQueryClient()
|
||||
const plan = planKey('2026-10-05')
|
||||
for (const key of [plan, HOME_KEY, ['dinners', 'quest_log']]) queryClient.setQueryData(key, {})
|
||||
const wrapper = ({ children }: { children: ReactNode }) => (
|
||||
<ApiProvider client={api}>
|
||||
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
|
||||
</ApiProvider>
|
||||
)
|
||||
const { result } = renderHook(() => usePhotoBackfill(), { wrapper })
|
||||
|
||||
act(() => result.current.start())
|
||||
await waitFor(() => expect(result.current.status).toBe('running'))
|
||||
expect(result.current.active).toBe(true)
|
||||
expect(queryClient.getQueryState(plan)?.isInvalidated).toBe(false)
|
||||
|
||||
await waitFor(() => expect(result.current.status).toBe('succeeded'), { timeout: 3000 })
|
||||
expect(result.current).toMatchObject({ active: false, error: null, result: RESULT })
|
||||
expect(fetchMock).toHaveBeenCalledTimes(4)
|
||||
for (const key of [plan, HOME_KEY, ['dinners', 'quest_log']]) {
|
||||
expect(queryClient.getQueryState(key)?.isInvalidated).toBe(true)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('parseBackfillResult', () => {
|
||||
it('refuses a result that is not a photo backfill', () => {
|
||||
expect(() => parseBackfillResult('job', { dinners: [{ dinner_id: 'd3', outcome: 'lost', reason: null }] })).toThrow(
|
||||
'job: outcome "lost" is not one of grounded, failed, skipped',
|
||||
)
|
||||
expect(() => parseBackfillResult('job', { stage: 'x' })).toThrow('job: result is not a photo backfill')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,110 @@
|
||||
import { useMutation, useQueryClient } from '@tanstack/react-query'
|
||||
import { useEffect, useMemo, useState } from 'react'
|
||||
import { useApi, useJob, type JobStatus } from '../api/index.ts'
|
||||
import { isRecord } from '../api/errors.ts'
|
||||
import { oneOf } from '../dinners/api.ts'
|
||||
import { HOME_KEY } from '../home/api.ts'
|
||||
|
||||
export const PHOTO_BACKFILL_PATH = '/api/v1/settings/photo-backfill'
|
||||
|
||||
// BACKFILL_JOB_KIND is the kind of the job a photo backfill runs as.
|
||||
export const BACKFILL_JOB_KIND = 'backfill_dinner_photos'
|
||||
|
||||
export const BACKFILL_OUTCOMES = ['grounded', 'failed', 'skipped'] as const
|
||||
export type BackfillOutcome = (typeof BACKFILL_OUTCOMES)[number]
|
||||
|
||||
// BackfillDinner is what a photo backfill did for one dinner. reason is
|
||||
// set only when it failed: the reason code now its grounding_error.
|
||||
export type BackfillDinner = { dinner_id: string; outcome: BackfillOutcome; reason: string | null }
|
||||
|
||||
// BackfillResult is the photo backfill job's result: each dinner it tried,
|
||||
// in the order it tried them.
|
||||
export type BackfillResult = { dinners: BackfillDinner[] }
|
||||
|
||||
// Every query showing a dinner's photo, read again once a backfill ends:
|
||||
// the dinners lists and details, the plans and the home dashboard.
|
||||
const PHOTO_KEYS = [['dinners'], ['plans'], HOME_KEY] as const
|
||||
|
||||
// parseBackfillResult checks a photo backfill job's result, which useJob
|
||||
// passes on unread.
|
||||
export function parseBackfillResult(where: string, body: unknown): BackfillResult {
|
||||
if (!isRecord(body) || !Array.isArray(body.dinners)) throw new Error(`${where}: result is not a photo backfill`)
|
||||
return {
|
||||
dinners: body.dinners.map((dinner): BackfillDinner => {
|
||||
if (!isRecord(dinner) || typeof dinner.dinner_id !== 'string' || (dinner.reason !== null && typeof dinner.reason !== 'string')) {
|
||||
throw new Error(`${where}: result has a dinner that is not a backfill outcome`)
|
||||
}
|
||||
return {
|
||||
dinner_id: dinner.dinner_id,
|
||||
outcome: oneOf(where, 'outcome', BACKFILL_OUTCOMES, dinner.outcome),
|
||||
reason: dinner.reason,
|
||||
}
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
// PhotoBackfill is a photo backfill started from this screen, as it goes.
|
||||
export type PhotoBackfill = {
|
||||
// start asks the server for a backfill over every dinner without a
|
||||
// photo. While one is already queued or running the server answers with
|
||||
// that job, so starting never runs two at once.
|
||||
start: () => void
|
||||
// active is true from start until the job ends, except while a poll's
|
||||
// failure leaves its progress unknown: starting again then asks the
|
||||
// server for the job it is running and polls it anew.
|
||||
active: boolean
|
||||
// Why the backfill could not be started.
|
||||
startError: Error | null
|
||||
// The job's status, undefined until it is started and first polled.
|
||||
status: JobStatus | undefined
|
||||
// The job's result, once it has one.
|
||||
result: BackfillResult | null
|
||||
// The job's failure, a poll's failure, or a result that is not a backfill.
|
||||
error: Error | null
|
||||
}
|
||||
|
||||
// usePhotoBackfill starts a photo backfill (POST
|
||||
// /api/v1/settings/photo-backfill, 202 {id}) and follows its job with
|
||||
// useJob. When the job ends, every query showing a dinner's photo is read
|
||||
// again, so dinners it grounded show their photo everywhere.
|
||||
export function usePhotoBackfill(): PhotoBackfill {
|
||||
const { apiFetch } = useApi()
|
||||
const queryClient = useQueryClient()
|
||||
const [jobId, setJobId] = useState<string | null>(null)
|
||||
const job = useJob<unknown>(BACKFILL_JOB_KIND, jobId)
|
||||
const begin = useMutation({
|
||||
mutationFn: async (): Promise<string> => {
|
||||
const body = await apiFetch<unknown>(PHOTO_BACKFILL_PATH, { method: 'POST' })
|
||||
if (!isRecord(body) || typeof body.id !== 'string') throw new Error(`POST ${PHOTO_BACKFILL_PATH}: response has no job id`)
|
||||
return body.id
|
||||
},
|
||||
onSuccess: (id) => {
|
||||
setJobId(id)
|
||||
job.poll()
|
||||
},
|
||||
})
|
||||
|
||||
const ended = job.status === 'succeeded' || job.status === 'failed'
|
||||
useEffect(() => {
|
||||
if (!ended) return
|
||||
for (const queryKey of PHOTO_KEYS) void queryClient.invalidateQueries({ queryKey })
|
||||
}, [ended, jobId, queryClient])
|
||||
|
||||
const parsed = useMemo(() => {
|
||||
if (job.result === null) return { result: null, error: null }
|
||||
try {
|
||||
return { result: parseBackfillResult(`GET /api/v1/jobs/${jobId}`, job.result), error: null }
|
||||
} catch (error) {
|
||||
return { result: null, error: error as Error }
|
||||
}
|
||||
}, [job.result, jobId])
|
||||
|
||||
return {
|
||||
start: () => begin.mutate(),
|
||||
active: begin.isPending || (jobId !== null && !ended && job.error === null),
|
||||
startError: begin.error,
|
||||
status: job.status,
|
||||
result: parsed.result,
|
||||
error: job.error ?? parsed.error,
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,7 @@ export type DinnerBody = {
|
||||
photo_url: string | null
|
||||
grounding_status: string
|
||||
grounding_error: string | null
|
||||
photo_backfill_eligible: boolean
|
||||
updated_by: string | null
|
||||
created_at: string
|
||||
updated_at: string
|
||||
@@ -35,6 +36,7 @@ export function dinnerBody(id: string, name: string, overrides: Partial<DinnerBo
|
||||
photo_url: null,
|
||||
grounding_status: 'grounded',
|
||||
grounding_error: null,
|
||||
photo_backfill_eligible: false,
|
||||
updated_by: null,
|
||||
created_at: '2026-10-01T15:00:00.000Z',
|
||||
updated_at: '2026-10-01T15:00:00.000Z',
|
||||
|
||||
Reference in New Issue
Block a user