Feature 10: Recipe grounding and photos (#78)

Co-authored-by: dev_agent <dev_agent@cubecraftlabs.com>
2026-10-06 07:27:58 -04:00
parent 67e44a506a
commit f25b28b39b
58 changed files with 7621 additions and 67 deletions
@@ -10,6 +10,11 @@
 #   printf '%s\n' '<Authentik client secret>' > deploy/secrets/oidc_client_secret
 #   sudo chown 10001:10001 deploy/secrets/* && sudo chmod 600 deploy/secrets/*
 #
+# SearXNG's secret key is the exception: compose reads it itself, so it
+# stays owned by you (mode 0600). Create it after the chown above:
+#
+#   (umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env)
+#
 # deploy/compose.yaml mounts them at /run/secrets/<name> and points the
 # *_FILE variables at them.

@@ -62,6 +67,11 @@ MEALQUEST_SESSION_MAX=4320h
 #   Base64 of 32 random bytes; encrypts secrets at rest (AES-256-GCM).
 # MEALQUEST_OIDC_CLIENT_SECRET_FILE=/run/secrets/oidc_client_secret
 #   The Authentik provider's client secret.
+# MEALQUEST_SEARXNG_URL=http://searxng:8080
+#   The SearXNG recipe-search sidecar on the compose network. Planned; not
+#   read by the service yet. The app takes its recipe-search provider and
+#   base URL from PATCH /api/v1/settings/recipe-search; see
+#   deploy/RUNBOOK.md "Recipe search (SearXNG)".

 # --- Planned; not read by the service yet -------------------------------------
 #
@@ -75,6 +85,3 @@ MEALQUEST_SESSION_MAX=4320h
 # Google Calendar OAuth web client (Feature 16).
 # MEALQUEST_GOOGLE_CLIENT_ID=<client id>.apps.googleusercontent.com
 # MEALQUEST_GOOGLE_CLIENT_SECRET_FILE=/run/secrets/google_client_secret
-#
-# SearXNG recipe search sidecar on the compose network (Feature 07).
-# MEALQUEST_SEARXNG_URL=http://searxng:8080
@@ -44,7 +44,8 @@ Elsewhere:

 The compose stack is `deploy/compose.yaml`: the `mealquest` service (built
 from the repository `Dockerfile`, runs as uid/gid 10001, listens on 8080 in
-the container) and the `litestream` service. Data lives in the named volume
+the container), the `litestream` service and the `searxng` recipe-search
+sidecar (see "Recipe search (SearXNG)"). Data lives in the named volume
 `mealquest-data`.

 ### 1. Choose and check the port
@@ -146,15 +147,20 @@ service refuses to start.
 | `secret_key` | base64 of 32 random bytes; encrypts stored secrets (AES-256-GCM) | `/run/secrets/secret_key` |
 | `oidc_client_secret` | Client Secret of the Authentik provider `mealquest` | `/run/secrets/oidc_client_secret` |
 | `litestream_key` | dedicated SSH private key for the NAS (see "Backups") | `/etc/litestream/sftp_key` |
+| `searxng.env` | `SEARXNG_SECRET=<64 hex characters>`; SearXNG's secret key | not mounted; compose passes it as an environment variable |
+
+`searxng.env` is the one exception to the owner rule: compose itself reads
+it (as `env_file`), so it is owned by `overseer`, mode 0600.

 ```sh
 mkdir -p deploy/secrets && chmod 700 deploy/secrets
 head -c 32 /dev/urandom | base64 > deploy/secrets/secret_key
 printf '%s\n' '<Authentik client secret>' > deploy/secrets/oidc_client_secret
 # litestream_key: see "Backups" > "Database — Litestream"
-sudo chown 10001:10001 deploy/secrets/*
+sudo chown 10001:10001 deploy/secrets/secret_key deploy/secrets/oidc_client_secret deploy/secrets/litestream_key
 sudo chmod 600 deploy/secrets/*
-sudo ls -la deploy/secrets   # three files, mealquest mealquest, -rw-------
+(umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env)
+sudo ls -la deploy/secrets   # three files mealquest mealquest, searxng.env overseer overseer; all -rw-------
 ```

 The `mealquest` host user (uid/gid 10001) is created in "Media — nightly
@@ -169,7 +175,7 @@ directory (see "Media — nightly rsync"), then build and start:
 ```sh
 docker compose -f deploy/compose.yaml build
 sudo systemctl enable --now mealquest   # starts mealquest-firewall first
-docker compose -f deploy/compose.yaml ps   # mealquest healthy, litestream up
+docker compose -f deploy/compose.yaml ps   # mealquest healthy, litestream up, searxng healthy
 curl -fsS http://10.60.1.14:8090/health
 curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health   # 200
 ```
@@ -192,6 +198,12 @@ git clean -n deploy      # dry run: must not list .env or secrets/ (both git-ign
 git clean -f deploy
 ```

+On 2026-10-06 `deploy/compose.yaml` was replaced and
+`deploy/compose.dev.yaml` and `deploy/searxng/settings.yml` were copied in
+from the Feature 10 branch (task 641f468b) to start the SearXNG sidecar;
+they too must go before that checkout (`git clean -n -d deploy` lists
+them).
+
 Then follow "Routine deploy"; the checkout brings the tracked copies back.

 ## Authentik
@@ -399,6 +411,71 @@ mealquest`, never with `docker compose up` or `down`. Per-service compose
 commands (`stop mealquest`, `up -d --force-recreate litestream`) are fine
 while the unit is active, because the firewall is already in place.

+## Recipe search (SearXNG)
+
+The `searxng` service in `deploy/compose.yaml` is a private SearXNG
+instance for recipe search. On the compose network it is reachable at
+`http://searxng:8080`; that is the address to configure in the app.
+
+Starting the sidecar does not connect the app to it. The service does not
+read `MEALQUEST_SEARXNG_URL`: `deploy/compose.yaml` sets it for a future
+wiring, and today it does nothing. The recipe-search provider and base URL
+live in the `recipe_search_settings` row and are set through
+`GET`/`PATCH /api/v1/settings/recipe-search` (the API behind the Settings
+screen's recipe search; the screen itself is not built yet). Until that row
+exists, recipe search answers 409 `not_configured`.
+
+After the first deploy, from a signed-in session (every non-`GET` request
+needs the `X-MealQuest-CSRF` header set to `csrfToken` from
+`GET /api/v1/me`):
+
+1. `PATCH /api/v1/settings/recipe-search` with
+   `{"provider": "searxng", "base_url": "http://searxng:8080"}`.
+2. `GET /api/v1/settings/recipe-search` shows `"provider": "searxng"` and
+   `"base_url": "http://searxng:8080"`.
+3. `POST /api/v1/settings/recipe-search/test` searches
+   "chicken tikka masala" and must answer 200 with
+   `{"query": "chicken tikka masala", "count": <n>}`. A 502
+   `search_failed` carries SearXNG's error.
+
+| Item | Value |
+| --- | --- |
+| Image | `searxng/searxng:2026.10.4-d48c4b555`, pinned by digest `sha256:76b0bf285aca014c7191fc4d9234c4bfb358624ac33d8883833d496c059ec072` |
+| Settings | `deploy/searxng/settings.yml`, mounted read-only at `/etc/searxng/settings.yml` |
+| Secret | `deploy/secrets/searxng.env` (`SEARXNG_SECRET`), see "Secrets" |
+| Network | `mealquest` only; **no published port** |
+| Health | `wget http://127.0.0.1:8080/healthz` inside the container |
+
+The settings enable the `json` output format, turn off the public limiter
+and bot detection (the only client is the `mealquest` service), and keep
+only the general web search engines DuckDuckGo, Brave, Google, Bing,
+Mojeek and Startpage. Engines that block the host (CAPTCHA, "too many
+requests") are suspended by SearXNG for a while and listed under
+`unresponsive_engines` in the JSON; the others still answer.
+
+Check from inside the app container (the only place it is reachable):
+
+```sh
+docker compose -f deploy/compose.yaml exec mealquest \
+  wget -qO- 'http://searxng:8080/search?q=chicken+tikka+masala+recipe&format=json'
+docker compose -f deploy/compose.yaml port searxng 8080   # prints ":0": nothing published
+```
+
+On 2026-10-06 that query returned 17 results. From this host four of the
+six engines were blocked (Brave, DuckDuckGo, Google and Startpage were
+rate-limited or served a CAPTCHA), so the results came from Bing and
+Mojeek. SearXNG retries suspended engines after their suspension ends, so
+the set that answers can change. From 10.60.1.85 nothing on 10.60.1.14 answered as SearXNG:
+8888 refused the connection and 8080 is NestQuest's API.
+
+For development, `deploy/compose.dev.yaml` publishes SearXNG on the
+dev machine's loopback only (`127.0.0.1:8888`). It is never used on
+office-srvr.
+
+To move to a newer image, pull the new tag, take its digest from
+`docker image inspect --format '{{json .RepoDigests}}' searxng/searxng:<tag>`,
+update both in `deploy/compose.yaml`, then deploy as usual.
+
 ## Routine deploy and rollback

 Production runs `main` (a release); only `overseer` merges `dev` → `main`.
@@ -413,7 +490,7 @@ git rev-parse --short HEAD          # the SHA being replaced: note it for rollba
 git checkout --detach origin/main   # or a release tag
 docker compose -f deploy/compose.yaml build
 sudo systemctl restart mealquest    # applies the firewall, then down + up -d
-docker compose -f deploy/compose.yaml ps   # mealquest healthy, litestream up
+docker compose -f deploy/compose.yaml ps   # mealquest healthy, litestream up, searxng healthy
 curl -fsS http://10.60.1.14:8090/health
 curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health   # 200
 git rev-parse HEAD                  # the deployed SHA
@@ -868,6 +945,17 @@ pushes. Rotate only when the private key may have leaked, write both
 halves, recreate the service with `--force-recreate mealquest`, and tell
 the household to re-enable notifications on each device.

+### SearXNG secret (`deploy/secrets/searxng.env`)
+
+Only signs SearXNG's own cookies and image-proxy links; nothing stored
+depends on it. Replace it and recreate the service:
+
+```sh
+(umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env)
+docker compose -f deploy/compose.yaml up -d --force-recreate searxng
+docker compose -f deploy/compose.yaml ps searxng   # healthy
+```
+
 ### Litestream / NAS key (`deploy/secrets/litestream_key`)

 Generate a new keypair as in "Database — Litestream", add the new public
@@ -0,0 +1,12 @@
+# Development override: publishes SearXNG on the dev machine's loopback so
+# it can be queried directly, e.g.
+#
+#   docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml up -d searxng
+#   curl 'http://127.0.0.1:8888/search?q=chicken+tikka+masala+recipe&format=json'
+#
+# Loopback only, never a LAN address. Never used in production: the
+# systemd unit starts deploy/compose.yaml alone.
+services:
+  searxng:
+    ports:
+      - "127.0.0.1:8888:8080"
@@ -11,7 +11,8 @@
 # Configuration comes from deploy/.env (copy deploy/.env.example) and secrets
 # from deploy/secrets/; neither is committed. Compose bind-mounts file secrets
 # with their host owner and mode, so each file in deploy/secrets/ must be
-# owned by the container user 10001:10001 with mode 0600.
+# owned by the container user 10001:10001 with mode 0600 (except
+# searxng.env, read by compose itself; see the searxng service).
 name: mealquest

 services:
@@ -31,6 +32,7 @@ services:
      MEALQUEST_DATA_DIR: /data
      MEALQUEST_SECRET_KEY_FILE: /run/secrets/secret_key
      MEALQUEST_OIDC_CLIENT_SECRET_FILE: /run/secrets/oidc_client_secret
+      MEALQUEST_SEARXNG_URL: http://searxng:8080
    secrets:
      - secret_key
      - oidc_client_secret
@@ -85,6 +87,42 @@ services:
    cap_drop:
      - ALL

+  # SearXNG, the recipe-search sidecar (Feature 10; settings:
+  # deploy/searxng/settings.yml). It sits on the mealquest network only and
+  # publishes no port: the mealquest service reaches it at
+  # http://searxng:8080 and nothing outside the compose network can.
+  # deploy/compose.dev.yaml publishes it on 127.0.0.1 for development.
+  #
+  # The secret key comes from deploy/secrets/searxng.env as SEARXNG_SECRET.
+  # Compose itself reads that file (it is not mounted), so unlike the other
+  # secrets it is owned by overseer, mode 0600. It runs as the image's
+  # searxng user (977) with a read-only root filesystem.
+  searxng:
+    image: searxng/searxng:2026.10.4-d48c4b555@sha256:76b0bf285aca014c7191fc4d9234c4bfb358624ac33d8883833d496c059ec072
+    user: "977:977"
+    restart: "no"
+    env_file:
+      - path: ./secrets/searxng.env
+        required: true
+    volumes:
+      - ./searxng/settings.yml:/etc/searxng/settings.yml:ro
+    networks:
+      - mealquest
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/healthz"]
+      interval: 30s
+      timeout: 3s
+      start_period: 15s
+      retries: 3
+    read_only: true
+    tmpfs:
+      - /tmp
+      - /var/cache/searxng:uid=977,gid=977
+    security_opt:
+      - no-new-privileges:true
+    cap_drop:
+      - ALL
+
 secrets:
  secret_key:
    file: ./secrets/secret_key
@@ -0,0 +1,61 @@
+# SearXNG settings for the MealQuest recipe-search sidecar (Feature 10).
+# Mounted read-only at /etc/searxng/settings.yml by deploy/compose.yaml.
+#
+# Internal use only: SearXNG is reachable on the compose network
+# `mealquest` and nowhere else, and its only client is the mealquest
+# service. That is why the public limiter and bot detection are off.
+#
+# The secret key is not in this file. SearXNG reads it from SEARXNG_SECRET,
+# set from deploy/secrets/searxng.env (never committed). Without it the
+# default "ultrasecretkey" stays in place and SearXNG refuses to start.
+#
+# Settings reference: https://docs.searxng.org/admin/settings/
+
+use_default_settings:
+  engines:
+    # General web search engines only; every other default engine is
+    # dropped.
+    keep_only:
+      - duckduckgo
+      - brave
+      - google
+      - bing
+      - mojeek
+      - startpage
+
+general:
+  instance_name: MealQuest search
+
+server:
+  # Internal use: no public limiter and no bot detection.
+  limiter: false
+  public_instance: false
+  image_proxy: false
+
+search:
+  safe_search: 0
+  # No autocomplete: the mealquest service never asks for suggestions.
+  autocomplete: ""
+  # json is the format the mealquest service reads; html stays for
+  # debugging through deploy/compose.dev.yaml.
+  formats:
+    - html
+    - json
+
+# Some general web engines are disabled or inactive by default; enable all
+# of the kept ones.
+engines:
+  - name: duckduckgo
+    disabled: false
+  - name: brave
+    disabled: false
+  - name: google
+    disabled: false
+  - name: bing
+    disabled: false
+  - name: mojeek
+    disabled: false
+    inactive: false
+  - name: startpage
+    disabled: false
+    inactive: false