From 944299b4555ecbcf55f028db2bc1f4ed8f130c4f Mon Sep 17 00:00:00 2001 From: dev_agent Date: Tue, 6 Oct 2026 06:58:46 -0400 Subject: [PATCH] Add SearXNG recipe-search sidecar to the compose stack deploy/searxng/settings.yml enables the json format, turns off the public limiter and bot detection (internal use only) and keeps only general web search engines; the secret key comes from SEARXNG_SECRET in the git-ignored deploy/secrets/searxng.env. deploy/compose.yaml adds the searxng service (digest-pinned image, mealquest network only, no published port, healthcheck, read-only root) and sets MEALQUEST_SEARXNG_URL=http://searxng:8080 for mealquest. deploy/compose.dev.yaml publishes it on 127.0.0.1:8888 for development. RUNBOOK and .env.example document the new secret and checks. Task 641f468b-68e9-42e2-b640-3199984cf75c. Co-Authored-By: Claude Opus 5.5 --- deploy/.env.example | 10 +++-- deploy/RUNBOOK.md | 75 ++++++++++++++++++++++++++++++++++--- deploy/compose.dev.yaml | 12 ++++++ deploy/compose.yaml | 40 +++++++++++++++++++- deploy/searxng/settings.yml | 61 ++++++++++++++++++++++++++++++ 5 files changed, 189 insertions(+), 9 deletions(-) create mode 100644 deploy/compose.dev.yaml create mode 100644 deploy/searxng/settings.yml diff --git a/deploy/.env.example b/deploy/.env.example index d0e969e..10ff213 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -10,6 +10,11 @@ # printf '%s\n' '' > deploy/secrets/oidc_client_secret # sudo chown 10001:10001 deploy/secrets/* && sudo chmod 600 deploy/secrets/* # +# SearXNG's secret key is the exception: compose reads it itself, so it +# stays owned by you (mode 0600). Create it after the chown above: +# +# (umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env) +# # deploy/compose.yaml mounts them at /run/secrets/ and points the # *_FILE variables at them. @@ -62,6 +67,8 @@ MEALQUEST_SESSION_MAX=4320h # Base64 of 32 random bytes; encrypts secrets at rest (AES-256-GCM). # MEALQUEST_OIDC_CLIENT_SECRET_FILE=/run/secrets/oidc_client_secret # The Authentik provider's client secret. +# MEALQUEST_SEARXNG_URL=http://searxng:8080 +# The SearXNG recipe-search sidecar on the compose network. # --- Planned; not read by the service yet ------------------------------------- # @@ -75,6 +82,3 @@ MEALQUEST_SESSION_MAX=4320h # Google Calendar OAuth web client (Feature 16). # MEALQUEST_GOOGLE_CLIENT_ID=.apps.googleusercontent.com # MEALQUEST_GOOGLE_CLIENT_SECRET_FILE=/run/secrets/google_client_secret -# -# SearXNG recipe search sidecar on the compose network (Feature 07). -# MEALQUEST_SEARXNG_URL=http://searxng:8080 diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md index c255c4e..1034307 100644 --- a/deploy/RUNBOOK.md +++ b/deploy/RUNBOOK.md @@ -44,7 +44,8 @@ Elsewhere: The compose stack is `deploy/compose.yaml`: the `mealquest` service (built from the repository `Dockerfile`, runs as uid/gid 10001, listens on 8080 in -the container) and the `litestream` service. Data lives in the named volume +the container), the `litestream` service and the `searxng` recipe-search +sidecar (see "Recipe search (SearXNG)"). Data lives in the named volume `mealquest-data`. ### 1. Choose and check the port @@ -146,15 +147,20 @@ service refuses to start. | `secret_key` | base64 of 32 random bytes; encrypts stored secrets (AES-256-GCM) | `/run/secrets/secret_key` | | `oidc_client_secret` | Client Secret of the Authentik provider `mealquest` | `/run/secrets/oidc_client_secret` | | `litestream_key` | dedicated SSH private key for the NAS (see "Backups") | `/etc/litestream/sftp_key` | +| `searxng.env` | `SEARXNG_SECRET=<64 hex characters>`; SearXNG's secret key | not mounted; compose passes it as an environment variable | + +`searxng.env` is the one exception to the owner rule: compose itself reads +it (as `env_file`), so it is owned by `overseer`, mode 0600. ```sh mkdir -p deploy/secrets && chmod 700 deploy/secrets head -c 32 /dev/urandom | base64 > deploy/secrets/secret_key printf '%s\n' '' > deploy/secrets/oidc_client_secret # litestream_key: see "Backups" > "Database — Litestream" -sudo chown 10001:10001 deploy/secrets/* +sudo chown 10001:10001 deploy/secrets/secret_key deploy/secrets/oidc_client_secret deploy/secrets/litestream_key sudo chmod 600 deploy/secrets/* -sudo ls -la deploy/secrets # three files, mealquest mealquest, -rw------- +(umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env) +sudo ls -la deploy/secrets # three files mealquest mealquest, searxng.env overseer overseer; all -rw------- ``` The `mealquest` host user (uid/gid 10001) is created in "Media — nightly @@ -169,7 +175,7 @@ directory (see "Media — nightly rsync"), then build and start: ```sh docker compose -f deploy/compose.yaml build sudo systemctl enable --now mealquest # starts mealquest-firewall first -docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up +docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up, searxng healthy curl -fsS http://10.60.1.14:8090/health curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 ``` @@ -192,6 +198,12 @@ git clean -n deploy # dry run: must not list .env or secrets/ (both git-ign git clean -f deploy ``` +On 2026-10-06 `deploy/compose.yaml` was replaced and +`deploy/compose.dev.yaml` and `deploy/searxng/settings.yml` were copied in +from the Feature 10 branch (task 641f468b) to start the SearXNG sidecar; +they too must go before that checkout (`git clean -n -d deploy` lists +them). + Then follow "Routine deploy"; the checkout brings the tracked copies back. ## Authentik @@ -399,6 +411,48 @@ mealquest`, never with `docker compose up` or `down`. Per-service compose commands (`stop mealquest`, `up -d --force-recreate litestream`) are fine while the unit is active, because the firewall is already in place. +## Recipe search (SearXNG) + +The `searxng` service in `deploy/compose.yaml` is a private SearXNG +instance that the `mealquest` service queries for recipes +(`MEALQUEST_SEARXNG_URL=http://searxng:8080`, set by `deploy/compose.yaml`). + +| Item | Value | +| --- | --- | +| Image | `searxng/searxng:2026.10.4-d48c4b555`, pinned by digest `sha256:76b0bf285aca014c7191fc4d9234c4bfb358624ac33d8883833d496c059ec072` | +| Settings | `deploy/searxng/settings.yml`, mounted read-only at `/etc/searxng/settings.yml` | +| Secret | `deploy/secrets/searxng.env` (`SEARXNG_SECRET`), see "Secrets" | +| Network | `mealquest` only; **no published port** | +| Health | `wget http://127.0.0.1:8080/healthz` inside the container | + +The settings enable the `json` output format, turn off the public limiter +and bot detection (the only client is the `mealquest` service), and keep +only the general web search engines DuckDuckGo, Brave, Google, Bing, +Mojeek and Startpage. Engines that block the host (CAPTCHA, "too many +requests") are suspended by SearXNG for a while and listed under +`unresponsive_engines` in the JSON; the others still answer. + +Check from inside the app container (the only place it is reachable): + +```sh +docker compose -f deploy/compose.yaml exec mealquest \ + wget -qO- 'http://searxng:8080/search?q=chicken+tikka+masala+recipe&format=json' +docker compose -f deploy/compose.yaml port searxng 8080 # prints ":0": nothing published +``` + +On 2026-10-06 that query returned 17 results (Bing and Mojeek answered; +Brave, DuckDuckGo, Google and Startpage were rate-limited or served a +CAPTCHA). From 10.60.1.85 nothing on 10.60.1.14 answered as SearXNG: +8888 refused the connection and 8080 is NestQuest's API. + +For development, `deploy/compose.dev.yaml` publishes SearXNG on the +dev machine's loopback only (`127.0.0.1:8888`). It is never used on +office-srvr. + +To move to a newer image, pull the new tag, take its digest from +`docker image inspect --format '{{json .RepoDigests}}' searxng/searxng:`, +update both in `deploy/compose.yaml`, then deploy as usual. + ## Routine deploy and rollback Production runs `main` (a release); only `overseer` merges `dev` → `main`. @@ -413,7 +467,7 @@ git rev-parse --short HEAD # the SHA being replaced: note it for rollba git checkout --detach origin/main # or a release tag docker compose -f deploy/compose.yaml build sudo systemctl restart mealquest # applies the firewall, then down + up -d -docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up +docker compose -f deploy/compose.yaml ps # mealquest healthy, litestream up, searxng healthy curl -fsS http://10.60.1.14:8090/health curl -s -o /dev/null -w "%{http_code}\n" https://mealquest.cubecraftlabs.com/health # 200 git rev-parse HEAD # the deployed SHA @@ -868,6 +922,17 @@ pushes. Rotate only when the private key may have leaked, write both halves, recreate the service with `--force-recreate mealquest`, and tell the household to re-enable notifications on each device. +### SearXNG secret (`deploy/secrets/searxng.env`) + +Only signs SearXNG's own cookies and image-proxy links; nothing stored +depends on it. Replace it and recreate the service: + +```sh +(umask 077; printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > deploy/secrets/searxng.env) +docker compose -f deploy/compose.yaml up -d --force-recreate searxng +docker compose -f deploy/compose.yaml ps searxng # healthy +``` + ### Litestream / NAS key (`deploy/secrets/litestream_key`) Generate a new keypair as in "Database — Litestream", add the new public diff --git a/deploy/compose.dev.yaml b/deploy/compose.dev.yaml new file mode 100644 index 0000000..ee071ba --- /dev/null +++ b/deploy/compose.dev.yaml @@ -0,0 +1,12 @@ +# Development override: publishes SearXNG on the dev machine's loopback so +# it can be queried directly, e.g. +# +# docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml up -d searxng +# curl 'http://127.0.0.1:8888/search?q=chicken+tikka+masala+recipe&format=json' +# +# Loopback only, never a LAN address. Never used in production: the +# systemd unit starts deploy/compose.yaml alone. +services: + searxng: + ports: + - "127.0.0.1:8888:8080" diff --git a/deploy/compose.yaml b/deploy/compose.yaml index c5bae84..4fe042a 100644 --- a/deploy/compose.yaml +++ b/deploy/compose.yaml @@ -11,7 +11,8 @@ # Configuration comes from deploy/.env (copy deploy/.env.example) and secrets # from deploy/secrets/; neither is committed. Compose bind-mounts file secrets # with their host owner and mode, so each file in deploy/secrets/ must be -# owned by the container user 10001:10001 with mode 0600. +# owned by the container user 10001:10001 with mode 0600 (except +# searxng.env, read by compose itself; see the searxng service). name: mealquest services: @@ -31,6 +32,7 @@ services: MEALQUEST_DATA_DIR: /data MEALQUEST_SECRET_KEY_FILE: /run/secrets/secret_key MEALQUEST_OIDC_CLIENT_SECRET_FILE: /run/secrets/oidc_client_secret + MEALQUEST_SEARXNG_URL: http://searxng:8080 secrets: - secret_key - oidc_client_secret @@ -85,6 +87,42 @@ services: cap_drop: - ALL + # SearXNG, the recipe-search sidecar (Feature 10; settings: + # deploy/searxng/settings.yml). It sits on the mealquest network only and + # publishes no port: the mealquest service reaches it at + # http://searxng:8080 and nothing outside the compose network can. + # deploy/compose.dev.yaml publishes it on 127.0.0.1 for development. + # + # The secret key comes from deploy/secrets/searxng.env as SEARXNG_SECRET. + # Compose itself reads that file (it is not mounted), so unlike the other + # secrets it is owned by overseer, mode 0600. It runs as the image's + # searxng user (977) with a read-only root filesystem. + searxng: + image: searxng/searxng:2026.10.4-d48c4b555@sha256:76b0bf285aca014c7191fc4d9234c4bfb358624ac33d8883833d496c059ec072 + user: "977:977" + restart: "no" + env_file: + - path: ./secrets/searxng.env + required: true + volumes: + - ./searxng/settings.yml:/etc/searxng/settings.yml:ro + networks: + - mealquest + healthcheck: + test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/healthz"] + interval: 30s + timeout: 3s + start_period: 15s + retries: 3 + read_only: true + tmpfs: + - /tmp + - /var/cache/searxng:uid=977,gid=977 + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + secrets: secret_key: file: ./secrets/secret_key diff --git a/deploy/searxng/settings.yml b/deploy/searxng/settings.yml new file mode 100644 index 0000000..4366d0c --- /dev/null +++ b/deploy/searxng/settings.yml @@ -0,0 +1,61 @@ +# SearXNG settings for the MealQuest recipe-search sidecar (Feature 10). +# Mounted read-only at /etc/searxng/settings.yml by deploy/compose.yaml. +# +# Internal use only: SearXNG is reachable on the compose network +# `mealquest` and nowhere else, and its only client is the mealquest +# service. That is why the public limiter and bot detection are off. +# +# The secret key is not in this file. SearXNG reads it from SEARXNG_SECRET, +# set from deploy/secrets/searxng.env (never committed). Without it the +# default "ultrasecretkey" stays in place and SearXNG refuses to start. +# +# Settings reference: https://docs.searxng.org/admin/settings/ + +use_default_settings: + engines: + # General web search engines only; every other default engine is + # dropped. + keep_only: + - duckduckgo + - brave + - google + - bing + - mojeek + - startpage + +general: + instance_name: MealQuest search + +server: + # Internal use: no public limiter and no bot detection. + limiter: false + public_instance: false + image_proxy: false + +search: + safe_search: 0 + # No autocomplete: the mealquest service never asks for suggestions. + autocomplete: "" + # json is the format the mealquest service reads; html stays for + # debugging through deploy/compose.dev.yaml. + formats: + - html + - json + +# Some general web engines are disabled or inactive by default; enable all +# of the kept ones. +engines: + - name: duckduckgo + disabled: false + - name: brave + disabled: false + - name: google + disabled: false + - name: bing + disabled: false + - name: mojeek + disabled: false + inactive: false + - name: startpage + disabled: false + inactive: false