Maniacs_Mansion/NestQuest
11
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Feature 17: API Deployment and Identity #217

Merged
approver_agent merged 6 commits from feature/911b11a7-bdbb-4305-86f6-d666e181ed51-api-deployment-and-identity into dev 2026-09-24 09:57:08 -04:00
Conversation 1 Commits 6 Files Changed 10
+813
10 changed files with 813 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+27
View File
@@ -0,0 +1,27 @@
# Keep the image build context minimal: only api/ and the bundled
# custom_components/nestquest/core/ are copied (see the Dockerfile);
# everything else — VCS state, virtualenvs, tests, the frontend, and
# any local secrets or database files — never enters the build.
.git
.gitignore
.venv
**/__pycache__
**/*.py[cod]
.pytest_cache
**/*.db
**/*.db-wal
**/*.db-shm
# Local configuration — never baked into an image.
.env
**/.env
deploy
design
docs
frontend
orchestration_tests
scripts
templates
tests
uv.lock
hacs.json
*.md
.gitignore
+13
View File
@@ -5,3 +5,16 @@ __pycache__/
.pytest_cache/
*.metadata.json
frontend/node_modules/
# Feature 17 (API deployment): local secret config and runtime data
# are never committed — compose reads deploy/.env; SQLite/WAL sidecar
# files are container/volume artifacts.
.env
**/.env
*.db
*.db-wal
*.db-shm
# Dedicated Litestream SFTP backup keypair — lives only on the API box.
deploy/*key*
deploy/litestream_key*
Dockerfile
+51
View File
@@ -0,0 +1,51 @@
# Container image for the NestQuest API service (Feature 17).
#
# Build context is the REPOSITORY ROOT (compose sets it), because the
# API loads the bundled domain layer from
# custom_components/nestquest/core/ by file location
# (api/nestquest_core.py): the image must carry both api/ and the
# bundled core at their relative positions. Everything else in the
# repository is excluded via .dockerignore.
#
# Multi-stage so no pip/build toolchain survives into the runtime:
# the builder installs the deployable's own manifest
# (api/requirements.txt) into a staging prefix and the runtime copies
# only the installed packages. The service runs as a non-root user;
# the SQLite database lives on the mounted volume at
# /var/lib/nestquest (compose supplies NESTQUEST_DB_PATH pointing
# there — configuration is environment-supplied, never baked in).
#
# Both stages use the same base image pinned by digest, and
# api/requirements.txt pins exact versions, so a rebuild reproduces
# the deployed image instead of floating to whatever is newest.
FROM python:3.12-slim@sha256:2f17fc044b579bab302c2e8054d3a686e2cb9a83de48e70534b94cd8ebbe06a9 AS builder
COPY api/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir --prefix=/install -r /tmp/requirements.txt
FROM python:3.12-slim@sha256:2f17fc044b579bab302c2e8054d3a686e2cb9a83de48e70534b94cd8ebbe06a9
# The staged dependency install from the builder (no pip, no wheels).
COPY --from=builder /install /usr/local
# /app is the repository-root equivalent: api/ resolves imports as
# ``api.*`` and nestquest_core.py finds the bundled core one level up.
WORKDIR /app
COPY api/ ./api/
COPY custom_components/nestquest/core/ ./custom_components/nestquest/core/
# Non-root runtime user; owns the database volume mount point.
RUN useradd --system --create-home --home-dir /home/nestquest nestquest \
&& mkdir -p /var/lib/nestquest \
&& chown nestquest:nestquest /var/lib/nestquest
USER nestquest
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
# The database path and the service token / OIDC settings are supplied
# by the environment (compose env_file), never committed into the image.
ENV NESTQUEST_DB_PATH=/var/lib/nestquest/nestquest.db
VOLUME /var/lib/nestquest
EXPOSE 8080
CMD ["uvicorn", "--factory", "api.app:create_app", "--host", "0.0.0.0", "--port", "8080"]
api/README.md
+83
View File
@@ -0,0 +1,83 @@
# NestQuest API
Standalone FastAPI service owning the NestQuest database (Feature 16).
Served with ``uvicorn --factory api.app:create_app``; configuration is
environment-supplied (``api/config.py``) and every variable is required.
## Runtime dependencies
``api/requirements.txt`` is the API deployable's OWN dependency
manifest (fastapi, uvicorn, pyjwt[crypto], httpx) — deliberately kept
out of the integration's ``pyproject.toml`` dev group, which exists only
so the monorepo test venv can run the API tests.
## Container deployment (scaffold)
The image is built from the repository ROOT context (the API loads the
bundled domain core from ``custom_components/nestquest/core/`` by file
location, so both trees must be in the build context):
docker build -t nestquest-api:latest .
Run it with compose — the SQLite file lives on the named volume
``nestquest-db`` so recreating the container keeps household data:
cp deploy/.env.example deploy/.env # then fill in real values
docker compose -f deploy/compose.yaml up -d --build
``deploy/.env`` is git-ignored and supplies the secrets (panel service
token, Authentik OIDC settings); ``NESTQUEST_DB_PATH`` is fixed to
``/var/lib/nestquest/nestquest.db`` by compose.
The published port binds ``NESTQUEST_BIND_ADDRESS`` from
``deploy/.env`` and defaults to ``127.0.0.1`` (localhost only). In
production TLS is terminated by the shared Traefik reverse proxy,
which serves ``https://nestquest.cubecraftlabs.com`` (Let's Encrypt,
auto-renewed, HTTP redirected to HTTPS) and forwards to the API box
on port 8080 — so on that box set ``NESTQUEST_BIND_ADDRESS`` to its
LAN address (``10.60.1.14``). Traefik only allows
``/api/v1/panel`` from the Home Assistant host. The full deployment,
restore and verification procedure is in ``deploy/RUNBOOK.md``.
## Continuous backup (Litestream)
Compose also runs a ``litestream`` sidecar
(``litestream/litestream:0.5.17``, config ``deploy/litestream.yml``).
It mounts the same ``nestquest-db`` volume and continuously replicates
``/var/lib/nestquest/nestquest.db`` over SFTP to the Synology NAS
``CubeCraft-SRVR1`` (``joshua@10.60.1.50``), directory
``/volume1/homes/Joshua/nestquest-backups`` (``/homes/Joshua/...`` in
the NAS's chrooted SFTP view) — so the backup is outside the Docker
host and its volume. A snapshot is taken daily and a week of
snapshots plus incremental changes is retained; the NAS host key is
pinned in the config.
The SFTP login uses a dedicated keypair that lives only on the API
box and is git-ignored. Create it once, before ``up``:
ssh-keygen -t ed25519 -N "" -C nestquest-litestream -f deploy/litestream_key
then append ``restrict `` + the contents of ``deploy/litestream_key.pub``
to ``joshua@10.60.1.50:~/.ssh/authorized_keys``. Compose mounts the
private key read-only at ``/etc/litestream/sftp_key``. The sidecar runs
as uid/gid 999 (the API's ``nestquest`` user), so give it the key:
sudo chown 999:999 deploy/litestream_key && chmod 600 deploy/litestream_key
The sidecar needs WRITE access to the database volume: Litestream adds
its own ``_litestream_seq``/``_litestream_lock`` tables to the database,
writes the ``-wal``/``-shm`` files and checkpoints the WAL, and fails
with "unable to open database file" on a read-only mount. That access
is contained rather than removed — the sidecar runs as the non-root
volume owner with a read-only root filesystem, ``no-new-privileges``
and all Linux capabilities dropped.
Check replication with
``docker compose -f deploy/compose.yaml logs litestream`` or
``docker compose -f deploy/compose.yaml exec litestream litestream ltx /var/lib/nestquest/nestquest.db``.
To restore (high level): stop the ``api`` service, run
``litestream restore -config /etc/litestream.yml -o <new path>
/var/lib/nestquest/nestquest.db`` from a litestream container with the
same mounts, verify the restored file, move it into place on the
volume (removing any stale ``-wal``/``-shm`` files), and start ``api``
again.
api/requirements.txt
+38
View File
@@ -0,0 +1,38 @@
# Runtime dependency manifest for the NestQuest API deployable.
#
# The monorepo's pyproject.toml carries fastapi/uvicorn/pyjwt/httpx in
# the INTEGRATION's dev group only (so the test venv can exercise the
# API); they are deliberately NOT integration runtime dependencies.
# Per that manifest's own comment, the API deployable declares its own
# runtime dependencies HERE — this file is what the container image
# installs (see the Dockerfile at the repository root).
#
# EXACT pins: the versions installed in the deployed image
# (nestquest-api:latest on the API box, read with importlib.metadata),
# so a rebuild resolves the same packages. The four direct
# dependencies come first; the rest is their resolved transitive
# closure, pinned too so nothing floats. The dev group's floors in
# pyproject.toml are unchanged. Bump deliberately, together with the
# Dockerfile's digest-pinned base image.
fastapi==0.141.1
uvicorn==0.53.0
pyjwt[crypto]==2.15.0
httpx==0.28.1
# Transitive dependencies of the above.
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.15.1
certifi==2026.7.22
cffi==2.1.1
click==8.5.0
cryptography==50.0.1
h11==0.16.0
httpcore==1.0.9
idna==3.20
pycparser==3.0
pydantic==2.13.5
pydantic_core==2.46.5
starlette==1.7.0
typing-inspection==0.4.4
typing_extensions==4.16.0
deploy/.env.example
+23
View File
@@ -0,0 +1,23 @@
# Example environment for the NestQuest API container deployment.
#
# Copy to ``deploy/.env`` (git-ignored) and fill in REAL values there;
# this example carries placeholder names only and is never a source of
# secrets. Compose loads ``deploy/.env`` into the api service; the
# API fails fast on startup when any of these is unset or empty
# (api/config.py — there are no silent defaults).
# The static panel-plane service token the HA integration presents as
# a Bearer credential on /api/v1/panel/* routes. Generate one, e.g.:
# python -c "import secrets; print(secrets.token_urlsafe(32))"
NESTQUEST_PANEL_TOKEN=
# The Authentik OIDC settings the admin plane verifies JWTs against.
NESTQUEST_OIDC_ISSUER=
NESTQUEST_OIDC_AUDIENCE=
NESTQUEST_OIDC_JWKS_URL=
# The host interface the API's port 8080 is published on (compose
# interpolation, not read by the API). Defaults to 127.0.0.1 when
# unset; on the API box set the LAN address the reverse proxy
# forwards to, e.g. 10.60.1.14.
NESTQUEST_BIND_ADDRESS=
deploy/RUNBOOK.md
+455
View File
@@ -0,0 +1,455 @@
# NestQuest API — deployment and restore runbook
Operator procedure to rebuild the production NestQuest API deployment end
to end from this repository, restore its database from backup, and verify
it. Every value below is the real, non-secret production value. **No
secret appears in this file** — secrets are named and their location is
given, never their content.
| Piece | Where | Real value |
| --- | --- | --- |
| API box (Docker host) | this repository's `deploy/` | `10.60.1.14`, port `8080` |
| Reverse proxy / TLS | shared Traefik on `nukapi-03` | `10.60.1.33` |
| Public hostname | Cloudflare DNS | `nestquest.cubecraftlabs.com` |
| Identity | Authentik | `https://auth.cubecraftlabs.com` |
| Home Assistant box (panel client) | LAN | `10.60.1.80` |
| Local DNS | Pi-hole | `10.60.1.101` |
| Backup target | Synology NAS `CubeCraft-SRVR1`, SFTP | `10.60.1.50:/homes/Joshua/nestquest-backups` |
All `docker compose` commands below are run **on the API box, from the
repository root**. The shorthand `dc` means:
alias dc='docker compose -f deploy/compose.yaml'
Compose v2.24 or newer is required (`env_file` uses the `path`/`required`
form).
---
## 1. Image build
The image is defined by the root `Dockerfile` and is built from the
**repository root** as the build context: the API (`api/`) loads the
bundled domain core from `custom_components/nestquest/core/` by file
location, so both trees must be in the context. `.dockerignore` keeps
everything else (including `deploy/`, `.env` files and `*.db`) out of the
build.
- Multi-stage on `python:3.12-slim`: the builder installs
`api/requirements.txt`, the runtime copies only the installed packages
plus `api/` and `custom_components/nestquest/core/` into `/app`.
- Runs as the non-root system user `nestquest` (uid/gid `999`), which owns
`/var/lib/nestquest`.
- Command: `uvicorn --factory api.app:create_app --host 0.0.0.0 --port 8080`.
Get the code and build:
git clone ssh://code.cubecraftlabs.com:2288/Maniacs_Mansion/NestQuest.git
cd NestQuest
git checkout <release branch or tag being deployed>
dc build api # equivalent: docker build -t nestquest-api:latest .
Confirm the runtime user (the Litestream sidecar and the key ownership in
section 5 depend on uid 999):
docker run --rm --entrypoint id nestquest-api:latest
# uid=999(nestquest) gid=999(nestquest) groups=999(nestquest)
## 2. Compose
`deploy/compose.yaml` defines two services and one named volume:
| Service | Image | Role |
| --- | --- | --- |
| `api` | `nestquest-api:latest` (built, section 1) | FastAPI service on `8080`, healthcheck on `/health` |
| `litestream` | `litestream/litestream:0.5.17` | Continuous SFTP backup of the database (section 7); starts only once `api` is healthy |
- **Volume `nestquest-db`** is mounted at `/var/lib/nestquest` in both
services. The database is `NESTQUEST_DB_PATH=/var/lib/nestquest/nestquest.db`
(fixed by compose, not a secret). `docker compose down` keeps named
volumes; **never run `down -v`** on the API box. Docker prefixes the
volume with the compose project name — with the default project
(`deploy`, the compose file's directory) it is `deploy_nestquest-db`;
confirm with `docker volume ls | grep nestquest-db`.
- **Published port**: `${NESTQUEST_BIND_ADDRESS:-127.0.0.1}:8080:8080`.
The default binds localhost only, so an unconfigured host is never
exposed. On the API box `deploy/.env` sets
`NESTQUEST_BIND_ADDRESS=10.60.1.14` so Traefik on nukapi-03 can reach
`http://10.60.1.14:8080`.
- **Secrets** come from `deploy/.env` (git-ignored, section 5) via
`env_file`.
Bring the stack up (after section 5's secrets exist):
dc up -d --build
dc ps # api "healthy", litestream "Up"
curl -s http://10.60.1.14:8080/health # {"status":"ok"}
## 3. TLS and reverse proxy (Traefik on nukapi-03)
TLS is terminated by the **shared** Traefik instance on `nukapi-03`
(`10.60.1.33`), not by this repository. Its NestQuest routing is the
owner-authored dynamic file-provider config
**`config/dynamic/nestquest-ccl.yml`** in the Traefik deployment on
nukapi-03 — that live file is authoritative; it is not stored in this
repository. What it must contain:
| Router | Rule | Middleware | Service |
| --- | --- | --- | --- |
| panel | ``Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)`` | `ipAllowList`, `sourceRange: ["10.60.1.80/32"]` (HA box only) | `http://10.60.1.14:8080` |
| admin (everything else: `/health`, `/docs`, `/openapi.json`, `/api/v1/admin/*`) | ``Host(`nestquest.cubecraftlabs.com`)`` | none — public; the API itself enforces Authentik JWTs (section 4) | `http://10.60.1.14:8080` |
Both routers use the HTTPS entrypoint `websecure` with `tls.certResolver: myresolver`
(HTTP is redirected to HTTPS by the shared Traefik). The panel router's
rule is longer, so Traefik's default rule-length priority evaluates it
before the catch-all admin router; if you set explicit priorities, keep
panel higher.
Reference shape, to rebuild the file if it is ever lost (entrypoint names
must match the shared Traefik's static config on nukapi-03):
```yaml
http:
routers:
nestquest-panel:
rule: "Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)"
entryPoints: [websecure]
middlewares: [nestquest-panel-allowlist]
service: nestquest-api
tls:
certResolver: myresolver
nestquest-admin:
rule: "Host(`nestquest.cubecraftlabs.com`)"
entryPoints: [websecure]
service: nestquest-api
tls:
certResolver: myresolver
middlewares:
nestquest-panel-allowlist:
ipAllowList:
sourceRange: ["10.60.1.80/32"]
services:
nestquest-api:
loadBalancer:
servers:
- url: "http://10.60.1.14:8080"
```
**Certificates.** `myresolver` is the shared Traefik's ACME (Let's
Encrypt) resolver using the **Cloudflare DNS-01 challenge**; the
Cloudflare API token is held by the Traefik deployment on
nukapi-03, never in this repository. Because DNS-01 proves control
through a DNS TXT record, issuance and renewal need no inbound HTTP and
work regardless of the Cloudflare proxy setting. Certificates renew
automatically.
**Cloudflare proxy note.** Public traffic for
`nestquest.cubecraftlabs.com` resolves through Cloudflare, so a request
from the internet reaches Traefik from a Cloudflare edge address, never
from `10.60.1.80` — which is exactly why the panel allowlist refuses it
(403). The HA box does not go through Cloudflare at all: the local DNS
override in section 6 sends it straight to Traefik on the LAN, so Traefik
sees its real address. Do **not** add Cloudflare ranges to the panel
allowlist or make the allowlist trust `X-Forwarded-For`; that would let
any internet client through to the panel plane.
## 4. Authentik (admin identity)
Admin-plane requests (`/api/v1/admin/*`) carry an Authentik OIDC access
token (JWT). The API verifies issuer, audience, RS256 signature against
the JWKS, and expiry, then requires the `groups` claim to contain
`nestquest-admins` (`api/auth.py`). Configure in Authentik
(`https://auth.cubecraftlabs.com`, admin interface):
1. **Provider** — *OAuth2/OpenID Provider*:
- Client type: **Public** (no client secret; the admin PWA is a browser
SPA).
- PKCE: the PWA always uses the authorization-code flow with PKCE,
`code_challenge_method=S256`, and never sends a client secret.
- Redirect URI (strict): `https://nestquest.cubecraftlabs.com/auth/callback`.
- Signing key: an **RSA** certificate (the API only accepts `RS256`).
- Scopes: the default `openid`, `email`, `profile` mappings (the PWA
requests `openid email profile`; Authentik's default `profile`
mapping emits the `groups` claim the API checks).
- Note the generated **Client ID** — it is the audience below and the
PWA's `VITE_AUTHENTIK_CLIENT_ID`.
2. **Application** — slug **`nestquest`**, bound to the provider above.
The slug fixes the issuer URL.
3. **Group** — **`nestquest-admins`**. Add each parent's Authentik user.
Membership is the single source of truth for "who is an admin"; the
API refuses a signed-in non-member with 403.
API settings (in `deploy/.env`, section 5):
| Variable | Value |
| --- | --- |
| `NESTQUEST_OIDC_ISSUER` | `https://auth.cubecraftlabs.com/application/o/nestquest/` |
| `NESTQUEST_OIDC_AUDIENCE` | the provider's Client ID |
| `NESTQUEST_OIDC_JWKS_URL` | `https://auth.cubecraftlabs.com/application/o/nestquest/jwks/` |
Check them against Authentik's discovery document (the issuer must match
byte for byte, including the trailing slash):
curl -s https://auth.cubecraftlabs.com/application/o/nestquest/.well-known/openid-configuration \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["issuer"]); print(d["jwks_uri"])'
## 5. Secrets
Nothing below is ever committed; `.gitignore` excludes `.env`,
`**/.env`, `deploy/*key*` and `deploy/litestream_key*`.
**`deploy/.env`** — created on the API box from the committed example:
cp deploy/.env.example deploy/.env
chmod 600 deploy/.env
# edit deploy/.env and fill in:
# NESTQUEST_PANEL_TOKEN = a new random token, e.g.
# python3 -c "import secrets; print(secrets.token_urlsafe(32))"
# (the same value is configured in the HA integration)
# NESTQUEST_OIDC_ISSUER = section 4
# NESTQUEST_OIDC_AUDIENCE = section 4 (Authentik Client ID)
# NESTQUEST_OIDC_JWKS_URL = section 4
# NESTQUEST_BIND_ADDRESS = 10.60.1.14
The API fails fast at startup if any of the four API
variables is unset or empty (`api/config.py`); `dc logs api` names the
missing one. `NESTQUEST_BIND_ADDRESS` is read by compose only.
**Litestream SFTP key `deploy/litestream_key`** — a dedicated keypair that
lives only on the API box. Create it once, before the first `up`:
ssh-keygen -t ed25519 -N "" -C nestquest-litestream -f deploy/litestream_key
sudo chown 999:999 deploy/litestream_key
chmod 600 deploy/litestream_key
Then append `restrict ` followed by the contents of
`deploy/litestream_key.pub` as one line to
`joshua@10.60.1.50:~/.ssh/authorized_keys` on the NAS. Compose mounts the
private key read-only at `/etc/litestream/sftp_key` in the sidecar, which
runs as uid 999 — hence the ownership. The NAS host key is pinned in the
committed `deploy/litestream.yml` (`host-key`, the NAS's ECDSA key), so no
`known_hosts` is involved.
| Secret | Supplied by | Consumed by |
| --- | --- | --- |
| `NESTQUEST_PANEL_TOKEN` | `deploy/.env` → `env_file` | `api` (panel plane); also set in the HA integration |
| `NESTQUEST_OIDC_*` (not secret, but environment-supplied) | `deploy/.env` → `env_file` | `api` (admin plane) |
| SFTP private key | `deploy/litestream_key` → read-only bind mount | `litestream` |
| Cloudflare DNS API token | Traefik deployment on nukapi-03 | Traefik ACME `myresolver` |
## 6. Network separation and firewall
- **Panel plane** (`/api/v1/panel/*`, static service token): reachable
only from the HA box. Traefik's `ipAllowList` on the panel router allows
`10.60.1.80/32` and returns **403** to every other source, before the
request reaches the API. The API additionally requires the panel token
(401 without it).
- **Admin plane** (`/api/v1/admin/*`) and `/health`, `/docs`: public
through Cloudflare and Traefik; admin routes are protected by the API's
Authentik JWT check (401 without a valid token, 403 without
`nestquest-admins` or when a panel token is presented).
- **Local DNS override** so the HA box reaches Traefik directly, keeping
its real source address: in Pi-hole (`10.60.1.101`) local DNS records,
add `nestquest.cubecraftlabs.com` → `10.60.1.33`. The HA box
must use Pi-hole as its resolver. Check from the HA box:
nslookup nestquest.cubecraftlabs.com 10.60.1.101 # Address: 10.60.1.33
- **Direct `:8080` access is firewalled.** The API box publishes `8080`
on `10.60.1.14`, which alone would let any LAN host bypass Traefik and
the panel `ipAllowList`. This is enforced on the API box in Docker's
`DOCKER-USER` chain (which Docker evaluates before its own forwarding
rules for published ports), in this order:
-A DOCKER-USER -s 10.60.1.33/32 -p tcp --dport 8080 -j RETURN # Traefik on nukapi-03
-A DOCKER-USER -s 10.60.1.14/32 -p tcp --dport 8080 -j RETURN # the API box itself
-A DOCKER-USER -p tcp --dport 8080 -j DROP # every other source
The rules are persisted by the systemd oneshot unit
**`nestquest-api-firewall.service`**, committed as
`deploy/systemd/nestquest-api-firewall.service` (ordered after
`docker.service`, it idempotently applies the three rules at every
boot). Install and enable it on the API box, from the repository
checkout, **before** exposing the API:
sudo cp deploy/systemd/nestquest-api-firewall.service /etc/systemd/system/ \
&& sudo systemctl daemon-reload \
&& sudo systemctl enable --now nestquest-api-firewall.service
The HA box and every other client must use the Traefik hostname,
never `:8080` directly. Check it:
sudo systemctl is-enabled nestquest-api-firewall.service # enabled
sudo systemctl is-active nestquest-api-firewall.service # active
sudo iptables -S DOCKER-USER # the three rules above, in that order
# on nukapi-03 (10.60.1.33) — allowed:
curl -s -o /dev/null -w '%{http_code}\n' http://10.60.1.14:8080/health # 200
# on any other LAN host, e.g. 10.60.1.101 — dropped:
curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://10.60.1.14:8080/health # 000, timeout
# public path still works:
curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/health # 200
On a rebuilt API box, install the unit with the commands above before
exposing the API. The committed file is the authoritative copy; the
deployed unit must match it:
diff deploy/systemd/nestquest-api-firewall.service \
/etc/systemd/system/nestquest-api-firewall.service # no output
## 7. Backup (Litestream to the Synology NAS)
The `litestream` sidecar (config `deploy/litestream.yml`, mounted at
`/etc/litestream.yml`) replicates `/var/lib/nestquest/nestquest.db` to:
- SFTP `joshua@10.60.1.50`, path `/homes/Joshua/nestquest-backups` (the
NAS's SFTP view is chrooted to the shared folders; over SSH the same
directory is `/volume1/homes/Joshua/nestquest-backups`).
- Every committed change is shipped as an LTX file within about a second.
Litestream 0.5.17 compacts them into coarser levels on its built-in
schedule (level 1 every 30s, level 2 every 5m, level 3 every 1h).
- **Snapshots**: `snapshot.interval: 24h` (a full snapshot daily);
**retention**: `snapshot.retention: 168h` — a week of snapshots plus the
changes between them, i.e. a 7-day point-in-time restore window.
The sidecar runs as uid 999 with a read-only root filesystem,
`no-new-privileges` and all capabilities dropped. It needs the database
volume read-write (it keeps its `_litestream_*` tables and writes the
WAL); its local state lives beside the database in
`/var/lib/nestquest/.nestquest.db-litestream/`.
Check it:
dc ps litestream # Up
dc logs --tail 50 litestream # no "error" lines
dc exec litestream litestream status # status "ok", local txid advancing
dc exec litestream litestream ltx /var/lib/nestquest/nestquest.db | tail # newest LTX files and timestamps
A change made in the app should appear as a new level-0 LTX file within
seconds. On the NAS, the files are under
`/volume1/homes/Joshua/nestquest-backups/ltx/`.
## 8. Restore
Restores the database from the SFTP replica into the `nestquest-db`
volume at `NESTQUEST_DB_PATH` (`/var/lib/nestquest/nestquest.db`). The
same procedure serves both a damaged database on the existing box and a
rebuilt box. The restore runs in one-off `litestream` containers so it
has the same user (999), volume, config and SFTP key as the sidecar.
**0. Fresh box only.** Complete sections 1 and 5, then start **only the
API** once so the volume exists and is owned by uid 999:
dc up -d api
Do not start `litestream` yet — it would begin replicating the new,
empty database the API just created. The swap in step 4 moves that empty
file aside, and step 5 creates and starts the sidecar with
`dc up -d litestream` once the API is healthy.
**1. Stop writers** — both the API and the sidecar (Litestream must not
replicate while the file is being replaced):
dc stop api litestream
**2. Restore beside the live database** (never over it), with a full
integrity check:
dc run --rm --no-deps litestream restore \
-config /etc/litestream.yml \
-integrity-check full \
-o /var/lib/nestquest/nestquest.db.restored \
/var/lib/nestquest/nestquest.db
Expect `post-restore integrity check passed` and exit code 0. For a
point in time within the retention window, add
`-timestamp 2026-09-24T12:00:00Z` (UTC). To preview without writing, add
`-dry-run`.
**3. Inspect the restored file** before switching to it — confirm known
children and definitions are present:
dc run --rm --no-deps --entrypoint sqlite3 litestream \
/var/lib/nestquest/nestquest.db.restored \
'PRAGMA integrity_check;' \
'SELECT id, display_name FROM children;' \
'SELECT id, title FROM quest_definitions;'
Expect `ok`, then the household's children and quest definitions.
**4. Swap it into place**, keeping the old files (database, its
`-wal`/`-shm`, and Litestream's local state) in a timestamped directory
on the volume:
dc run --rm --no-deps --entrypoint sh litestream -c '
set -e
cd /var/lib/nestquest
ts=$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "pre-restore-$ts"
for f in nestquest.db nestquest.db-wal nestquest.db-shm .nestquest.db-litestream; do
if [ -e "$f" ]; then mv "$f" "pre-restore-$ts/"; fi
done
mv nestquest.db.restored nestquest.db
ls -la'
**5. Start the API, then the sidecar:**
dc start api
dc ps api # wait for "healthy"
dc up -d litestream
dc ps litestream # Up
dc logs --tail 20 litestream
Use `dc up -d litestream`, not `dc start`: on a fresh box step 0 never
created the sidecar container, and `start` cannot start a container that
does not exist. `up -d` creates it if needed (or starts the existing one),
so continuous replication is back online on both paths.
The sidecar resumes replication from the restored position (it logs
`detected database behind replica` / `fetched latest L0 file from
replica`, then continues normally).
**6. Verify through the API** — the panel snapshot lists the restored
children. On the API box against `10.60.1.14:8080` directly (other LAN
hosts, including the HA box, are dropped on `:8080` by the section 6
firewall; from the HA box use the Traefik hostname instead):
PANEL_TOKEN=$(grep '^NESTQUEST_PANEL_TOKEN=' deploy/.env | cut -d= -f2-)
curl -s -H "Authorization: Bearer $PANEL_TOKEN" \
http://10.60.1.14:8080/api/v1/panel/snapshot
Expect HTTP 200 and `children` naming the known child(ren). Then run
section 9.
Once satisfied, remove the old copy:
`dc run --rm --no-deps --entrypoint rm litestream -rf /var/lib/nestquest/pre-restore-<ts>`.
This exact sequence (stop, restore with integrity check, inspect, swap,
start, snapshot shows the restored child, replication resumes) was
rehearsed against a local file replica with the committed compose file;
the live proof against the SFTP replica is recorded separately by the
controller.
## 9. Verification checklist
Run after every deploy and every restore. `$PANEL_TOKEN` is the value from
`deploy/.env`; never paste it into shared logs.
| # | Check | Command | Expect |
| --- | --- | --- | --- |
| 1 | Health over TLS | `curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/health` | `200` |
| 2 | Valid certificate | `curl -sv https://nestquest.cubecraftlabs.com/health 2>&1 \| grep -E 'issuer\|expire date'` | Let's Encrypt issuer, future expiry (no `-k` needed) |
| 3 | Docs public | `curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/docs` | `200` |
| 4 | Panel refused externally | from any host other than `10.60.1.80` (e.g. a phone off Wi-Fi, or a LAN machine): `curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $PANEL_TOKEN" https://nestquest.cubecraftlabs.com/api/v1/panel/snapshot` | `403` (Traefik allowlist) |
| 5 | Panel allowed from HA box | on `10.60.1.80`: same command | `200`, snapshot JSON |
| 6 | Panel needs the token | on `10.60.1.80`, without the header | `401` |
| 7 | Admin JWT accepted | `curl -s -w '\n%{http_code}\n' -H "Authorization: Bearer $ADMIN_JWT" https://nestquest.cubecraftlabs.com/api/v1/admin/ping` with a token of a `nestquest-admins` member | `200`, `{"status":"ok","sub":...}` |
| 8 | Non-admin refused | same, with a token of a user **not** in `nestquest-admins` | `403`, "Admin access requires membership in the nestquest-admins group" |
| 9 | No/invalid token refused | same, without a header or with garbage | `401` |
| 10 | Panel token refused on admin | same, with `$PANEL_TOKEN` | `403` |
| 11 | Backup running | `dc exec litestream litestream status` | status `ok` |
To obtain `$ADMIN_JWT`: sign in to the admin PWA as the user, open the
browser developer tools, and copy the session-storage value
`nestquest.access_token`. Tokens are short-lived and are credentials —
do not store them.
deploy/compose.yaml
+79
View File
@@ -0,0 +1,79 @@
# Compose deployment scaffold for the NestQuest API service (Feature
# 17). The SQLite database lives on the NAMED VOLUME ``nestquest-db``
# mounted at /var/lib/nestquest — recreating the container keeps the
# household data (``docker compose down`` leaves named volumes alone).
#
# Configuration is supplied by the environment: the fixed database
# path here, and the secret values (panel service token, Authentik
# OIDC settings) from ``deploy/.env`` — a git-ignored file you create
# from ``deploy/.env.example`` (placeholder names only; never commit
# real values).
#
# TLS is terminated by the shared Traefik reverse proxy on another
# host, which forwards to this service on port 8080. The published
# port binds NESTQUEST_BIND_ADDRESS (read from deploy/.env, the
# project directory's env file): it defaults to 127.0.0.1 so an
# unconfigured deployment is never exposed; set it to the host's LAN
# address on the API box so the proxy can reach it.
services:
api:
build:
context: ..
dockerfile: Dockerfile
image: nestquest-api:latest
init: true
restart: unless-stopped
ports:
- "${NESTQUEST_BIND_ADDRESS:-127.0.0.1}:8080:8080"
environment:
# The SQLite file inside the named volume; fixed, not a secret.
NESTQUEST_DB_PATH: /var/lib/nestquest/nestquest.db
env_file:
# Secret values (NESTQUEST_PANEL_TOKEN, NESTQUEST_OIDC_*); absent
# until you create deploy/.env from the example.
- path: .env
required: false
volumes:
- nestquest-db:/var/lib/nestquest
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health')"]
interval: 10s
timeout: 5s
retries: 5
start_period: 15s
# Continuous backup: streams the SQLite database on the shared named
# volume to the Synology NAS over SFTP (config: deploy/litestream.yml).
# The SFTP private key deploy/litestream_key is git-ignored and must be
# created on the API box before starting (see api/README.md).
#
# The database volume is mounted READ-WRITE on purpose: Litestream
# cannot run against a read-only mount ("unable to open database file
# (14)", even with its metadata moved elsewhere) because it writes its
# _litestream_seq/_litestream_lock tables into the database, opens the
# -wal/-shm files for writing, and checkpoints the WAL. The write
# access is contained instead: it runs as the API's own non-root user
# (uid/gid 999, owner of the volume) with a read-only root filesystem,
# no privilege escalation and no Linux capabilities.
litestream:
image: litestream/litestream:0.5.17@sha256:4b02b9859a6b6b4087d8b8944e15f7e984bd7957cba322bbeee38b0e27b9656a
restart: unless-stopped
command: ["replicate", "-config", "/etc/litestream.yml"]
user: "999:999"
read_only: true
tmpfs:
- /tmp
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
depends_on:
api:
condition: service_healthy
volumes:
- nestquest-db:/var/lib/nestquest
- ./litestream.yml:/etc/litestream.yml:ro
- ./litestream_key:/etc/litestream/sftp_key:ro
volumes:
nestquest-db:
deploy/litestream.yml
+32
View File
@@ -0,0 +1,32 @@
# Litestream continuous backup of the NestQuest SQLite database (Feature
# 17). The ``litestream`` compose service mounts the same named volume
# as the API and streams every committed change to the Synology NAS
# over SFTP, so the backup lives OUTSIDE the container's volume.
#
# Nothing here is secret. The SFTP private key is a dedicated,
# git-ignored keypair (deploy/litestream_key) mounted read-only by
# compose at /etc/litestream/sftp_key; its public half is in
# joshua@10.60.1.50:~/.ssh/authorized_keys. ``host-key`` pins the
# NAS's host key so the connection is verified; it must be the ECDSA
# key, because that is the algorithm Litestream's SSH client negotiates
# with this NAS (pinning the ed25519 key fails with "host key mismatch").
#
# The Synology SFTP server is chrooted to the shared folders, so the
# directory seen over SSH as /volume1/homes/Joshua/nestquest-backups is
# /homes/Joshua/nestquest-backups over SFTP.
# Take a full snapshot daily and keep a week of snapshots plus the
# incremental changes between them (point-in-time restore window).
snapshot:
interval: 24h
retention: 168h
dbs:
- path: /var/lib/nestquest/nestquest.db
replicas:
- type: sftp
host: 10.60.1.50
user: joshua
path: /homes/Joshua/nestquest-backups
key-path: /etc/litestream/sftp_key
host-key: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNujay9nYEiXuRQf4wDRDU1gp+CDvpeMdhuhC2XypIihzKGEtB654kwrUGfA2kDWAoSPGlmojeS3OIRtHPkj6GY=
deploy/systemd/nestquest-api-firewall.service
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Restrict NestQuest API port 8080 to Traefik (DOCKER-USER)
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "iptables -C DOCKER-USER -s 10.60.1.33/32 -p tcp --dport 8080 -j RETURN 2>/dev/null || iptables -I DOCKER-USER 1 -s 10.60.1.33 -p tcp --dport 8080 -j RETURN; iptables -C DOCKER-USER -s 10.60.1.14/32 -p tcp --dport 8080 -j RETURN 2>/dev/null || iptables -I DOCKER-USER 2 -s 10.60.1.14 -p tcp --dport 8080 -j RETURN; iptables -C DOCKER-USER -p tcp --dport 8080 -j DROP 2>/dev/null || iptables -A DOCKER-USER -p tcp --dport 8080 -j DROP"
[Install]
WantedBy=multi-user.target