|
|
|
@@ -0,0 +1,455 @@
|
|
|
|
|
# NestQuest API — deployment and restore runbook
|
|
|
|
|
|
|
|
|
|
Operator procedure to rebuild the production NestQuest API deployment end
|
|
|
|
|
to end from this repository, restore its database from backup, and verify
|
|
|
|
|
it. Every value below is the real, non-secret production value. **No
|
|
|
|
|
secret appears in this file** — secrets are named and their location is
|
|
|
|
|
given, never their content.
|
|
|
|
|
|
|
|
|
|
| Piece | Where | Real value |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| API box (Docker host) | this repository's `deploy/` | `10.60.1.14`, port `8080` |
|
|
|
|
|
| Reverse proxy / TLS | shared Traefik on `nukapi-03` | `10.60.1.33` |
|
|
|
|
|
| Public hostname | Cloudflare DNS | `nestquest.cubecraftlabs.com` |
|
|
|
|
|
| Identity | Authentik | `https://auth.cubecraftlabs.com` |
|
|
|
|
|
| Home Assistant box (panel client) | LAN | `10.60.1.80` |
|
|
|
|
|
| Local DNS | Pi-hole | `10.60.1.101` |
|
|
|
|
|
| Backup target | Synology NAS `CubeCraft-SRVR1`, SFTP | `10.60.1.50:/homes/Joshua/nestquest-backups` |
|
|
|
|
|
|
|
|
|
|
All `docker compose` commands below are run **on the API box, from the
|
|
|
|
|
repository root**. The shorthand `dc` means:
|
|
|
|
|
|
|
|
|
|
alias dc='docker compose -f deploy/compose.yaml'
|
|
|
|
|
|
|
|
|
|
Compose v2.24 or newer is required (`env_file` uses the `path`/`required`
|
|
|
|
|
form).
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 1. Image build
|
|
|
|
|
|
|
|
|
|
The image is defined by the root `Dockerfile` and is built from the
|
|
|
|
|
**repository root** as the build context: the API (`api/`) loads the
|
|
|
|
|
bundled domain core from `custom_components/nestquest/core/` by file
|
|
|
|
|
location, so both trees must be in the context. `.dockerignore` keeps
|
|
|
|
|
everything else (including `deploy/`, `.env` files and `*.db`) out of the
|
|
|
|
|
build.
|
|
|
|
|
|
|
|
|
|
- Multi-stage on `python:3.12-slim`: the builder installs
|
|
|
|
|
`api/requirements.txt`, the runtime copies only the installed packages
|
|
|
|
|
plus `api/` and `custom_components/nestquest/core/` into `/app`.
|
|
|
|
|
- Runs as the non-root system user `nestquest` (uid/gid `999`), which owns
|
|
|
|
|
`/var/lib/nestquest`.
|
|
|
|
|
- Command: `uvicorn --factory api.app:create_app --host 0.0.0.0 --port 8080`.
|
|
|
|
|
|
|
|
|
|
Get the code and build:
|
|
|
|
|
|
|
|
|
|
git clone ssh://code.cubecraftlabs.com:2288/Maniacs_Mansion/NestQuest.git
|
|
|
|
|
cd NestQuest
|
|
|
|
|
git checkout <release branch or tag being deployed>
|
|
|
|
|
dc build api # equivalent: docker build -t nestquest-api:latest .
|
|
|
|
|
|
|
|
|
|
Confirm the runtime user (the Litestream sidecar and the key ownership in
|
|
|
|
|
section 5 depend on uid 999):
|
|
|
|
|
|
|
|
|
|
docker run --rm --entrypoint id nestquest-api:latest
|
|
|
|
|
# uid=999(nestquest) gid=999(nestquest) groups=999(nestquest)
|
|
|
|
|
|
|
|
|
|
## 2. Compose
|
|
|
|
|
|
|
|
|
|
`deploy/compose.yaml` defines two services and one named volume:
|
|
|
|
|
|
|
|
|
|
| Service | Image | Role |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `api` | `nestquest-api:latest` (built, section 1) | FastAPI service on `8080`, healthcheck on `/health` |
|
|
|
|
|
| `litestream` | `litestream/litestream:0.5.17` | Continuous SFTP backup of the database (section 7); starts only once `api` is healthy |
|
|
|
|
|
|
|
|
|
|
- **Volume `nestquest-db`** is mounted at `/var/lib/nestquest` in both
|
|
|
|
|
services. The database is `NESTQUEST_DB_PATH=/var/lib/nestquest/nestquest.db`
|
|
|
|
|
(fixed by compose, not a secret). `docker compose down` keeps named
|
|
|
|
|
volumes; **never run `down -v`** on the API box. Docker prefixes the
|
|
|
|
|
volume with the compose project name — with the default project
|
|
|
|
|
(`deploy`, the compose file's directory) it is `deploy_nestquest-db`;
|
|
|
|
|
confirm with `docker volume ls | grep nestquest-db`.
|
|
|
|
|
- **Published port**: `${NESTQUEST_BIND_ADDRESS:-127.0.0.1}:8080:8080`.
|
|
|
|
|
The default binds localhost only, so an unconfigured host is never
|
|
|
|
|
exposed. On the API box `deploy/.env` sets
|
|
|
|
|
`NESTQUEST_BIND_ADDRESS=10.60.1.14` so Traefik on nukapi-03 can reach
|
|
|
|
|
`http://10.60.1.14:8080`.
|
|
|
|
|
- **Secrets** come from `deploy/.env` (git-ignored, section 5) via
|
|
|
|
|
`env_file`.
|
|
|
|
|
|
|
|
|
|
Bring the stack up (after section 5's secrets exist):
|
|
|
|
|
|
|
|
|
|
dc up -d --build
|
|
|
|
|
dc ps # api "healthy", litestream "Up"
|
|
|
|
|
curl -s http://10.60.1.14:8080/health # {"status":"ok"}
|
|
|
|
|
|
|
|
|
|
## 3. TLS and reverse proxy (Traefik on nukapi-03)
|
|
|
|
|
|
|
|
|
|
TLS is terminated by the **shared** Traefik instance on `nukapi-03`
|
|
|
|
|
(`10.60.1.33`), not by this repository. Its NestQuest routing is the
|
|
|
|
|
owner-authored dynamic file-provider config
|
|
|
|
|
**`config/dynamic/nestquest-ccl.yml`** in the Traefik deployment on
|
|
|
|
|
nukapi-03 — that live file is authoritative; it is not stored in this
|
|
|
|
|
repository. What it must contain:
|
|
|
|
|
|
|
|
|
|
| Router | Rule | Middleware | Service |
|
|
|
|
|
| --- | --- | --- | --- |
|
|
|
|
|
| panel | ``Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)`` | `ipAllowList`, `sourceRange: ["10.60.1.80/32"]` (HA box only) | `http://10.60.1.14:8080` |
|
|
|
|
|
| admin (everything else: `/health`, `/docs`, `/openapi.json`, `/api/v1/admin/*`) | ``Host(`nestquest.cubecraftlabs.com`)`` | none — public; the API itself enforces Authentik JWTs (section 4) | `http://10.60.1.14:8080` |
|
|
|
|
|
|
|
|
|
|
Both routers use the HTTPS entrypoint `websecure` with `tls.certResolver: myresolver`
|
|
|
|
|
(HTTP is redirected to HTTPS by the shared Traefik). The panel router's
|
|
|
|
|
rule is longer, so Traefik's default rule-length priority evaluates it
|
|
|
|
|
before the catch-all admin router; if you set explicit priorities, keep
|
|
|
|
|
panel higher.
|
|
|
|
|
|
|
|
|
|
Reference shape, to rebuild the file if it is ever lost (entrypoint names
|
|
|
|
|
must match the shared Traefik's static config on nukapi-03):
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
http:
|
|
|
|
|
routers:
|
|
|
|
|
nestquest-panel:
|
|
|
|
|
rule: "Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)"
|
|
|
|
|
entryPoints: [websecure]
|
|
|
|
|
middlewares: [nestquest-panel-allowlist]
|
|
|
|
|
service: nestquest-api
|
|
|
|
|
tls:
|
|
|
|
|
certResolver: myresolver
|
|
|
|
|
nestquest-admin:
|
|
|
|
|
rule: "Host(`nestquest.cubecraftlabs.com`)"
|
|
|
|
|
entryPoints: [websecure]
|
|
|
|
|
service: nestquest-api
|
|
|
|
|
tls:
|
|
|
|
|
certResolver: myresolver
|
|
|
|
|
middlewares:
|
|
|
|
|
nestquest-panel-allowlist:
|
|
|
|
|
ipAllowList:
|
|
|
|
|
sourceRange: ["10.60.1.80/32"]
|
|
|
|
|
services:
|
|
|
|
|
nestquest-api:
|
|
|
|
|
loadBalancer:
|
|
|
|
|
servers:
|
|
|
|
|
- url: "http://10.60.1.14:8080"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Certificates.** `myresolver` is the shared Traefik's ACME (Let's
|
|
|
|
|
Encrypt) resolver using the **Cloudflare DNS-01 challenge**; the
|
|
|
|
|
Cloudflare API token is held by the Traefik deployment on
|
|
|
|
|
nukapi-03, never in this repository. Because DNS-01 proves control
|
|
|
|
|
through a DNS TXT record, issuance and renewal need no inbound HTTP and
|
|
|
|
|
work regardless of the Cloudflare proxy setting. Certificates renew
|
|
|
|
|
automatically.
|
|
|
|
|
|
|
|
|
|
**Cloudflare proxy note.** Public traffic for
|
|
|
|
|
`nestquest.cubecraftlabs.com` resolves through Cloudflare, so a request
|
|
|
|
|
from the internet reaches Traefik from a Cloudflare edge address, never
|
|
|
|
|
from `10.60.1.80` — which is exactly why the panel allowlist refuses it
|
|
|
|
|
(403). The HA box does not go through Cloudflare at all: the local DNS
|
|
|
|
|
override in section 6 sends it straight to Traefik on the LAN, so Traefik
|
|
|
|
|
sees its real address. Do **not** add Cloudflare ranges to the panel
|
|
|
|
|
allowlist or make the allowlist trust `X-Forwarded-For`; that would let
|
|
|
|
|
any internet client through to the panel plane.
|
|
|
|
|
|
|
|
|
|
## 4. Authentik (admin identity)
|
|
|
|
|
|
|
|
|
|
Admin-plane requests (`/api/v1/admin/*`) carry an Authentik OIDC access
|
|
|
|
|
token (JWT). The API verifies issuer, audience, RS256 signature against
|
|
|
|
|
the JWKS, and expiry, then requires the `groups` claim to contain
|
|
|
|
|
`nestquest-admins` (`api/auth.py`). Configure in Authentik
|
|
|
|
|
(`https://auth.cubecraftlabs.com`, admin interface):
|
|
|
|
|
|
|
|
|
|
1. **Provider** — *OAuth2/OpenID Provider*:
|
|
|
|
|
- Client type: **Public** (no client secret; the admin PWA is a browser
|
|
|
|
|
SPA).
|
|
|
|
|
- PKCE: the PWA always uses the authorization-code flow with PKCE,
|
|
|
|
|
`code_challenge_method=S256`, and never sends a client secret.
|
|
|
|
|
- Redirect URI (strict): `https://nestquest.cubecraftlabs.com/auth/callback`.
|
|
|
|
|
- Signing key: an **RSA** certificate (the API only accepts `RS256`).
|
|
|
|
|
- Scopes: the default `openid`, `email`, `profile` mappings (the PWA
|
|
|
|
|
requests `openid email profile`; Authentik's default `profile`
|
|
|
|
|
mapping emits the `groups` claim the API checks).
|
|
|
|
|
- Note the generated **Client ID** — it is the audience below and the
|
|
|
|
|
PWA's `VITE_AUTHENTIK_CLIENT_ID`.
|
|
|
|
|
2. **Application** — slug **`nestquest`**, bound to the provider above.
|
|
|
|
|
The slug fixes the issuer URL.
|
|
|
|
|
3. **Group** — **`nestquest-admins`**. Add each parent's Authentik user.
|
|
|
|
|
Membership is the single source of truth for "who is an admin"; the
|
|
|
|
|
API refuses a signed-in non-member with 403.
|
|
|
|
|
|
|
|
|
|
API settings (in `deploy/.env`, section 5):
|
|
|
|
|
|
|
|
|
|
| Variable | Value |
|
|
|
|
|
| --- | --- |
|
|
|
|
|
| `NESTQUEST_OIDC_ISSUER` | `https://auth.cubecraftlabs.com/application/o/nestquest/` |
|
|
|
|
|
| `NESTQUEST_OIDC_AUDIENCE` | the provider's Client ID |
|
|
|
|
|
| `NESTQUEST_OIDC_JWKS_URL` | `https://auth.cubecraftlabs.com/application/o/nestquest/jwks/` |
|
|
|
|
|
|
|
|
|
|
Check them against Authentik's discovery document (the issuer must match
|
|
|
|
|
byte for byte, including the trailing slash):
|
|
|
|
|
|
|
|
|
|
curl -s https://auth.cubecraftlabs.com/application/o/nestquest/.well-known/openid-configuration \
|
|
|
|
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["issuer"]); print(d["jwks_uri"])'
|
|
|
|
|
|
|
|
|
|
## 5. Secrets
|
|
|
|
|
|
|
|
|
|
Nothing below is ever committed; `.gitignore` excludes `.env`,
|
|
|
|
|
`**/.env`, `deploy/*key*` and `deploy/litestream_key*`.
|
|
|
|
|
|
|
|
|
|
**`deploy/.env`** — created on the API box from the committed example:
|
|
|
|
|
|
|
|
|
|
cp deploy/.env.example deploy/.env
|
|
|
|
|
chmod 600 deploy/.env
|
|
|
|
|
# edit deploy/.env and fill in:
|
|
|
|
|
# NESTQUEST_PANEL_TOKEN = a new random token, e.g.
|
|
|
|
|
# python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
|
|
|
|
# (the same value is configured in the HA integration)
|
|
|
|
|
# NESTQUEST_OIDC_ISSUER = section 4
|
|
|
|
|
# NESTQUEST_OIDC_AUDIENCE = section 4 (Authentik Client ID)
|
|
|
|
|
# NESTQUEST_OIDC_JWKS_URL = section 4
|
|
|
|
|
# NESTQUEST_BIND_ADDRESS = 10.60.1.14
|
|
|
|
|
|
|
|
|
|
The API fails fast at startup if any of the four API
|
|
|
|
|
variables is unset or empty (`api/config.py`); `dc logs api` names the
|
|
|
|
|
missing one. `NESTQUEST_BIND_ADDRESS` is read by compose only.
|
|
|
|
|
|
|
|
|
|
**Litestream SFTP key `deploy/litestream_key`** — a dedicated keypair that
|
|
|
|
|
lives only on the API box. Create it once, before the first `up`:
|
|
|
|
|
|
|
|
|
|
ssh-keygen -t ed25519 -N "" -C nestquest-litestream -f deploy/litestream_key
|
|
|
|
|
sudo chown 999:999 deploy/litestream_key
|
|
|
|
|
chmod 600 deploy/litestream_key
|
|
|
|
|
|
|
|
|
|
Then append `restrict ` followed by the contents of
|
|
|
|
|
`deploy/litestream_key.pub` as one line to
|
|
|
|
|
`joshua@10.60.1.50:~/.ssh/authorized_keys` on the NAS. Compose mounts the
|
|
|
|
|
private key read-only at `/etc/litestream/sftp_key` in the sidecar, which
|
|
|
|
|
runs as uid 999 — hence the ownership. The NAS host key is pinned in the
|
|
|
|
|
committed `deploy/litestream.yml` (`host-key`, the NAS's ECDSA key), so no
|
|
|
|
|
`known_hosts` is involved.
|
|
|
|
|
|
|
|
|
|
| Secret | Supplied by | Consumed by |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `NESTQUEST_PANEL_TOKEN` | `deploy/.env` → `env_file` | `api` (panel plane); also set in the HA integration |
|
|
|
|
|
| `NESTQUEST_OIDC_*` (not secret, but environment-supplied) | `deploy/.env` → `env_file` | `api` (admin plane) |
|
|
|
|
|
| SFTP private key | `deploy/litestream_key` → read-only bind mount | `litestream` |
|
|
|
|
|
| Cloudflare DNS API token | Traefik deployment on nukapi-03 | Traefik ACME `myresolver` |
|
|
|
|
|
|
|
|
|
|
## 6. Network separation and firewall
|
|
|
|
|
|
|
|
|
|
- **Panel plane** (`/api/v1/panel/*`, static service token): reachable
|
|
|
|
|
only from the HA box. Traefik's `ipAllowList` on the panel router allows
|
|
|
|
|
`10.60.1.80/32` and returns **403** to every other source, before the
|
|
|
|
|
request reaches the API. The API additionally requires the panel token
|
|
|
|
|
(401 without it).
|
|
|
|
|
- **Admin plane** (`/api/v1/admin/*`) and `/health`, `/docs`: public
|
|
|
|
|
through Cloudflare and Traefik; admin routes are protected by the API's
|
|
|
|
|
Authentik JWT check (401 without a valid token, 403 without
|
|
|
|
|
`nestquest-admins` or when a panel token is presented).
|
|
|
|
|
- **Local DNS override** so the HA box reaches Traefik directly, keeping
|
|
|
|
|
its real source address: in Pi-hole (`10.60.1.101`) local DNS records,
|
|
|
|
|
add `nestquest.cubecraftlabs.com` → `10.60.1.33`. The HA box
|
|
|
|
|
must use Pi-hole as its resolver. Check from the HA box:
|
|
|
|
|
|
|
|
|
|
nslookup nestquest.cubecraftlabs.com 10.60.1.101 # Address: 10.60.1.33
|
|
|
|
|
|
|
|
|
|
- **Direct `:8080` access is firewalled.** The API box publishes `8080`
|
|
|
|
|
on `10.60.1.14`, which alone would let any LAN host bypass Traefik and
|
|
|
|
|
the panel `ipAllowList`. This is enforced on the API box in Docker's
|
|
|
|
|
`DOCKER-USER` chain (which Docker evaluates before its own forwarding
|
|
|
|
|
rules for published ports), in this order:
|
|
|
|
|
|
|
|
|
|
-A DOCKER-USER -s 10.60.1.33/32 -p tcp --dport 8080 -j RETURN # Traefik on nukapi-03
|
|
|
|
|
-A DOCKER-USER -s 10.60.1.14/32 -p tcp --dport 8080 -j RETURN # the API box itself
|
|
|
|
|
-A DOCKER-USER -p tcp --dport 8080 -j DROP # every other source
|
|
|
|
|
|
|
|
|
|
The rules are persisted by the systemd oneshot unit
|
|
|
|
|
**`nestquest-api-firewall.service`**, committed as
|
|
|
|
|
`deploy/systemd/nestquest-api-firewall.service` (ordered after
|
|
|
|
|
`docker.service`, it idempotently applies the three rules at every
|
|
|
|
|
boot). Install and enable it on the API box, from the repository
|
|
|
|
|
checkout, **before** exposing the API:
|
|
|
|
|
|
|
|
|
|
sudo cp deploy/systemd/nestquest-api-firewall.service /etc/systemd/system/ \
|
|
|
|
|
&& sudo systemctl daemon-reload \
|
|
|
|
|
&& sudo systemctl enable --now nestquest-api-firewall.service
|
|
|
|
|
|
|
|
|
|
The HA box and every other client must use the Traefik hostname,
|
|
|
|
|
never `:8080` directly. Check it:
|
|
|
|
|
|
|
|
|
|
sudo systemctl is-enabled nestquest-api-firewall.service # enabled
|
|
|
|
|
sudo systemctl is-active nestquest-api-firewall.service # active
|
|
|
|
|
sudo iptables -S DOCKER-USER # the three rules above, in that order
|
|
|
|
|
|
|
|
|
|
# on nukapi-03 (10.60.1.33) — allowed:
|
|
|
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://10.60.1.14:8080/health # 200
|
|
|
|
|
# on any other LAN host, e.g. 10.60.1.101 — dropped:
|
|
|
|
|
curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://10.60.1.14:8080/health # 000, timeout
|
|
|
|
|
# public path still works:
|
|
|
|
|
curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/health # 200
|
|
|
|
|
|
|
|
|
|
On a rebuilt API box, install the unit with the commands above before
|
|
|
|
|
exposing the API. The committed file is the authoritative copy; the
|
|
|
|
|
deployed unit must match it:
|
|
|
|
|
|
|
|
|
|
diff deploy/systemd/nestquest-api-firewall.service \
|
|
|
|
|
/etc/systemd/system/nestquest-api-firewall.service # no output
|
|
|
|
|
|
|
|
|
|
## 7. Backup (Litestream to the Synology NAS)
|
|
|
|
|
|
|
|
|
|
The `litestream` sidecar (config `deploy/litestream.yml`, mounted at
|
|
|
|
|
`/etc/litestream.yml`) replicates `/var/lib/nestquest/nestquest.db` to:
|
|
|
|
|
|
|
|
|
|
- SFTP `joshua@10.60.1.50`, path `/homes/Joshua/nestquest-backups` (the
|
|
|
|
|
NAS's SFTP view is chrooted to the shared folders; over SSH the same
|
|
|
|
|
directory is `/volume1/homes/Joshua/nestquest-backups`).
|
|
|
|
|
- Every committed change is shipped as an LTX file within about a second.
|
|
|
|
|
Litestream 0.5.17 compacts them into coarser levels on its built-in
|
|
|
|
|
schedule (level 1 every 30s, level 2 every 5m, level 3 every 1h).
|
|
|
|
|
- **Snapshots**: `snapshot.interval: 24h` (a full snapshot daily);
|
|
|
|
|
**retention**: `snapshot.retention: 168h` — a week of snapshots plus the
|
|
|
|
|
changes between them, i.e. a 7-day point-in-time restore window.
|
|
|
|
|
|
|
|
|
|
The sidecar runs as uid 999 with a read-only root filesystem,
|
|
|
|
|
`no-new-privileges` and all capabilities dropped. It needs the database
|
|
|
|
|
volume read-write (it keeps its `_litestream_*` tables and writes the
|
|
|
|
|
WAL); its local state lives beside the database in
|
|
|
|
|
`/var/lib/nestquest/.nestquest.db-litestream/`.
|
|
|
|
|
|
|
|
|
|
Check it:
|
|
|
|
|
|
|
|
|
|
dc ps litestream # Up
|
|
|
|
|
dc logs --tail 50 litestream # no "error" lines
|
|
|
|
|
dc exec litestream litestream status # status "ok", local txid advancing
|
|
|
|
|
dc exec litestream litestream ltx /var/lib/nestquest/nestquest.db | tail # newest LTX files and timestamps
|
|
|
|
|
|
|
|
|
|
A change made in the app should appear as a new level-0 LTX file within
|
|
|
|
|
seconds. On the NAS, the files are under
|
|
|
|
|
`/volume1/homes/Joshua/nestquest-backups/ltx/`.
|
|
|
|
|
|
|
|
|
|
## 8. Restore
|
|
|
|
|
|
|
|
|
|
Restores the database from the SFTP replica into the `nestquest-db`
|
|
|
|
|
volume at `NESTQUEST_DB_PATH` (`/var/lib/nestquest/nestquest.db`). The
|
|
|
|
|
same procedure serves both a damaged database on the existing box and a
|
|
|
|
|
rebuilt box. The restore runs in one-off `litestream` containers so it
|
|
|
|
|
has the same user (999), volume, config and SFTP key as the sidecar.
|
|
|
|
|
|
|
|
|
|
**0. Fresh box only.** Complete sections 1 and 5, then start **only the
|
|
|
|
|
API** once so the volume exists and is owned by uid 999:
|
|
|
|
|
|
|
|
|
|
dc up -d api
|
|
|
|
|
|
|
|
|
|
Do not start `litestream` yet — it would begin replicating the new,
|
|
|
|
|
empty database the API just created. The swap in step 4 moves that empty
|
|
|
|
|
file aside, and step 5 creates and starts the sidecar with
|
|
|
|
|
`dc up -d litestream` once the API is healthy.
|
|
|
|
|
|
|
|
|
|
**1. Stop writers** — both the API and the sidecar (Litestream must not
|
|
|
|
|
replicate while the file is being replaced):
|
|
|
|
|
|
|
|
|
|
dc stop api litestream
|
|
|
|
|
|
|
|
|
|
**2. Restore beside the live database** (never over it), with a full
|
|
|
|
|
integrity check:
|
|
|
|
|
|
|
|
|
|
dc run --rm --no-deps litestream restore \
|
|
|
|
|
-config /etc/litestream.yml \
|
|
|
|
|
-integrity-check full \
|
|
|
|
|
-o /var/lib/nestquest/nestquest.db.restored \
|
|
|
|
|
/var/lib/nestquest/nestquest.db
|
|
|
|
|
|
|
|
|
|
Expect `post-restore integrity check passed` and exit code 0. For a
|
|
|
|
|
point in time within the retention window, add
|
|
|
|
|
`-timestamp 2026-09-24T12:00:00Z` (UTC). To preview without writing, add
|
|
|
|
|
`-dry-run`.
|
|
|
|
|
|
|
|
|
|
**3. Inspect the restored file** before switching to it — confirm known
|
|
|
|
|
children and definitions are present:
|
|
|
|
|
|
|
|
|
|
dc run --rm --no-deps --entrypoint sqlite3 litestream \
|
|
|
|
|
/var/lib/nestquest/nestquest.db.restored \
|
|
|
|
|
'PRAGMA integrity_check;' \
|
|
|
|
|
'SELECT id, display_name FROM children;' \
|
|
|
|
|
'SELECT id, title FROM quest_definitions;'
|
|
|
|
|
|
|
|
|
|
Expect `ok`, then the household's children and quest definitions.
|
|
|
|
|
|
|
|
|
|
**4. Swap it into place**, keeping the old files (database, its
|
|
|
|
|
`-wal`/`-shm`, and Litestream's local state) in a timestamped directory
|
|
|
|
|
on the volume:
|
|
|
|
|
|
|
|
|
|
dc run --rm --no-deps --entrypoint sh litestream -c '
|
|
|
|
|
set -e
|
|
|
|
|
cd /var/lib/nestquest
|
|
|
|
|
ts=$(date -u +%Y%m%dT%H%M%SZ)
|
|
|
|
|
mkdir -p "pre-restore-$ts"
|
|
|
|
|
for f in nestquest.db nestquest.db-wal nestquest.db-shm .nestquest.db-litestream; do
|
|
|
|
|
if [ -e "$f" ]; then mv "$f" "pre-restore-$ts/"; fi
|
|
|
|
|
done
|
|
|
|
|
mv nestquest.db.restored nestquest.db
|
|
|
|
|
ls -la'
|
|
|
|
|
|
|
|
|
|
**5. Start the API, then the sidecar:**
|
|
|
|
|
|
|
|
|
|
dc start api
|
|
|
|
|
dc ps api # wait for "healthy"
|
|
|
|
|
dc up -d litestream
|
|
|
|
|
dc ps litestream # Up
|
|
|
|
|
dc logs --tail 20 litestream
|
|
|
|
|
|
|
|
|
|
Use `dc up -d litestream`, not `dc start`: on a fresh box step 0 never
|
|
|
|
|
created the sidecar container, and `start` cannot start a container that
|
|
|
|
|
does not exist. `up -d` creates it if needed (or starts the existing one),
|
|
|
|
|
so continuous replication is back online on both paths.
|
|
|
|
|
|
|
|
|
|
The sidecar resumes replication from the restored position (it logs
|
|
|
|
|
`detected database behind replica` / `fetched latest L0 file from
|
|
|
|
|
replica`, then continues normally).
|
|
|
|
|
|
|
|
|
|
**6. Verify through the API** — the panel snapshot lists the restored
|
|
|
|
|
children. On the API box against `10.60.1.14:8080` directly (other LAN
|
|
|
|
|
hosts, including the HA box, are dropped on `:8080` by the section 6
|
|
|
|
|
firewall; from the HA box use the Traefik hostname instead):
|
|
|
|
|
|
|
|
|
|
PANEL_TOKEN=$(grep '^NESTQUEST_PANEL_TOKEN=' deploy/.env | cut -d= -f2-)
|
|
|
|
|
curl -s -H "Authorization: Bearer $PANEL_TOKEN" \
|
|
|
|
|
http://10.60.1.14:8080/api/v1/panel/snapshot
|
|
|
|
|
|
|
|
|
|
Expect HTTP 200 and `children` naming the known child(ren). Then run
|
|
|
|
|
section 9.
|
|
|
|
|
|
|
|
|
|
Once satisfied, remove the old copy:
|
|
|
|
|
`dc run --rm --no-deps --entrypoint rm litestream -rf /var/lib/nestquest/pre-restore-<ts>`.
|
|
|
|
|
|
|
|
|
|
This exact sequence (stop, restore with integrity check, inspect, swap,
|
|
|
|
|
start, snapshot shows the restored child, replication resumes) was
|
|
|
|
|
rehearsed against a local file replica with the committed compose file;
|
|
|
|
|
the live proof against the SFTP replica is recorded separately by the
|
|
|
|
|
controller.
|
|
|
|
|
|
|
|
|
|
## 9. Verification checklist
|
|
|
|
|
|
|
|
|
|
Run after every deploy and every restore. `$PANEL_TOKEN` is the value from
|
|
|
|
|
`deploy/.env`; never paste it into shared logs.
|
|
|
|
|
|
|
|
|
|
| # | Check | Command | Expect |
|
|
|
|
|
| --- | --- | --- | --- |
|
|
|
|
|
| 1 | Health over TLS | `curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/health` | `200` |
|
|
|
|
|
| 2 | Valid certificate | `curl -sv https://nestquest.cubecraftlabs.com/health 2>&1 \| grep -E 'issuer\|expire date'` | Let's Encrypt issuer, future expiry (no `-k` needed) |
|
|
|
|
|
| 3 | Docs public | `curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/docs` | `200` |
|
|
|
|
|
| 4 | Panel refused externally | from any host other than `10.60.1.80` (e.g. a phone off Wi-Fi, or a LAN machine): `curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $PANEL_TOKEN" https://nestquest.cubecraftlabs.com/api/v1/panel/snapshot` | `403` (Traefik allowlist) |
|
|
|
|
|
| 5 | Panel allowed from HA box | on `10.60.1.80`: same command | `200`, snapshot JSON |
|
|
|
|
|
| 6 | Panel needs the token | on `10.60.1.80`, without the header | `401` |
|
|
|
|
|
| 7 | Admin JWT accepted | `curl -s -w '\n%{http_code}\n' -H "Authorization: Bearer $ADMIN_JWT" https://nestquest.cubecraftlabs.com/api/v1/admin/ping` with a token of a `nestquest-admins` member | `200`, `{"status":"ok","sub":...}` |
|
|
|
|
|
| 8 | Non-admin refused | same, with a token of a user **not** in `nestquest-admins` | `403`, "Admin access requires membership in the nestquest-admins group" |
|
|
|
|
|
| 9 | No/invalid token refused | same, without a header or with garbage | `401` |
|
|
|
|
|
| 10 | Panel token refused on admin | same, with `$PANEL_TOKEN` | `403` |
|
|
|
|
|
| 11 | Backup running | `dc exec litestream litestream status` | status `ok` |
|
|
|
|
|
|
|
|
|
|
To obtain `$ADMIN_JWT`: sign in to the admin PWA as the user, open the
|
|
|
|
|
browser developer tools, and copy the session-storage value
|
|
|
|
|
`nestquest.access_token`. Tokens are short-lived and are credentials —
|
|
|
|
|
do not store them.
|
overseerandClaude Opus 5.5
dev_agent
orchestrator-agent