Task 6e203451: Address runbook review findings F-01..F-03

- F-01: document the enforced DOCKER-USER firewall on :8080 (Traefik and
  host RETURN, DROP everyone else), its persisting unit
  nestquest-api-firewall.service, and operator verification commands;
  restore step 6 no longer suggests hitting :8080 from the HA box.
- F-02: restore step 5 uses `dc up -d litestream` so the sidecar is
  created on a fresh box instead of `dc start`, which cannot start a
  never-created container.
- F-03: pin both reference routers to the `websecure` entrypoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-24 08:50:26 -04:00
co-authored by Claude Opus 5.5
parent eb540d804a
commit d1f8f4b15b
+46 -9
View File
@@ -99,7 +99,7 @@ repository. What it must contain:
| panel | ``Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)`` | `ipAllowList`, `sourceRange: ["10.60.1.80/32"]` (HA box only) | `http://10.60.1.14:8080` | | panel | ``Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)`` | `ipAllowList`, `sourceRange: ["10.60.1.80/32"]` (HA box only) | `http://10.60.1.14:8080` |
| admin (everything else: `/health`, `/docs`, `/openapi.json`, `/api/v1/admin/*`) | ``Host(`nestquest.cubecraftlabs.com`)`` | none — public; the API itself enforces Authentik JWTs (section 4) | `http://10.60.1.14:8080` | | admin (everything else: `/health`, `/docs`, `/openapi.json`, `/api/v1/admin/*`) | ``Host(`nestquest.cubecraftlabs.com`)`` | none — public; the API itself enforces Authentik JWTs (section 4) | `http://10.60.1.14:8080` |
Both routers use the HTTPS entrypoint with `tls.certResolver: myresolver` Both routers use the HTTPS entrypoint `websecure` with `tls.certResolver: myresolver`
(HTTP is redirected to HTTPS by the shared Traefik). The panel router's (HTTP is redirected to HTTPS by the shared Traefik). The panel router's
rule is longer, so Traefik's default rule-length priority evaluates it rule is longer, so Traefik's default rule-length priority evaluates it
before the catch-all admin router; if you set explicit priorities, keep before the catch-all admin router; if you set explicit priorities, keep
@@ -113,14 +113,14 @@ http:
routers: routers:
nestquest-panel: nestquest-panel:
rule: "Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)" rule: "Host(`nestquest.cubecraftlabs.com`) && PathPrefix(`/api/v1/panel`)"
entryPoints: [<https entrypoint>] entryPoints: [websecure]
middlewares: [nestquest-panel-allowlist] middlewares: [nestquest-panel-allowlist]
service: nestquest-api service: nestquest-api
tls: tls:
certResolver: myresolver certResolver: myresolver
nestquest-admin: nestquest-admin:
rule: "Host(`nestquest.cubecraftlabs.com`)" rule: "Host(`nestquest.cubecraftlabs.com`)"
entryPoints: [<https entrypoint>] entryPoints: [websecure]
service: nestquest-api service: nestquest-api
tls: tls:
certResolver: myresolver certResolver: myresolver
@@ -255,8 +255,37 @@ committed `deploy/litestream.yml` (`host-key`, the NAS's ECDSA key), so no
nslookup nestquest.cubecraftlabs.com 10.60.1.101 # Address: 10.60.1.33 nslookup nestquest.cubecraftlabs.com 10.60.1.101 # Address: 10.60.1.33
- The API box publishes `8080` only on `10.60.1.14`; the HA box and - **Direct `:8080` access is firewalled.** The API box publishes `8080`
everyone else must use the Traefik hostname, never `:8080` directly. on `10.60.1.14`, which alone would let any LAN host bypass Traefik and
the panel `ipAllowList`. This is enforced on the API box in Docker's
`DOCKER-USER` chain (which Docker evaluates before its own forwarding
rules for published ports), in this order:
-A DOCKER-USER -s 10.60.1.33/32 -p tcp --dport 8080 -j RETURN # Traefik on nukapi-03
-A DOCKER-USER -s 10.60.1.14/32 -p tcp --dport 8080 -j RETURN # the API box itself
-A DOCKER-USER -p tcp --dport 8080 -j DROP # every other source
The rules are persisted by the systemd oneshot unit
**`nestquest-api-firewall.service`** on the API box (enabled; ordered
after `docker.service`, it applies the three rules at every boot).
The HA box and every other client must use the Traefik hostname,
never `:8080` directly. Check it:
sudo systemctl is-enabled nestquest-api-firewall.service # enabled
sudo systemctl is-active nestquest-api-firewall.service # active
sudo iptables -S DOCKER-USER # the three rules above, in that order
# on nukapi-03 (10.60.1.33) — allowed:
curl -s -o /dev/null -w '%{http_code}\n' http://10.60.1.14:8080/health # 200
# on any other LAN host, e.g. 10.60.1.101 — dropped:
curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://10.60.1.14:8080/health # 000, timeout
# public path still works:
curl -s -o /dev/null -w '%{http_code}\n' https://nestquest.cubecraftlabs.com/health # 200
On a rebuilt API box, recreate and enable the unit before exposing the
API (`sudo systemctl enable --now nestquest-api-firewall.service`); the
deployed unit is the authoritative copy
(`systemctl cat nestquest-api-firewall.service`).
## 7. Backup (Litestream to the Synology NAS) ## 7. Backup (Litestream to the Synology NAS)
@@ -305,7 +334,8 @@ API** once so the volume exists and is owned by uid 999:
Do not start `litestream` yet — it would begin replicating the new, Do not start `litestream` yet — it would begin replicating the new,
empty database the API just created. The swap in step 4 moves that empty empty database the API just created. The swap in step 4 moves that empty
file aside. file aside, and step 5 creates and starts the sidecar with
`dc up -d litestream` once the API is healthy.
**1. Stop writers** — both the API and the sidecar (Litestream must not **1. Stop writers** — both the API and the sidecar (Litestream must not
replicate while the file is being replaced): replicate while the file is being replaced):
@@ -356,16 +386,23 @@ on the volume:
dc start api dc start api
dc ps api # wait for "healthy" dc ps api # wait for "healthy"
dc start litestream dc up -d litestream
dc ps litestream # Up
dc logs --tail 20 litestream dc logs --tail 20 litestream
Use `dc up -d litestream`, not `dc start`: on a fresh box step 0 never
created the sidecar container, and `start` cannot start a container that
does not exist. `up -d` creates it if needed (or starts the existing one),
so continuous replication is back online on both paths.
The sidecar resumes replication from the restored position (it logs The sidecar resumes replication from the restored position (it logs
`detected database behind replica` / `fetched latest L0 file from `detected database behind replica` / `fetched latest L0 file from
replica`, then continues normally). replica`, then continues normally).
**6. Verify through the API** — the panel snapshot lists the restored **6. Verify through the API** — the panel snapshot lists the restored
children. From the HA box (the only allowed panel client), or on the API children. On the API box against `10.60.1.14:8080` directly (other LAN
box against `10.60.1.14:8080` directly: hosts, including the HA box, are dropped on `:8080` by the section 6
firewall; from the HA box use the Traefik hostname instead):
PANEL_TOKEN=$(grep '^NESTQUEST_PANEL_TOKEN=' deploy/.env | cut -d= -f2-) PANEL_TOKEN=$(grep '^NESTQUEST_PANEL_TOKEN=' deploy/.env | cut -d= -f2-)
curl -s -H "Authorization: Bearer $PANEL_TOKEN" \ curl -s -H "Authorization: Bearer $PANEL_TOKEN" \