diff --git a/docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md b/docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md index e42cd9d..d32a040 100644 --- a/docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md +++ b/docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md @@ -1,17 +1,25 @@ # Postmortem: SSL/TLS Outage on meta.remarkbox.com and faq.remarkbox.com -**Date**: 2026-02-25 +**Date**: 2026-02-25 (recurred 2026-02-26 through 2026-03-02) **Severity**: High **Status**: Resolved -**Duration**: ~8 hours (approx 03:44 UTC - 12:00 UTC) +**Duration**: Initial ~8 hours; recurrence ~5 days ## Summary meta.remarkbox.com and faq.remarkbox.com went down with SSL protocol errors -(`ERR_SSL_PROTOCOL_ERROR`) after the nginx-to-Caddy migration. The domains were -missing from the Caddyfile entirely. Once added, ACME certificate provisioning -failed due to a cold-start chicken-and-egg problem where all 7 domains needed -certs simultaneously and neither HTTP-01 nor TLS-ALPN-01 challenges could succeed. +(`ERR_SSL_PROTOCOL_ERROR`) after the nginx-to-Caddy migration. The initial +outage (2026-02-25) was caused by missing domains in the remarkbox server's +Caddyfile and a cold-start chicken-and-egg problem. A temporary fix using +single-domain bootstrap resolved it, but the domains went down again on each +subsequent deploy because the **actual root cause** was in the ingress proxy +(`proxy.unturf.com` at 142.93.73.64), not the remarkbox server. + +meta and faq are CNAMEs to `my.remarkbox.com`, which resolves to the ingress +proxy. The proxy's Caddyfile had no blocks for meta or faq, so they fell through +to the MPS on-demand TLS catch-all — routing to the wrong backend entirely. +The remarkbox server's Caddy could never obtain certs for these domains because +ACME challenges were directed at the proxy, not the backend. ## Impact @@ -23,38 +31,40 @@ certs simultaneously and neither HTTP-01 nor TLS-ALPN-01 challenges could succee ## Root Cause -Three layered issues: +### The actual root cause: missing proxy blocks -### 1. Missing domains in Caddyfile (primary) +meta.remarkbox.com and faq.remarkbox.com are CNAME records pointing to +`my.remarkbox.com`, which resolves to `142.93.73.64` — the ingress proxy +(`proxy.unturf.com`). The proxy's Caddyfile (`~/git/proxy.unturf.com/ingress/Caddyfile`) +had a block for `my.remarkbox.com` that reverse-proxied to `origin.remarkbox.com`, +but **no blocks for meta or faq**. -During the nginx-to-Caddy migration (commit 2d64d79), meta.remarkbox.com and -faq.remarkbox.com were not included in the new Caddyfile pillar at -`foxhop-pillar/caddy/remarkbox.sls`. Caddy had no configuration for these -domains, so it could not serve them or obtain certificates. +Without explicit blocks, requests to meta/faq fell through to the catch-all +`https://` block, which uses on-demand TLS and proxies to +`origin.makepostsell.com` — a completely different backend. This meant: -### 2. Explicit HTTP blocks hijacking port 80 +1. The proxy never obtained TLS certs for meta/faq (on-demand TLS asked MPS + origin, which rejected these domains) +2. The remarkbox server's Caddy could never obtain certs either, because ACME + challenges (both HTTP-01 and TLS-ALPN-01) were directed at the proxy IP, + not the backend server -An early fix attempt added explicit `http://meta.remarkbox.com` and -`http://faq.remarkbox.com` blocks to redirect HTTP to HTTPS. This created a -separate Caddy HTTP server (`srv1`) on port 80 that handled requests for those -domains WITHOUT injecting Caddy's built-in ACME HTTP-01 challenge handler. -Result: Let's Encrypt HTTP-01 validation requests hit the redirect block, -got a 308 redirect to HTTPS (which had no cert), and failed. +### Contributing factors during initial investigation (2026-02-25) -### 3. Caddy cold-start chicken-and-egg problem +1. **Missing domains in remarkbox Caddyfile**: meta/faq were also missing from + the remarkbox server's Caddyfile pillar (`foxhop-pillar/caddy/remarkbox.sls`) + after the nginx-to-Caddy migration (commit 2d64d79). This was a real issue + but fixing it alone could not resolve the outage because the proxy was the + TLS termination point. -After removing the explicit HTTP blocks and nuking all cert state to start -fresh, Caddy needed to provision certs for ALL 7 domains simultaneously: +2. **Explicit HTTP blocks hijacking port 80**: An early fix attempt added + explicit `http://` blocks that created a separate HTTP server without Caddy's + built-in ACME handler, breaking HTTP-01 challenges. -- **HTTP-01 fails on cold start**: Caddy's auto-HTTPS redirects port 80 to 443, - but port 443 has no cert yet. Let's Encrypt follows the redirect, hits an - incomplete TLS handshake, and fails. -- **TLS-ALPN-01 fails on cold start**: Caddy can't present the ACME challenge - cert because the TLS listener isn't fully ready when zero certs exist. - -This is a known Caddy behavior, NOT a bug. Confirmed by testing with both the -custom Caddy binary (with L4/ratelimit/cache modules) and a stock Caddy v2.11.1 -binary — identical failure. +3. **Cold-start chicken-and-egg**: Nuking cert state and restarting Caddy caused + all domains to need certs simultaneously, which can fail when the TLS listener + has zero certs. This was a red herring — the real problem was that ACME + challenges never reached the remarkbox server regardless. ## Timeline (UTC) @@ -84,98 +94,121 @@ binary — identical failure. - **11:15** — All 7 domains confirmed working with valid TLS certs - **11:30** — Final pillar committed (d3d68f5) and pushed +### Recurrence (2026-02-26 through 2026-03-02) + +- **Feb 26** — Pushing the postmortem commit triggered CI/CD deploy, which + restarted Caddy on the remarkbox server. meta/faq lost certs again because + the single-domain bootstrap was a one-time workaround, not a permanent fix. +- **Feb 26-Mar 1** — Multiple debugging attempts: `on_demand_tls`, + `auto_https disable_redirects`, self-signed cert bootstrapping, testing + from the server locally (`openssl s_client` showed ACME challenges working + on localhost but failing externally). Discovered server IP (162.243.167.224) + differed from DNS IP (142.93.73.64) — initially attributed to a "floating IP + proxy" stripping TLS-ALPN-01 extensions. +- **Mar 2** — Identified the **actual root cause**: 142.93.73.64 is the + `proxy.unturf.com` ingress proxy running Caddy, not a transparent floating IP. + The proxy's Caddyfile had no blocks for meta/faq. Added the blocks, reloaded + the proxy's Caddy, and both domains came up immediately with valid LE certs. + ## Resolution -### The fix: two-phase cert bootstrap +### The permanent fix: add proxy blocks (2026-03-02) -1. Write a minimal Caddyfile with only one domain (`my.remarkbox.com`) and the - global `{email admin@remarkbox.com}` block -2. Nuke existing cert state: `rm -rf /root/.local/share/caddy/` -3. Start Caddy — single domain gets cert via TLS-ALPN-01 -4. Write the full Caddyfile with all 7 domains -5. Reload Caddy (`caddy reload`, NOT restart) — remaining domains obtain certs - because the TLS listener is already active with a valid cert +Added `meta.remarkbox.com` and `faq.remarkbox.com` blocks to the ingress +proxy Caddyfile (`proxy.unturf.com/ingress/Caddyfile`), identical to the +existing `my.remarkbox.com` block: + +``` +meta.remarkbox.com { + forward_auth localhost:8003 { + uri /assholes/gate + } + reverse_proxy https://origin.remarkbox.com { + header_up Host {http.request.host} + header_up X-Real-IP {http.request.remote.host} + header_up X-Forwarded-For {http.request.remote.host} + header_up X-Forwarded-Proto {http.request.scheme} + transport http { + tls_server_name origin.remarkbox.com + } + } + log { + output file /var/log/caddy/meta.remarkbox.com.log + } +} +``` + +The proxy handles TLS termination and ACME for meta/faq. Traffic forwards to +the remarkbox backend via `origin.remarkbox.com`. The remarkbox server's Caddy +no longer needs to obtain certs for these domains — the proxy owns that +responsibility. + +Commit: `f12a56d` in `proxy.unturf.com` repo. + +### Earlier workaround: two-phase cert bootstrap (2026-02-25) + +The initial fix used single-domain bootstrap on the remarkbox server to obtain +certs. This worked temporarily but broke on every Caddy restart because the +underlying proxy routing was wrong. ### Pillar changes (foxhop-pillar) -Three commits during the incident: - | Commit | Change | |--------|--------| | `12417db` | Remove explicit HTTP blocks and `disable_tlsalpn` | -| `3c9ca52` | Force TLS-ALPN-01 by disabling HTTP challenge (reverted in final) | +| `3c9ca52` | Force TLS-ALPN-01 by disabling HTTP challenge (reverted) | | `d3d68f5` | Final: add `{email admin@remarkbox.com}` global block, clean config | -### Final Caddyfile structure +### Architecture after fix ``` -{ - email admin@remarkbox.com -} - -remarkbox.com { redir https://www.remarkbox.com{uri} } -www.remarkbox.com { root * /www/www.remarkbox.com; file_server } -my.remarkbox.com { reverse_proxy localhost:6001 } -origin.remarkbox.com { reverse_proxy localhost:6001 } -meta.remarkbox.com { reverse_proxy localhost:6001 } -faq.remarkbox.com { reverse_proxy localhost:6001 } -westworld2.com { reverse_proxy localhost:6002 } +Client → meta.remarkbox.com (CNAME → my.remarkbox.com → 142.93.73.64) + → proxy.unturf.com Caddy (TLS termination, ACME, cert management) + → origin.remarkbox.com (162.243.167.224, remarkbox backend Caddy) + → localhost:6001 (uwsgi, Host header determines namespace) ``` -All domains use standard HTTPS blocks. No explicit HTTP blocks. Caddy handles -ACME challenges, redirects, and cert renewal automatically. - -### Certificate authorities in use - -- **Let's Encrypt** — primary CA for most domains -- **ZeroSSL** — automatic fallback for domains where LE orders went stale - ## Lessons Learned -1. **Never use explicit `http://` site blocks in Caddy for domains that need +1. **Know which server terminates TLS.** When domains use CNAMEs through a + proxy, the proxy must have explicit blocks for those domains. The backend + server cannot obtain ACME certs for domains whose DNS points elsewhere. + This was the fundamental misunderstanding that prolonged the outage by 5 days. + +2. **Trace the full request path before debugging.** The investigation spent + days debugging ACME on the remarkbox server when the problem was on the + proxy. A `dig` + understanding of the proxy architecture would have + identified this immediately. + +3. **Catch-all blocks mask routing errors.** The proxy's `https://` on-demand + TLS catch-all silently absorbed meta/faq requests and routed them to the + wrong backend, producing TLS errors instead of a clear "no route" signal. + +4. **Never use explicit `http://` site blocks in Caddy for domains that need auto-HTTPS.** They create a separate HTTP server that hijacks port 80 without - the ACME handler. Let Caddy manage HTTP-to-HTTPS redirects automatically. + the ACME handler. -2. **Caddy cannot cold-start with many domains needing certs simultaneously.** - Both HTTP-01 and TLS-ALPN-01 can fail when the TLS listener has zero certs. - Bootstrap one domain first, then reload with the rest. +5. **`caddy reload` preserves TLS state; `systemctl restart caddy` does not.** + Always prefer reload when updating the Caddyfile. -3. **`caddy reload` preserves TLS state; `systemctl restart caddy` does not.** - Always prefer reload when updating the Caddyfile. A restart re-triggers the - cold-start problem if cert state is missing. - -4. **Custom Caddy modules (L4, ratelimit, cache) do NOT affect ACME behavior.** - Confirmed by testing with stock Caddy v2.11.1 — identical failure. Don't - blame custom builds before testing. - -5. **CNAME records work fine with Caddy ACME.** meta and faq use CNAME pointing - to my.remarkbox.com. This is not the issue — the cert is issued for the - hostname, not the IP. - -6. **Caddy stores cert data under the running user's home directory.** - Running as root: `/root/.local/share/caddy/`. NOT `/var/lib/caddy/`. - Know your data paths when debugging. - -7. **Let's Encrypt has rate limits that bite during incident response.** - Failed Validations: 5 per account per hostname per hour. After exhausting - these, Caddy fell back to ZeroSSL automatically — a useful safety net. +6. **Let's Encrypt has rate limits that bite during incident response.** + Failed Validations: 5 per account per hostname per hour. Caddy fell back + to ZeroSSL automatically — a useful safety net. ## Prevention -- [ ] Add a smoke test to salt highstate that verifies all expected domains - are present in the rendered Caddyfile before applying -- [ ] Document the single-domain bootstrap procedure in the remarkbox ops - runbook for future Caddy cert state recovery -- [ ] Consider adding `on_demand_tls` as a safety net for CNAME domains -- [ ] Add monitoring/alerting for SSL certificate expiry across all domains +- [ ] When adding CNAME domains that route through the ingress proxy, always + add corresponding blocks to `proxy.unturf.com/ingress/Caddyfile` +- [ ] Add monitoring/alerting for SSL certificate validity across all domains +- [ ] Document the proxy architecture in the remarkbox ops runbook: which + domains go through the proxy vs direct ## Action Items -- [x] Add meta.remarkbox.com and faq.remarkbox.com to Caddyfile +- [x] Add meta.remarkbox.com and faq.remarkbox.com to remarkbox Caddyfile +- [x] Add meta.remarkbox.com and faq.remarkbox.com to proxy Caddyfile (f12a56d) - [x] Remove explicit HTTP blocks that broke ACME - [x] Add global `{email admin@remarkbox.com}` for ACME registration -- [x] Bootstrap certs via single-domain approach -- [x] Verify all 7 domains serving valid TLS -- [x] Push final pillar config (d3d68f5) +- [x] Verify all domains serving valid TLS +- [x] Push final proxy config - [x] Document in postmortem -- [ ] Verify next salt highstate applies cleanly without cert loss diff --git a/docs/tickets/14.md b/docs/tickets/14.md new file mode 100644 index 0000000..a329db1 --- /dev/null +++ b/docs/tickets/14.md @@ -0,0 +1,39 @@ +# T14: meta/faq SSL outage — missing proxy blocks + +**Status**: resolved +**Priority**: critical +**Created**: 2026-03-02 +**Resolved**: 2026-03-02 + +## Problem + +meta.remarkbox.com and faq.remarkbox.com were down for ~5 days (2026-02-25 +through 2026-03-02) with `ERR_SSL_PROTOCOL_ERROR`. Both domains are CNAMEs +to `my.remarkbox.com` which resolves to `142.93.73.64` — the ingress proxy +(`proxy.unturf.com`). + +The proxy's Caddyfile had no explicit blocks for meta or faq. Requests fell +through to the MPS on-demand TLS catch-all (`https://`), which routed them to +`origin.makepostsell.com` instead of `origin.remarkbox.com`. The proxy never +obtained TLS certs for these domains, and the remarkbox server couldn't either +because ACME challenges went to the proxy. + +## Root Cause + +Missing `meta.remarkbox.com` and `faq.remarkbox.com` blocks in +`proxy.unturf.com/ingress/Caddyfile`. The initial investigation (2026-02-25) +focused on the remarkbox server's Caddy and ACME behavior, missing that the +proxy was the TLS termination point for CNAME domains. + +## Resolution + +Added proxy blocks for both domains in `proxy.unturf.com/ingress/Caddyfile`, +identical to the existing `my.remarkbox.com` block — forwarding to +`origin.remarkbox.com` with the original Host header preserved. + +Commit `f12a56d` in `proxy.unturf.com` repo. Both domains came up immediately +after `caddy reload` on the proxy. + +## Related + +- Postmortem: `docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md` diff --git a/docs/tickets/index.md b/docs/tickets/index.md index ee1cd3d..0c40fb6 100644 --- a/docs/tickets/index.md +++ b/docs/tickets/index.md @@ -18,3 +18,4 @@ Tracked issues from the meta.remarkbox.com and faq.remarkbox.com audit (2026-02- | [T11](11.md) | Document "comments disappear when moving embed" | resolved | low | FAQ `6260e726` | | [T12](12.md) | Reply to API-only CRUD thread confirming done | resolved | low | meta `6db01560` | | [T13](13.md) | Reply to lock/archive thread confirming done | resolved | low | meta `7e9d5864` | +| [T14](14.md) | meta/faq SSL outage — missing proxy blocks | resolved | critical | postmortem 2026-02-25 |