engineering/remarkbox
1
0
Fork 0
Code Issues 3 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Update postmortem with actual root cause and add T14.

Browse source 
The 5-day meta/faq SSL outage was caused by missing proxy blocks in
proxy.unturf.com, not a Caddy cold-start issue. Fixed in proxy commit f12a56d.
This commit is contained in:
russell@unturf.com 2026-03-02 15:22:39 -05:00
parent 4287aeee2c
commit a57187672c
3 changed files with 170 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

227
docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md
View file

@ -1,17 +1,25 @@
# Postmortem: SSL/TLS Outage on meta.remarkbox.com and faq.remarkbox.com
**Date**: 2026-02-25
**Date**: 2026-02-25 (recurred 2026-02-26 through 2026-03-02)
**Severity**: High
**Status**: Resolved
**Duration**: ~8 hours (approx 03:44 UTC - 12:00 UTC)
**Duration**: Initial ~8 hours; recurrence ~5 days
## Summary
meta.remarkbox.com and faq.remarkbox.com went down with SSL protocol errors
(`ERR_SSL_PROTOCOL_ERROR`) after the nginx-to-Caddy migration. The domains were
missing from the Caddyfile entirely. Once added, ACME certificate provisioning
failed due to a cold-start chicken-and-egg problem where all 7 domains needed
certs simultaneously and neither HTTP-01 nor TLS-ALPN-01 challenges could succeed.
(`ERR_SSL_PROTOCOL_ERROR`) after the nginx-to-Caddy migration. The initial
outage (2026-02-25) was caused by missing domains in the remarkbox server's
Caddyfile and a cold-start chicken-and-egg problem. A temporary fix using
single-domain bootstrap resolved it, but the domains went down again on each
subsequent deploy because the **actual root cause** was in the ingress proxy
(`proxy.unturf.com` at 142.93.73.64), not the remarkbox server.
meta and faq are CNAMEs to `my.remarkbox.com`, which resolves to the ingress
proxy. The proxy's Caddyfile had no blocks for meta or faq, so they fell through
to the MPS on-demand TLS catch-all — routing to the wrong backend entirely.
The remarkbox server's Caddy could never obtain certs for these domains because
ACME challenges were directed at the proxy, not the backend.
## Impact
@ -23,38 +31,40 @@ certs simultaneously and neither HTTP-01 nor TLS-ALPN-01 challenges could succee
## Root Cause
Three layered issues:
### The actual root cause: missing proxy blocks
### 1. Missing domains in Caddyfile (primary)
meta.remarkbox.com and faq.remarkbox.com are CNAME records pointing to
`my.remarkbox.com`, which resolves to `142.93.73.64` — the ingress proxy
(`proxy.unturf.com`). The proxy's Caddyfile (`~/git/proxy.unturf.com/ingress/Caddyfile`)
had a block for `my.remarkbox.com` that reverse-proxied to `origin.remarkbox.com`,
but **no blocks for meta or faq**.
During the nginx-to-Caddy migration (commit 2d64d79), meta.remarkbox.com and
faq.remarkbox.com were not included in the new Caddyfile pillar at
`foxhop-pillar/caddy/remarkbox.sls`. Caddy had no configuration for these
domains, so it could not serve them or obtain certificates.
Without explicit blocks, requests to meta/faq fell through to the catch-all
`https://` block, which uses on-demand TLS and proxies to
`origin.makepostsell.com` — a completely different backend. This meant:
### 2. Explicit HTTP blocks hijacking port 80
1. The proxy never obtained TLS certs for meta/faq (on-demand TLS asked MPS
origin, which rejected these domains)
2. The remarkbox server's Caddy could never obtain certs either, because ACME
challenges (both HTTP-01 and TLS-ALPN-01) were directed at the proxy IP,
not the backend server
An early fix attempt added explicit `http://meta.remarkbox.com` and
`http://faq.remarkbox.com` blocks to redirect HTTP to HTTPS. This created a
separate Caddy HTTP server (`srv1`) on port 80 that handled requests for those
domains WITHOUT injecting Caddy's built-in ACME HTTP-01 challenge handler.
Result: Let's Encrypt HTTP-01 validation requests hit the redirect block,
got a 308 redirect to HTTPS (which had no cert), and failed.
### Contributing factors during initial investigation (2026-02-25)
### 3. Caddy cold-start chicken-and-egg problem
1. **Missing domains in remarkbox Caddyfile**: meta/faq were also missing from
the remarkbox server's Caddyfile pillar (`foxhop-pillar/caddy/remarkbox.sls`)
after the nginx-to-Caddy migration (commit 2d64d79). This was a real issue
but fixing it alone could not resolve the outage because the proxy was the
TLS termination point.
After removing the explicit HTTP blocks and nuking all cert state to start
fresh, Caddy needed to provision certs for ALL 7 domains simultaneously:
2. **Explicit HTTP blocks hijacking port 80**: An early fix attempt added
explicit `http://` blocks that created a separate HTTP server without Caddy's
built-in ACME handler, breaking HTTP-01 challenges.
- **HTTP-01 fails on cold start**: Caddy's auto-HTTPS redirects port 80 to 443,
but port 443 has no cert yet. Let's Encrypt follows the redirect, hits an
incomplete TLS handshake, and fails.
- **TLS-ALPN-01 fails on cold start**: Caddy can't present the ACME challenge
cert because the TLS listener isn't fully ready when zero certs exist.
This is a known Caddy behavior, NOT a bug. Confirmed by testing with both the
custom Caddy binary (with L4/ratelimit/cache modules) and a stock Caddy v2.11.1
binary — identical failure.
3. **Cold-start chicken-and-egg**: Nuking cert state and restarting Caddy caused
all domains to need certs simultaneously, which can fail when the TLS listener
has zero certs. This was a red herring — the real problem was that ACME
challenges never reached the remarkbox server regardless.
## Timeline (UTC)
@ -84,98 +94,121 @@ binary — identical failure.
- **11:15** — All 7 domains confirmed working with valid TLS certs
- **11:30** — Final pillar committed (d3d68f5) and pushed
### Recurrence (2026-02-26 through 2026-03-02)
- **Feb 26** — Pushing the postmortem commit triggered CI/CD deploy, which
restarted Caddy on the remarkbox server. meta/faq lost certs again because
the single-domain bootstrap was a one-time workaround, not a permanent fix.
- **Feb 26-Mar 1** — Multiple debugging attempts: `on_demand_tls`,
`auto_https disable_redirects`, self-signed cert bootstrapping, testing
from the server locally (`openssl s_client` showed ACME challenges working
on localhost but failing externally). Discovered server IP (162.243.167.224)
differed from DNS IP (142.93.73.64) — initially attributed to a "floating IP
proxy" stripping TLS-ALPN-01 extensions.
- **Mar 2** — Identified the **actual root cause**: 142.93.73.64 is the
`proxy.unturf.com` ingress proxy running Caddy, not a transparent floating IP.
The proxy's Caddyfile had no blocks for meta/faq. Added the blocks, reloaded
the proxy's Caddy, and both domains came up immediately with valid LE certs.
## Resolution
### The fix: two-phase cert bootstrap
### The permanent fix: add proxy blocks (2026-03-02)
1. Write a minimal Caddyfile with only one domain (`my.remarkbox.com`) and the
global `{email admin@remarkbox.com}` block
2. Nuke existing cert state: `rm -rf /root/.local/share/caddy/`
3. Start Caddy — single domain gets cert via TLS-ALPN-01
4. Write the full Caddyfile with all 7 domains
5. Reload Caddy (`caddy reload`, NOT restart) — remaining domains obtain certs
because the TLS listener is already active with a valid cert
Added `meta.remarkbox.com` and `faq.remarkbox.com` blocks to the ingress
proxy Caddyfile (`proxy.unturf.com/ingress/Caddyfile`), identical to the
existing `my.remarkbox.com` block:
```
meta.remarkbox.com {
forward_auth localhost:8003 {
uri /assholes/gate
}
reverse_proxy https://origin.remarkbox.com {
header_up Host {http.request.host}
header_up X-Real-IP {http.request.remote.host}
header_up X-Forwarded-For {http.request.remote.host}
header_up X-Forwarded-Proto {http.request.scheme}
transport http {
tls_server_name origin.remarkbox.com
}
}
log {
output file /var/log/caddy/meta.remarkbox.com.log
}
}
```
The proxy handles TLS termination and ACME for meta/faq. Traffic forwards to
the remarkbox backend via `origin.remarkbox.com`. The remarkbox server's Caddy
no longer needs to obtain certs for these domains — the proxy owns that
responsibility.
Commit: `f12a56d` in `proxy.unturf.com` repo.
### Earlier workaround: two-phase cert bootstrap (2026-02-25)
The initial fix used single-domain bootstrap on the remarkbox server to obtain
certs. This worked temporarily but broke on every Caddy restart because the
underlying proxy routing was wrong.
### Pillar changes (foxhop-pillar)
Three commits during the incident:
| Commit | Change |
|--------|--------|
| `12417db` | Remove explicit HTTP blocks and `disable_tlsalpn` |
| `3c9ca52` | Force TLS-ALPN-01 by disabling HTTP challenge (reverted in final) |
| `3c9ca52` | Force TLS-ALPN-01 by disabling HTTP challenge (reverted) |
| `d3d68f5` | Final: add `{email admin@remarkbox.com}` global block, clean config |
### Final Caddyfile structure
### Architecture after fix
```
{
email admin@remarkbox.com
}
remarkbox.com { redir https://www.remarkbox.com{uri} }
www.remarkbox.com { root * /www/www.remarkbox.com; file_server }
my.remarkbox.com { reverse_proxy localhost:6001 }
origin.remarkbox.com { reverse_proxy localhost:6001 }
meta.remarkbox.com { reverse_proxy localhost:6001 }
faq.remarkbox.com { reverse_proxy localhost:6001 }
westworld2.com { reverse_proxy localhost:6002 }
Client → meta.remarkbox.com (CNAME → my.remarkbox.com → 142.93.73.64)
→ proxy.unturf.com Caddy (TLS termination, ACME, cert management)
→ origin.remarkbox.com (162.243.167.224, remarkbox backend Caddy)
→ localhost:6001 (uwsgi, Host header determines namespace)
```
All domains use standard HTTPS blocks. No explicit HTTP blocks. Caddy handles
ACME challenges, redirects, and cert renewal automatically.
### Certificate authorities in use
- **Let's Encrypt** — primary CA for most domains
- **ZeroSSL** — automatic fallback for domains where LE orders went stale
## Lessons Learned
1. **Never use explicit `http://` site blocks in Caddy for domains that need
1. **Know which server terminates TLS.** When domains use CNAMEs through a
proxy, the proxy must have explicit blocks for those domains. The backend
server cannot obtain ACME certs for domains whose DNS points elsewhere.
This was the fundamental misunderstanding that prolonged the outage by 5 days.
2. **Trace the full request path before debugging.** The investigation spent
days debugging ACME on the remarkbox server when the problem was on the
proxy. A `dig` + understanding of the proxy architecture would have
identified this immediately.
3. **Catch-all blocks mask routing errors.** The proxy's `https://` on-demand
TLS catch-all silently absorbed meta/faq requests and routed them to the
wrong backend, producing TLS errors instead of a clear "no route" signal.
4. **Never use explicit `http://` site blocks in Caddy for domains that need
auto-HTTPS.** They create a separate HTTP server that hijacks port 80 without
the ACME handler. Let Caddy manage HTTP-to-HTTPS redirects automatically.
the ACME handler.
2. **Caddy cannot cold-start with many domains needing certs simultaneously.**
Both HTTP-01 and TLS-ALPN-01 can fail when the TLS listener has zero certs.
Bootstrap one domain first, then reload with the rest.
5. **`caddy reload` preserves TLS state; `systemctl restart caddy` does not.**
Always prefer reload when updating the Caddyfile.
3. **`caddy reload` preserves TLS state; `systemctl restart caddy` does not.**
Always prefer reload when updating the Caddyfile. A restart re-triggers the
cold-start problem if cert state is missing.
4. **Custom Caddy modules (L4, ratelimit, cache) do NOT affect ACME behavior.**
Confirmed by testing with stock Caddy v2.11.1 — identical failure. Don't
blame custom builds before testing.
5. **CNAME records work fine with Caddy ACME.** meta and faq use CNAME pointing
to my.remarkbox.com. This is not the issue — the cert is issued for the
hostname, not the IP.
6. **Caddy stores cert data under the running user's home directory.**
Running as root: `/root/.local/share/caddy/`. NOT `/var/lib/caddy/`.
Know your data paths when debugging.
7. **Let's Encrypt has rate limits that bite during incident response.**
Failed Validations: 5 per account per hostname per hour. After exhausting
these, Caddy fell back to ZeroSSL automatically — a useful safety net.
6. **Let's Encrypt has rate limits that bite during incident response.**
Failed Validations: 5 per account per hostname per hour. Caddy fell back
to ZeroSSL automatically — a useful safety net.
## Prevention
- [ ] Add a smoke test to salt highstate that verifies all expected domains
are present in the rendered Caddyfile before applying
- [ ] Document the single-domain bootstrap procedure in the remarkbox ops
runbook for future Caddy cert state recovery
- [ ] Consider adding `on_demand_tls` as a safety net for CNAME domains
- [ ] Add monitoring/alerting for SSL certificate expiry across all domains
- [ ] When adding CNAME domains that route through the ingress proxy, always
add corresponding blocks to `proxy.unturf.com/ingress/Caddyfile`
- [ ] Add monitoring/alerting for SSL certificate validity across all domains
- [ ] Document the proxy architecture in the remarkbox ops runbook: which
domains go through the proxy vs direct
## Action Items
- [x] Add meta.remarkbox.com and faq.remarkbox.com to Caddyfile
- [x] Add meta.remarkbox.com and faq.remarkbox.com to remarkbox Caddyfile
- [x] Add meta.remarkbox.com and faq.remarkbox.com to proxy Caddyfile (f12a56d)
- [x] Remove explicit HTTP blocks that broke ACME
- [x] Add global `{email admin@remarkbox.com}` for ACME registration
- [x] Bootstrap certs via single-domain approach
- [x] Verify all 7 domains serving valid TLS
- [x] Push final pillar config (d3d68f5)
- [x] Verify all domains serving valid TLS
- [x] Push final proxy config
- [x] Document in postmortem
- [ ] Verify next salt highstate applies cleanly without cert loss

39
docs/tickets/14.md Normal file
View file

@ -0,0 +1,39 @@
# T14: meta/faq SSL outage — missing proxy blocks
**Status**: resolved
**Priority**: critical
**Created**: 2026-03-02
**Resolved**: 2026-03-02
## Problem
meta.remarkbox.com and faq.remarkbox.com were down for ~5 days (2026-02-25
through 2026-03-02) with `ERR_SSL_PROTOCOL_ERROR`. Both domains are CNAMEs
to `my.remarkbox.com` which resolves to `142.93.73.64` — the ingress proxy
(`proxy.unturf.com`).
The proxy's Caddyfile had no explicit blocks for meta or faq. Requests fell
through to the MPS on-demand TLS catch-all (`https://`), which routed them to
`origin.makepostsell.com` instead of `origin.remarkbox.com`. The proxy never
obtained TLS certs for these domains, and the remarkbox server couldn't either
because ACME challenges went to the proxy.
## Root Cause
Missing `meta.remarkbox.com` and `faq.remarkbox.com` blocks in
`proxy.unturf.com/ingress/Caddyfile`. The initial investigation (2026-02-25)
focused on the remarkbox server's Caddy and ACME behavior, missing that the
proxy was the TLS termination point for CNAME domains.
## Resolution
Added proxy blocks for both domains in `proxy.unturf.com/ingress/Caddyfile`,
identical to the existing `my.remarkbox.com` block — forwarding to
`origin.remarkbox.com` with the original Host header preserved.
Commit `f12a56d` in `proxy.unturf.com` repo. Both domains came up immediately
after `caddy reload` on the proxy.
## Related
- Postmortem: `docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md`

1
docs/tickets/index.md
View file

@ -18,3 +18,4 @@ Tracked issues from the meta.remarkbox.com and faq.remarkbox.com audit (2026-02-
| [T11](11.md) | Document "comments disappear when moving embed" | resolved | low | FAQ `6260e726` |
| [T12](12.md) | Reply to API-only CRUD thread confirming done | resolved | low | meta `6db01560` |
| [T13](13.md) | Reply to lock/archive thread confirming done | resolved | low | meta `7e9d5864` |
| [T14](14.md) | meta/faq SSL outage — missing proxy blocks | resolved | critical | postmortem 2026-02-25 |