remarkbox/docs/tickets/14.md
russell@unturf.com a57187672c Update postmortem with actual root cause and add T14.
The 5-day meta/faq SSL outage was caused by missing proxy blocks in
proxy.unturf.com, not a Caddy cold-start issue. Fixed in proxy commit f12a56d.
2026-03-02 15:22:39 -05:00

1.4 KiB

T14: meta/faq SSL outage — missing proxy blocks

Status: resolved Priority: critical Created: 2026-03-02 Resolved: 2026-03-02

Problem

meta.remarkbox.com and faq.remarkbox.com were down for ~5 days (2026-02-25 through 2026-03-02) with ERR_SSL_PROTOCOL_ERROR. Both domains are CNAMEs to my.remarkbox.com which resolves to 142.93.73.64 — the ingress proxy (proxy.unturf.com).

The proxy's Caddyfile had no explicit blocks for meta or faq. Requests fell through to the MPS on-demand TLS catch-all (https://), which routed them to origin.makepostsell.com instead of origin.remarkbox.com. The proxy never obtained TLS certs for these domains, and the remarkbox server couldn't either because ACME challenges went to the proxy.

Root Cause

Missing meta.remarkbox.com and faq.remarkbox.com blocks in proxy.unturf.com/ingress/Caddyfile. The initial investigation (2026-02-25) focused on the remarkbox server's Caddy and ACME behavior, missing that the proxy was the TLS termination point for CNAME domains.

Resolution

Added proxy blocks for both domains in proxy.unturf.com/ingress/Caddyfile, identical to the existing my.remarkbox.com block — forwarding to origin.remarkbox.com with the original Host header preserved.

Commit f12a56d in proxy.unturf.com repo. Both domains came up immediately after caddy reload on the proxy.

  • Postmortem: docs/postmortem-2026-02-25-ssl-outage-caddy-acme.md