make_post_sell/docs/tickets/mps-4.md

MPS-4: Eliminate intermittent 502s from uWSGI worker recycling

Problem

Visitors intermittently see 502 Bad Gateway errors that resolve on refresh. Root cause: uWSGI workers hit the --reload-on-rss 256 memory limit every 4-5 minutes under normal watch mode traffic, triggering a kill+respawn cycle. With only 2 workers (--processes=2), when both recycle near-simultaneously Caddy's reverse_proxy gets no available backend and returns 502.

Evidence (2026-02-11 ~09:15-09:22 UTC)

Worker memory growth (from ps aux):

• Worker 882747 (spawned 09:18): 242MB RSS after 4 minutes
• Worker 882737 (spawned 09:17): 188MB RSS after 5 minutes
• Both approaching the 256MB kill threshold simultaneously

Worker recycling frequency (from journalctl):

09:15:24 - worker 2 (882687) "Seeya!" → killed → Respawned as 882699
09:15:29 - worker 1 (882640) "Seeya!" → killed → Respawned as 882708
09:15:39 - worker 2 (882699) "Seeya!" → killed → Respawned as 882717
09:17:36 - worker 2 (882717) "Seeya!" → killed → Respawned as 882737
09:18:22 - worker 1 (882708) "Seeya!" → killed → Respawned as 882747

Workers survive only ~15 seconds to ~3 minutes under load before hitting the RSS limit. The 09:15:24 and 09:15:29 kills are only 5 seconds apart — both workers recycling nearly simultaneously.

Cold start penalty: First request after respawn takes 400-600ms (vs normal 100-130ms) while the app re-initializes:

• 882699 first request: 413ms
• 882717 first request: 409ms
• 882747 first request: 533ms

System resources: 4GB total RAM, 1.9GB swap used — memory pressure.

Current uWSGI config

--reload-on-rss 256
--processes=2
--threads 8
--max-requests 10000
--http=127.0.0.1:6001

Solution

Tune uWSGI config to prevent simultaneous worker unavailability:

1. Raise RSS limit

Raise --reload-on-rss from 256 to 512. Workers currently grow to 242MB in 4 minutes — 256 is too aggressive and causes constant churn. At 512MB with 2 workers, worst case is ~1GB for workers, still well within the 4GB system budget (Caddy + master + crypto_watcher use ~300MB combined).

2. Add --reload-on-rss-stagger

If available in the installed uWSGI version, or use --max-requests with variance (--max-requests-delta) to prevent both workers from recycling at the same instant. Set --max-requests-delta 1000 to add randomness (each worker gets max-requests ± 1000).

3. Use lazy-apps mode

Add --lazy-apps so each worker loads the application independently after fork. This costs a bit more memory but means the master doesn't need to re-fork the full app — workers initialize in parallel and the surviving worker keeps serving while the new one starts.

4. Add --harakiri timeout

Add --harakiri 30 as a safety net — if any request takes >30 seconds (stuck DB query, deadlock), kill that worker instead of blocking a slot forever.

Proposed new config

--reload-on-rss 512
--processes=2
--threads 8
--max-requests 10000
--max-requests-delta 2000
--harakiri 30
--die-on-term
--http=127.0.0.1:6001
--lazy-apps

Testing

1. Apply config change on prod (systemctl edit --full my.makepostsell.com)
2. systemctl restart my.makepostsell.com
3. Monitor with: watch -n 5 'ps -o pid,rss,vsz,etimes,cmd -p $(pgrep -f "uwsgi.*make_post")'
4. Verify no 502s during a full watch mode auto-play cycle
5. Monitor swap usage — if swap grows past 2.5GB, back off to 384MB limit

Risk

Low. Config-only change, easily reversible with a service restart. The service unit is managed by salt (/home/fox/foxhop-pillar/caddy/makepostsell.sls) so the salt pillar should be updated after validating the new values.

Depends On

Nothing. Can be applied immediately.

Blocks

MPS-5 (memory investigation — the RSS tuning buys time but doesn't fix the underlying memory growth).