# MPS-4: Eliminate intermittent 502s from uWSGI worker recycling

## Problem

Visitors intermittently see 502 Bad Gateway errors that resolve on refresh.
Root cause: uWSGI workers hit the `--reload-on-rss 256` memory limit every
4-5 minutes under normal watch mode traffic, triggering a kill+respawn cycle.
With only 2 workers (`--processes=2`), when both recycle near-simultaneously
Caddy's `reverse_proxy` gets no available backend and returns 502.

### Evidence (2026-02-11 ~09:15-09:22 UTC)

**Worker memory growth** (from `ps aux`):
- Worker 882747 (spawned 09:18): 242MB RSS after 4 minutes
- Worker 882737 (spawned 09:17): 188MB RSS after 5 minutes
- Both approaching the 256MB kill threshold simultaneously

**Worker recycling frequency** (from `journalctl`):
```
09:15:24 - worker 2 (882687) "Seeya!" → killed → Respawned as 882699
09:15:29 - worker 1 (882640) "Seeya!" → killed → Respawned as 882708
09:15:39 - worker 2 (882699) "Seeya!" → killed → Respawned as 882717
09:17:36 - worker 2 (882717) "Seeya!" → killed → Respawned as 882737
09:18:22 - worker 1 (882708) "Seeya!" → killed → Respawned as 882747
```

Workers survive only ~15 seconds to ~3 minutes under load before hitting
the RSS limit. The 09:15:24 and 09:15:29 kills are only 5 seconds apart —
both workers recycling nearly simultaneously.

**Cold start penalty**: First request after respawn takes 400-600ms
(vs normal 100-130ms) while the app re-initializes:
- 882699 first request: 413ms
- 882717 first request: 409ms
- 882747 first request: 533ms

**System resources**: 4GB total RAM, 1.9GB swap used — memory pressure.

### Current uWSGI config

```
--reload-on-rss 256
--processes=2
--threads 8
--max-requests 10000
--http=127.0.0.1:6001
```

## Solution

Tune uWSGI config to prevent simultaneous worker unavailability:

### 1. Raise RSS limit

Raise `--reload-on-rss` from 256 to 512. Workers currently grow to 242MB
in 4 minutes — 256 is too aggressive and causes constant churn. At 512MB
with 2 workers, worst case is ~1GB for workers, still well within the 4GB
system budget (Caddy + master + crypto_watcher use ~300MB combined).

### 2. Add `--reload-on-rss-stagger`

If available in the installed uWSGI version, or use `--max-requests` with
variance (`--max-requests-delta`) to prevent both workers from recycling at
the same instant. Set `--max-requests-delta 1000` to add randomness
(each worker gets max-requests ± 1000).

### 3. Use lazy-apps mode

Add `--lazy-apps` so each worker loads the application independently after
fork. This costs a bit more memory but means the master doesn't need to
re-fork the full app — workers initialize in parallel and the surviving
worker keeps serving while the new one starts.

### 4. Add `--harakiri` timeout

Add `--harakiri 30` as a safety net — if any request takes >30 seconds
(stuck DB query, deadlock), kill that worker instead of blocking a slot
forever.

### Proposed new config

```
--reload-on-rss 512
--processes=2
--threads 8
--max-requests 10000
--max-requests-delta 2000
--harakiri 30
--die-on-term
--http=127.0.0.1:6001
--lazy-apps
```

## Testing

1. Apply config change on prod (`systemctl edit --full my.makepostsell.com`)
2. `systemctl restart my.makepostsell.com`
3. Monitor with: `watch -n 5 'ps -o pid,rss,vsz,etimes,cmd -p $(pgrep -f "uwsgi.*make_post")'`
4. Verify no 502s during a full watch mode auto-play cycle
5. Monitor swap usage — if swap grows past 2.5GB, back off to 384MB limit

## Risk

Low. Config-only change, easily reversible with a service restart.
The service unit is managed by salt (`/home/fox/foxhop-pillar/caddy/makepostsell.sls`)
so the salt pillar should be updated after validating the new values.

## Depends On

Nothing. Can be applied immediately.

## Blocks

MPS-5 (memory investigation — the RSS tuning buys time but doesn't fix the
underlying memory growth).
