docs(CLAUDE.md): budget discipline — Hermes/Qwen first, Opus deferred

2026-05-19: huge-N #000057 control-arm sweep (f63b00d9dc02e4) burned
our Opus quota. Pin the policy into repo CLAUDE.md so future agents
don't reach for Opus on autopilot:

- Sweep + bench-qa default endpoints: Hermes-3-8B + Qwen via uncloseai
- Opus drops out of control-arm grids until fox explicitly re-adds it
- Existing bench/results/ data stays valid; just no new Opus expansion
- Translation still uses local pinned opus-mt (sandwich MT edges), not
  Hermes/Qwen 8B
- Grok rejoin path noted as an HTML comment (paid spend, fox-gated)

Adds two subsections under '## Live endpoints':
- 'Budget discipline — Hermes / Qwen first, Opus deferred'
- 'Per-call model selection (general agent work, not sweep arms)'
This commit is contained in:
russell@unturf.com 2026-05-19 17:22:28 -04:00
parent 981ba84f4a
commit 2365bd1da4
No known key found for this signature in database

View file

@ -347,6 +347,33 @@ revert without reading why. When in doubt, walk the
- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
`robots.txt` returned 404 → no rules.
### Budget discipline — Hermes / Qwen first, Opus deferred
2026-05-19: the huge-N `#000057` control-arm sweep
(`f63b00d``9dc02e4`) burned our Opus quota. Until fox explicitly
re-adds it, sweep + `bench-qa` LLM calls route to **Hermes-3-8B**
(`hermes.ai.unturf.com/v1`) and **Qwen via uncloseai.com**; Opus drops
out of control-arm grids. This is a budget rule, not a quality claim —
bench results already on disk under `bench/results/` stay valid; just
don't expand them with new Opus tokens without a go. Translation at
scale is still NOT a Hermes/Qwen 8B job (sandwich MT uses local pinned
`opus-mt`, see crosslang rule). Grok candidacy noted from ajax synthetic-
data-distillation benchmarks.
<!-- Grok API rejoins the grid once fox approves credit-card spend.
When unflagged, add a `grok-*` arm to `bench/control_sweep.py`
and document model id + endpoint here. -->
### Per-call model selection (general agent work, not sweep arms)
Within a 4-hour Max-plan window, haiku/sonnet/opus draw equally from
quota — prefer the largest model that fits (fewer retries = better
quota efficiency). During **paid overflow** (beyond Max), pay-per-token
applies: start haiku, escalate sonnet then opus only on failure.
Hermes/Qwen via `uncloseai.com` serve classification & code-adjacent
work, **never human-language translation** (8B quality too low; that's
the sandwich-MT `opus-mt` edge).
## Retrieval pipeline (`arborist/qa/query.py`)
Multi-stage. Each stage exists because something earlier wasn't