docs(#000057): capture known-good serving invocations for the salt buildout

The exact llama.cpp (qwen/4090) and vLLM (hermes/3090) launch commands,
recon'd 2026-05-20, so the foxhop-states salt states can be written
accurately rather than guessed. Notes the convention (salt manages the
systemd unit; engine binaries + model artifacts stay manual on
/mnt/data as documented prereqs) and the live-hermes cutover constraint
(keep >=1 hermes online; qwen is expendable).
This commit is contained in:
russell@unturf.com 2026-05-20 15:20:34 -04:00
parent 1506062ab2
commit 69a9642296
No known key found for this signature in database

View file

@ -121,7 +121,37 @@ sysfs perm, to capture CPU watts; GPU watts need no special perm.
To fill the matrix, the missing serving configs must be stood up:
vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU.
That's an infra buildout (install engines, fetch models, manage the
single-slot live endpoints), tracked separately from the harness.
single-slot live endpoints), to be captured as **salt states in
`~/git/foxhop-states`** (where `ai.foxhop.net`'s `llama-qwen` state
already lives; the 3090 box is not yet salt-managed).
**Known-good serving invocations (captured 2026-05-20 — the basis for
the salt states):**
- **qwen / llama.cpp / 4090** (live, salt `llama-qwen/init.sls`):
```
/mnt/data/llm/llama.cpp/build/bin/llama-server \
-m /mnt/data/llm/cache/Qwen3.6-27B-UD-Q4_K_XL.gguf \
--host 0.0.0.0 --port 18888 -ngl 99 -c 65536 -fa on \
--cache-type-k q8_0 --cache-type-v q8_0 -np 1 \
--spec-type draft-mtp --spec-draft-n-max 6 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5
```
- **hermes / vLLM / 3090** (live, MANUAL tmux — not yet salt; vLLM
0.10.1.1 on python3.12):
```
python -m vllm.entrypoints.openai.api_server \
--model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \
--host 0.0.0.0 --port 18888 --max-model-len 82000 \
--enable-auto-tool-choice --tool-call-parser hermes
```
Per the existing convention, salt manages the **systemd unit**; the big
artifacts (engine binaries, model files / gguf) live on `/mnt/data` and
are provisioned manually, documented as state prereqs. Migrating the
live 3090 hermes from its manual tmux session to a salt systemd unit
must keep ≥1 hermes online (qwen is the expendable/exploratory
workload) — a careful cutover, not a blind highstate.
**Why cost matters most here:** reasoning chains are the headline cost
finding — `qwen-think` spends 1300-3300 completion tokens/answer vs