docs(#000057): capture known-good serving invocations for the salt buildout
The exact llama.cpp (qwen/4090) and vLLM (hermes/3090) launch commands, recon'd 2026-05-20, so the foxhop-states salt states can be written accurately rather than guessed. Notes the convention (salt manages the systemd unit; engine binaries + model artifacts stay manual on /mnt/data as documented prereqs) and the live-hermes cutover constraint (keep >=1 hermes online; qwen is expendable).
This commit is contained in:
parent
1506062ab2
commit
69a9642296
1 changed files with 31 additions and 1 deletions
|
|
@ -121,7 +121,37 @@ sysfs perm, to capture CPU watts; GPU watts need no special perm.
|
|||
To fill the matrix, the missing serving configs must be stood up:
|
||||
vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU.
|
||||
That's an infra buildout (install engines, fetch models, manage the
|
||||
single-slot live endpoints), tracked separately from the harness.
|
||||
single-slot live endpoints), to be captured as **salt states in
|
||||
`~/git/foxhop-states`** (where `ai.foxhop.net`'s `llama-qwen` state
|
||||
already lives; the 3090 box is not yet salt-managed).
|
||||
|
||||
**Known-good serving invocations (captured 2026-05-20 — the basis for
|
||||
the salt states):**
|
||||
|
||||
- **qwen / llama.cpp / 4090** (live, salt `llama-qwen/init.sls`):
|
||||
```
|
||||
/mnt/data/llm/llama.cpp/build/bin/llama-server \
|
||||
-m /mnt/data/llm/cache/Qwen3.6-27B-UD-Q4_K_XL.gguf \
|
||||
--host 0.0.0.0 --port 18888 -ngl 99 -c 65536 -fa on \
|
||||
--cache-type-k q8_0 --cache-type-v q8_0 -np 1 \
|
||||
--spec-type draft-mtp --spec-draft-n-max 6 \
|
||||
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5
|
||||
```
|
||||
- **hermes / vLLM / 3090** (live, MANUAL tmux — not yet salt; vLLM
|
||||
0.10.1.1 on python3.12):
|
||||
```
|
||||
python -m vllm.entrypoints.openai.api_server \
|
||||
--model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \
|
||||
--host 0.0.0.0 --port 18888 --max-model-len 82000 \
|
||||
--enable-auto-tool-choice --tool-call-parser hermes
|
||||
```
|
||||
|
||||
Per the existing convention, salt manages the **systemd unit**; the big
|
||||
artifacts (engine binaries, model files / gguf) live on `/mnt/data` and
|
||||
are provisioned manually, documented as state prereqs. Migrating the
|
||||
live 3090 hermes from its manual tmux session to a salt systemd unit
|
||||
must keep ≥1 hermes online (qwen is the expendable/exploratory
|
||||
workload) — a careful cutover, not a blind highstate.
|
||||
|
||||
**Why cost matters most here:** reasoning chains are the headline cost
|
||||
finding — `qwen-think` spends 1300-3300 completion tokens/answer vs
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue