From 69a96422967aa0ba623fd9d577b95599bad46d93 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Wed, 20 May 2026 15:20:34 -0400 Subject: [PATCH] docs(#000057): capture known-good serving invocations for the salt buildout The exact llama.cpp (qwen/4090) and vLLM (hermes/3090) launch commands, recon'd 2026-05-20, so the foxhop-states salt states can be written accurately rather than guessed. Notes the convention (salt manages the systemd unit; engine binaries + model artifacts stay manual on /mnt/data as documented prereqs) and the live-hermes cutover constraint (keep >=1 hermes online; qwen is expendable). --- docs/benchmark-matrix.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/docs/benchmark-matrix.md b/docs/benchmark-matrix.md index 59280a7..55a27fb 100644 --- a/docs/benchmark-matrix.md +++ b/docs/benchmark-matrix.md @@ -121,7 +121,37 @@ sysfs perm, to capture CPU watts; GPU watts need no special perm. To fill the matrix, the missing serving configs must be stood up: vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU. That's an infra buildout (install engines, fetch models, manage the -single-slot live endpoints), tracked separately from the harness. +single-slot live endpoints), to be captured as **salt states in +`~/git/foxhop-states`** (where `ai.foxhop.net`'s `llama-qwen` state +already lives; the 3090 box is not yet salt-managed). + +**Known-good serving invocations (captured 2026-05-20 — the basis for +the salt states):** + +- **qwen / llama.cpp / 4090** (live, salt `llama-qwen/init.sls`): + ``` + /mnt/data/llm/llama.cpp/build/bin/llama-server \ + -m /mnt/data/llm/cache/Qwen3.6-27B-UD-Q4_K_XL.gguf \ + --host 0.0.0.0 --port 18888 -ngl 99 -c 65536 -fa on \ + --cache-type-k q8_0 --cache-type-v q8_0 -np 1 \ + --spec-type draft-mtp --spec-draft-n-max 6 \ + --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5 + ``` +- **hermes / vLLM / 3090** (live, MANUAL tmux — not yet salt; vLLM + 0.10.1.1 on python3.12): + ``` + python -m vllm.entrypoints.openai.api_server \ + --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \ + --host 0.0.0.0 --port 18888 --max-model-len 82000 \ + --enable-auto-tool-choice --tool-call-parser hermes + ``` + +Per the existing convention, salt manages the **systemd unit**; the big +artifacts (engine binaries, model files / gguf) live on `/mnt/data` and +are provisioned manually, documented as state prereqs. Migrating the +live 3090 hermes from its manual tmux session to a salt systemd unit +must keep ≥1 hermes online (qwen is the expendable/exploratory +workload) — a careful cutover, not a blind highstate. **Why cost matters most here:** reasoning chains are the headline cost finding — `qwen-think` spends 1300-3300 completion tokens/answer vs