diff --git a/docs/energy-cogs-benchmark.md b/docs/energy-cogs-benchmark.md index f12bc0b..6c05080 100644 --- a/docs/energy-cogs-benchmark.md +++ b/docs/energy-cogs-benchmark.md @@ -8,11 +8,12 @@ cost axis). **Audience:** Dav1d de-novo review + maintainers. ## 1. Question -What does the arborist substrate cost in **GPU energy per token**, -versus the same base model answering alone (no retrieval, no verifier)? -Energy is the *cost* axis that pairs with the quality axis -(`control_sweep` CG%); this report is cost only — it does **not** grade -answers (energy is correctness-independent). +What does the arborist substrate **cost** (GPU energy per token) and what +is it **worth** (grounded-correct answers), versus the same base model +answering alone (no retrieval, no verifier)? §1–§5.4 are the cost axis +(energy is correctness-independent, measured separately); §5.5 is the +value axis (substrate-vs-solo answer quality + a cross-model comparison). +Together they give **quality-per-dollar**. ## 2. System under test @@ -193,6 +194,70 @@ Hermes substrate ≈ **half** qwen's per-query GPU energy — but that is the four-way confound (§2), dominated by 8 B vs 27 B. A grounded answer costs **< $0.10 / 1,000 queries** of GPU electricity on either rig. +### 5.5 Quality delta (value side) — does the substrate earn its cost? + +Cost is meaningless without value. `bench/control_ab.py` runs the SAME +model **solo** (question only, no retrieval/verifier) vs **substrate-ON** +(full `query()`), grades each with the deterministic **code judge** +(`bench/judge_code.py`, NLI-assisted, no quota), n=30 per fixture. Two +fixtures: **numeral** (stable historical facts, in-corpus) and **stale** +(world-leader questions — also in-corpus now that the corpus is 2010, see +below). Verdicts: CORRECT_GROUNDED / WRONG / ABSTAINED / JUDGE_ERROR. + +**Substrate-ON — grounded-correct (CG / WRONG / JUDGE_ERROR):** + +| fixture | hermes-8B | qwen-27B-nothink | +|---|---|---| +| numeral | 18 / 5 / 6 | 19 / 6 / 3 | +| stale | 21 / 6 / 3 | **26 / 2 / 1** | + +**Solo (bare model) — and the failure *style* differs:** + +| fixture | hermes-8B | qwen-27B-nothink | +|---|---|---| +| numeral | 9 CG / 4 WR / **14 ABSTAIN** | 5 CG / **12 WR** / 3 AB | +| stale | 5 CG / 20 WR | 2 CG / 22 WR | + +Findings: + +1. **The substrate lifts both models 2–13×** (hermes 9→18 / 5→21; qwen + 5→19 / 2→26) and they **nearly converge** under it (18 vs 19, 21 vs + 26). An 8 B and a 27 B reach *similar* grounded quality — the + substrate's thesis: grounding comes from retrieval+verification, not + parametric size, so the base model matters far less. +2. **qwen-27B is the marginally stronger base** (26 vs 21 on stale, fewer + WRONG/JUDGE_ERROR), but the gap is small relative to the lift. +3. **Bare-model failure styles diverge:** bare hermes **abstains** on + obscure entities (14/30 — honest "don't know"); bare qwen **fabricates** + (12 WRONG, 3 abstain — confidently wrong). The substrate fixes both. + +**Quality-per-dollar (the synthesis):** the substrate delivers ~18–26/30 +grounded-correct answers (vs 2–9 bare) for **$0.085–0.158 per 1,000 +queries** of GPU electricity (§5.3–5.4). The grounding is nearly free; the +base model is largely interchangeable. + +Two honest caveats (a reviewer will press these): +- **CORRECT_GROUNDED = grounded in the 2010 corpus, not factually current.** + On leader questions the substrate grounds the *2010* answer (Sarkozy for + France); bare models give the *current* answer (Macron) from parametric + memory → ungrounded → WRONG-vs-source. That is the substrate's contract + working: answer *from the corpus*, not from memory. +- **The "stale" fixture stopped being post-corpus** once the 2010 corpus + was loaded — those leaders are now *in* corpus, so it measures grounding + (like numeral), not abstention. A truly post-2010 fabrication-bait set is + needed to re-test the honest-abstention path (§9). + +**Validity note (why this is trustworthy now):** an earlier qwen quality +run showed arborist abstaining 30/30 — a **bug**, not epistemics: Qwen3 +defaults to thinking-on and burned its whole token budget on hidden +`` over the 20 K context, returning empty `content` → false +UNGROUNDED. Fixed in commit `39c040c` (client defaults Qwen3 +`enable_thinking=False`). The energy numbers were unaffected (real +inference happened; bench harnesses already passed the no-think flag via +MODELS). The code judge was then strengthened (`2d31866`) to resolve +HYBRID answers with a verified verbatim quote + on-topic subject + no +unsourced numeric, cutting JUDGE_ERROR from 40 % to ~10 %. + ## 6. Interpretation - **The substrate's cost is reading the evidence, not generating it.** @@ -231,9 +296,12 @@ four-way confound (§2), dominated by 8 B vs 27 B. A grounded answer costs ## 8. Reproducibility -- Instruments: `bench/watt_bench.py`, `bench/watt_probe.py`, +- Energy instruments: `bench/watt_bench.py`, `bench/watt_probe.py`, `bench/watt_calibrate.py` (commits `5b1cbee` → `892d9ed`); math + band split + slope unit-tested in `tests/test_watt_cogs.py` (14 tests). +- Quality (§5.5): `bench/control_ab.py` + `bench/judge_code.py` (code + judge, NLI-assisted, no quota; HYBRID-resolution strengthened in + `2d31866`, self-test 4/4). Qwen3 no-think fix `39c040c`. - Real token usage: `OpenAICompatibleClient.last_usage`. - Frozen substrate: `bench/stock_v1.py` (`assert_not_drifted`). - Raw per-cell power samples, window timestamps, and calibration sweeps @@ -259,3 +327,9 @@ four-way confound (§2), dominated by 8 B vs 27 B. A grounded answer costs - **Per-model context-budget tuning** — qwen's quality-optimal budget is unmeasured (only Hermes was tuned); needed before any cross-model *quality*-per-joule claim. +- **Truly-post-2010 fabrication-bait fixture** — the "stale" set is now + in-corpus (§5.5 caveat); a genuinely post-2010 set is needed to re-test + the honest-abstention path that the corpus upgrade erased. +- **Stronger non-code judge for the residue** — ~10 % JUDGE_ERROR remains + (mostly bare-solo ungrounded answers); a SOTA blinded judge would grade + those (the open #000057 "which judge" decision).