From 2acbc0e691ae54b1d222fe500192454794feb6f2 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 9 May 2026 15:57:18 -0400 Subject: [PATCH] =?UTF-8?q?#000006=20amend:=20witness-sweep=20first=20data?= =?UTF-8?q?=20=E2=80=94=20canonical-shape=20divergences?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rolling research log gains a 2026-05-09 amend for the multi-witness pipeline (#000028) running against live Hermes for the first time. Qualitatively different from the random-word triangulation amends above — those measure honesty under no ground truth; this measures agreement under available ground truth (kernel IS the ground truth on canonical-shape questions). First sweep: 8 canonical-shape questions, 5 LLM-DIVERGED, 3 STRICT-WITNESSED. Three distinct failure shapes captured: 1. Wrong arithmetic on float-shape input 0.1 + 0.2 → kernel 3/10 vs Hermes 1/10 (off by 2/10; possibly trained on the IEEE-754 trap as the "answer" itself rather than recognizing the kernel returns the exact rational) 2. Implication-tautology error A IMPL B → kernel (NOT A OR B) vs Hermes TRUE (NOT B) IMPL (NOT A) → same divergence (Hermes treats the formula as a tautology rather than canonicalizing to CNF; A OR NOT A — the genuine tautology — correctly returns TRUE) 3. Symbolic erasure on algebra-shape input (x+1)**2 → kernel x²+2x+1 vs Hermes 1 x**2 + 2*x + 1 → same divergence (Hermes collapses to a constant — possibly evaluating at x=0 — instead of returning the canonical expanded polynomial) The amend frames witness-sweep as the *capability-failure* signal source, distinct from random-word emergent's *honesty-failure* signal. Different signals, different repair paths: - Verifier ladder hardening / warrant-tier sharpening addresses honesty failures (false STRICTs). - Prompt engineering / fine-tune-on-divergence-corpus addresses capability failures. The verifier ladder cannot help with witness-divergence — the LLM fundamentally produced a wrong answer that no number of citation checks recovers. This is the calibration-data stream the original #000028 ticket imagined. End-to-end validates: canonical persistence (#000027), multi-witness fan-out (#000028), audit-event chain — 0 breaks on chain-check-shards post-sweep, 5 providence_canonical_witness events appended, capital ledger captured 5 canonical_witness op_type rows. Pattern documented for future witness-sweep amends as the divergence corpus grows (cron + commit harness in bench/scripts/witness_sweep_cron.sh handles the unattended collection). Bench artifact: bench/results/witness-sweep.json. Calibration corpus: bench/fixtures/5f/falsification-witness-v1.jsonl (10 rows after two extraction passes today). --- .../ticket-000006-bench-emergent-findings.md | 157 ++++++++++++++++++ 1 file changed, 157 insertions(+) diff --git a/docs/tickets/ticket-000006-bench-emergent-findings.md b/docs/tickets/ticket-000006-bench-emergent-findings.md index 2e5f64e..75d4cbf 100644 --- a/docs/tickets/ticket-000006-bench-emergent-findings.md +++ b/docs/tickets/ticket-000006-bench-emergent-findings.md @@ -461,3 +461,160 @@ needs (300 cycles total; ~30 metaphor signals across them; needs **Bench artifact:** `bench/emergent_log.jsonl` (300 lines). Background run completed 2026-05-04T02:46Z (PID 125680, ~25min wall-clock for 100 cycles). + + +## Amend — 2026-05-09 (witness-sweep first data: canonical-shape questions) + +A qualitatively different experimental shape from the random-word +triangulation work above. The earlier amends measured *honesty +under no ground truth* (random-word triplets are genuinely +ungrounded; honest UNGROUNDED is the right outcome). This amend +measures *agreement under available ground truth* — the +multi-witness (#000028) pipeline running against canonical-shape +questions where the kernel IS the ground truth. + +### Setup + +`make bench-witness-sweep` against `~/.arborist/shards` + live +Hermes endpoint (`hermes.ai.unturf.com`, Hermes-3-Llama-3.1-8B-FP8). +Eight canonical-shape questions, two arithmetic / two algebra / +three logic / one tautology check. Witness mode fans out kernel + +cache + LLM in parallel; the LLM's free-form answer is +canonicalized through the same `pi_star_ref` for byte comparison. + +Recorded artifact: `bench/results/witness-sweep.json`. + +### Aggregate + +| Agreement label | Count | % | +|--------------------|-------|-------| +| `STRICT-WITNESSED` | 3 | 37.5% | +| `LLM-DIVERGED` | 5 | 62.5% | + +Wall median 159 ms; max 1.1 s (the cold-cache miss on the first +question). LLM is the wall-time bottleneck; kernel + cache run +in parallel under it. + +### Per-question + +``` +question kernel Hermes raw label +───────────────────────────────────────────────────────────────────────────────────────── +0.1 + 0.2 3/10 1/10 DIVERGED +1/3 + 1/6 1/2 1/2 WITNESSED +2**10 1024/1 1024 WITNESSED +A IMPL B (NOT A OR B) TRUE DIVERGED +(NOT B) IMPL (NOT A) (NOT A OR B) TRUE DIVERGED +A OR NOT A TRUE TRUE WITNESSED +(x+1)**2 x**2 + 2*x + 1 1 DIVERGED +x**2 + 2*x + 1 x**2 + 2*x + 1 1 DIVERGED +``` + +(Algebra outputs shown collapsed; the actual canonical bytes use +`sp.srepr` form — both algebra questions canonicalize to +identical bytes, which is itself a working test of #000030 Phase +1's polynomial-identity collapse.) + +### Three distinct failure shapes in eight cases + +**Shape 1 — Wrong arithmetic on float-shape input.** +`0.1 + 0.2 → 1/10`. Off by 2/10. The famous "0.1+0.2 ≠ 0.3" IEEE- +754 trap; Hermes plausibly trained on the trap as the *answer* +without recognizing that `arithmetic@v1` returns the exact +rational `3/10` (no floating-point drift). Single occurrence — +need more arithmetic-shape data to know if this generalizes +beyond the canonical 0.1+0.2 case. + +**Shape 2 — Implication-tautology error.** +`A IMPL B → TRUE` and `(NOT B) IMPL (NOT A) → TRUE`. Hermes is +treating `A IMPL B` as if the implication itself is a tautology, +which it isn't (it's only `TRUE` when A is false or B is true). +The kernel's CNF output is `(NOT A OR B)` — the canonical form +that holds the implication's actual content. Two occurrences out +of three implication-shape questions, while the genuine tautology +`A OR NOT A` correctly returns `TRUE`. Hermes appears to confuse +"propositional formula" with "tautological propositional +formula." + +**Shape 3 — Symbolic erasure on algebra-shape input.** +Both algebra questions (`(x+1)**2` and `x**2 + 2*x + 1`) → `1`. +The model returns a constant when given a polynomial in `x`. Two +occurrences out of two algebra-shape questions — pattern is +consistent within the sample. Plausible explanation: Hermes +collapses the symbolic question to "what's the value at x=0?" +(both give `1` at x=0) — but that's not what `algebra-symbolic@v1` +asks for. The kernel returns the canonical expanded polynomial, +not a numeric evaluation. + +### Calibration value of these divergences + +Each `LLM-DIVERGED` row is a supervised-correction fixture for 5F +Falsification by construction (#000028's witness pipeline). They +land as +`bench/fixtures/5f/falsification-witness-v1.jsonl` after one +`make bench-witness-divergence` invocation. Today's sweep +contributed 5 fresh divergence rows; cumulative across two +extraction passes the file holds 10 rows. + +Unlike the random-word emergent log (which catches *honesty* +failures — STRICTs that shouldn't have been STRICT), the +witness-sweep log catches *capability* failures — places where +the LLM's free-form answer doesn't match a canonical answer that +exists. Different signal, different repair path: + +| Signal source | Repair path | +|----------------------|------------------------------------------------------------| +| Random-word emergent | Verifier ladder hardening; warrant-tier sharpening | +| Witness-sweep | Prompt-engineering for canonical-shape questions; possibly fine-tune Hermes on the witness-divergence corpus once it's large enough | + +The verifier ladder cannot help with witness-divergence — the LLM +fundamentally produced a wrong answer that no number of citation +checks recovers. This is the data stream for the 5F Falsification +"prompt improvement" loop the original ticket #000028 imagined. + +### What the substrate validates + +Three things worked correctly under live load: + +1. **Canonical persistence (#000027).** All 8 questions hit + `cache_hit` on the second invocation (same canonical bytes + re-used; verified by re-running the sweep — second run shows + 100% cache hits). The audit chain held: 0 breaks across all + shards on `make chain-check-shards` post-sweep. +2. **Multi-witness fan-out (#000028).** LLM, kernel, cache all + ran; agreement matrix recorded; capital ledger captured 5 + `op_type='canonical_witness'` rows for the diverged cases (no + ledger row on agreement, by design). +3. **Audit event chain.** 5 `providence_canonical_witness` events + appended (one per fired witness); the extractor picked them + up cleanly on `make bench-witness-divergence` and wrote them + as 5F-shape fixtures. + +End-to-end: live LLM → witness pipeline → 5F calibration corpus, +unattended after `make bench-witness-sweep`. + +### Pattern for future witness-sweep amends + +When the witness-divergence corpus grows (cron + commit harness +in `bench/scripts/witness_sweep_cron.sh`), this amend will gain +additional sub-sections per pattern observed. The three shapes +above are the **first** patterns; expect more as the corpus +deepens (especially around the calculus / linear-algebra π*'s +once they get exercised in the sweep). + +Each new pattern gets: +- a representative example, +- a count vs total opportunities, +- a hypothesis about the failure mechanism, +- a suggested repair path (prompt? fine-tune? new test fixture?). + +This is the same shape as the random-word amends above — rolling +research log, never closes, accumulates patterns across cycles. + +### Bench artifact + +`bench/results/witness-sweep.json` — per-question + summary. +Re-runnable via `make bench-witness-sweep`. +`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted +divergence fixtures (10 rows after two extraction passes today). +