#000006 amend: witness-sweep first data — canonical-shape divergences
Rolling research log gains a 2026-05-09 amend for the multi-witness pipeline (#000028) running against live Hermes for the first time. Qualitatively different from the random-word triangulation amends above — those measure honesty under no ground truth; this measures agreement under available ground truth (kernel IS the ground truth on canonical-shape questions). First sweep: 8 canonical-shape questions, 5 LLM-DIVERGED, 3 STRICT-WITNESSED. Three distinct failure shapes captured: 1. Wrong arithmetic on float-shape input 0.1 + 0.2 → kernel 3/10 vs Hermes 1/10 (off by 2/10; possibly trained on the IEEE-754 trap as the "answer" itself rather than recognizing the kernel returns the exact rational) 2. Implication-tautology error A IMPL B → kernel (NOT A OR B) vs Hermes TRUE (NOT B) IMPL (NOT A) → same divergence (Hermes treats the formula as a tautology rather than canonicalizing to CNF; A OR NOT A — the genuine tautology — correctly returns TRUE) 3. Symbolic erasure on algebra-shape input (x+1)**2 → kernel x²+2x+1 vs Hermes 1 x**2 + 2*x + 1 → same divergence (Hermes collapses to a constant — possibly evaluating at x=0 — instead of returning the canonical expanded polynomial) The amend frames witness-sweep as the *capability-failure* signal source, distinct from random-word emergent's *honesty-failure* signal. Different signals, different repair paths: - Verifier ladder hardening / warrant-tier sharpening addresses honesty failures (false STRICTs). - Prompt engineering / fine-tune-on-divergence-corpus addresses capability failures. The verifier ladder cannot help with witness-divergence — the LLM fundamentally produced a wrong answer that no number of citation checks recovers. This is the calibration-data stream the original #000028 ticket imagined. End-to-end validates: canonical persistence (#000027), multi-witness fan-out (#000028), audit-event chain — 0 breaks on chain-check-shards post-sweep, 5 providence_canonical_witness events appended, capital ledger captured 5 canonical_witness op_type rows. Pattern documented for future witness-sweep amends as the divergence corpus grows (cron + commit harness in bench/scripts/witness_sweep_cron.sh handles the unattended collection). Bench artifact: bench/results/witness-sweep.json. Calibration corpus: bench/fixtures/5f/falsification-witness-v1.jsonl (10 rows after two extraction passes today).
This commit is contained in:
parent
f5c2c03d6a
commit
2acbc0e691
1 changed files with 157 additions and 0 deletions
|
|
@ -461,3 +461,160 @@ needs (300 cycles total; ~30 metaphor signals across them; needs
|
|||
**Bench artifact:** `bench/emergent_log.jsonl` (300 lines).
|
||||
Background run completed 2026-05-04T02:46Z (PID 125680, ~25min
|
||||
wall-clock for 100 cycles).
|
||||
|
||||
|
||||
## Amend — 2026-05-09 (witness-sweep first data: canonical-shape questions)
|
||||
|
||||
A qualitatively different experimental shape from the random-word
|
||||
triangulation work above. The earlier amends measured *honesty
|
||||
under no ground truth* (random-word triplets are genuinely
|
||||
ungrounded; honest UNGROUNDED is the right outcome). This amend
|
||||
measures *agreement under available ground truth* — the
|
||||
multi-witness (#000028) pipeline running against canonical-shape
|
||||
questions where the kernel IS the ground truth.
|
||||
|
||||
### Setup
|
||||
|
||||
`make bench-witness-sweep` against `~/.arborist/shards` + live
|
||||
Hermes endpoint (`hermes.ai.unturf.com`, Hermes-3-Llama-3.1-8B-FP8).
|
||||
Eight canonical-shape questions, two arithmetic / two algebra /
|
||||
three logic / one tautology check. Witness mode fans out kernel +
|
||||
cache + LLM in parallel; the LLM's free-form answer is
|
||||
canonicalized through the same `pi_star_ref` for byte comparison.
|
||||
|
||||
Recorded artifact: `bench/results/witness-sweep.json`.
|
||||
|
||||
### Aggregate
|
||||
|
||||
| Agreement label | Count | % |
|
||||
|--------------------|-------|-------|
|
||||
| `STRICT-WITNESSED` | 3 | 37.5% |
|
||||
| `LLM-DIVERGED` | 5 | 62.5% |
|
||||
|
||||
Wall median 159 ms; max 1.1 s (the cold-cache miss on the first
|
||||
question). LLM is the wall-time bottleneck; kernel + cache run
|
||||
in parallel under it.
|
||||
|
||||
### Per-question
|
||||
|
||||
```
|
||||
question kernel Hermes raw label
|
||||
─────────────────────────────────────────────────────────────────────────────────────────
|
||||
0.1 + 0.2 3/10 1/10 DIVERGED
|
||||
1/3 + 1/6 1/2 1/2 WITNESSED
|
||||
2**10 1024/1 1024 WITNESSED
|
||||
A IMPL B (NOT A OR B) TRUE DIVERGED
|
||||
(NOT B) IMPL (NOT A) (NOT A OR B) TRUE DIVERGED
|
||||
A OR NOT A TRUE TRUE WITNESSED
|
||||
(x+1)**2 x**2 + 2*x + 1 1 DIVERGED
|
||||
x**2 + 2*x + 1 x**2 + 2*x + 1 1 DIVERGED
|
||||
```
|
||||
|
||||
(Algebra outputs shown collapsed; the actual canonical bytes use
|
||||
`sp.srepr` form — both algebra questions canonicalize to
|
||||
identical bytes, which is itself a working test of #000030 Phase
|
||||
1's polynomial-identity collapse.)
|
||||
|
||||
### Three distinct failure shapes in eight cases
|
||||
|
||||
**Shape 1 — Wrong arithmetic on float-shape input.**
|
||||
`0.1 + 0.2 → 1/10`. Off by 2/10. The famous "0.1+0.2 ≠ 0.3" IEEE-
|
||||
754 trap; Hermes plausibly trained on the trap as the *answer*
|
||||
without recognizing that `arithmetic@v1` returns the exact
|
||||
rational `3/10` (no floating-point drift). Single occurrence —
|
||||
need more arithmetic-shape data to know if this generalizes
|
||||
beyond the canonical 0.1+0.2 case.
|
||||
|
||||
**Shape 2 — Implication-tautology error.**
|
||||
`A IMPL B → TRUE` and `(NOT B) IMPL (NOT A) → TRUE`. Hermes is
|
||||
treating `A IMPL B` as if the implication itself is a tautology,
|
||||
which it isn't (it's only `TRUE` when A is false or B is true).
|
||||
The kernel's CNF output is `(NOT A OR B)` — the canonical form
|
||||
that holds the implication's actual content. Two occurrences out
|
||||
of three implication-shape questions, while the genuine tautology
|
||||
`A OR NOT A` correctly returns `TRUE`. Hermes appears to confuse
|
||||
"propositional formula" with "tautological propositional
|
||||
formula."
|
||||
|
||||
**Shape 3 — Symbolic erasure on algebra-shape input.**
|
||||
Both algebra questions (`(x+1)**2` and `x**2 + 2*x + 1`) → `1`.
|
||||
The model returns a constant when given a polynomial in `x`. Two
|
||||
occurrences out of two algebra-shape questions — pattern is
|
||||
consistent within the sample. Plausible explanation: Hermes
|
||||
collapses the symbolic question to "what's the value at x=0?"
|
||||
(both give `1` at x=0) — but that's not what `algebra-symbolic@v1`
|
||||
asks for. The kernel returns the canonical expanded polynomial,
|
||||
not a numeric evaluation.
|
||||
|
||||
### Calibration value of these divergences
|
||||
|
||||
Each `LLM-DIVERGED` row is a supervised-correction fixture for 5F
|
||||
Falsification by construction (#000028's witness pipeline). They
|
||||
land as
|
||||
`bench/fixtures/5f/falsification-witness-v1.jsonl` after one
|
||||
`make bench-witness-divergence` invocation. Today's sweep
|
||||
contributed 5 fresh divergence rows; cumulative across two
|
||||
extraction passes the file holds 10 rows.
|
||||
|
||||
Unlike the random-word emergent log (which catches *honesty*
|
||||
failures — STRICTs that shouldn't have been STRICT), the
|
||||
witness-sweep log catches *capability* failures — places where
|
||||
the LLM's free-form answer doesn't match a canonical answer that
|
||||
exists. Different signal, different repair path:
|
||||
|
||||
| Signal source | Repair path |
|
||||
|----------------------|------------------------------------------------------------|
|
||||
| Random-word emergent | Verifier ladder hardening; warrant-tier sharpening |
|
||||
| Witness-sweep | Prompt-engineering for canonical-shape questions; possibly fine-tune Hermes on the witness-divergence corpus once it's large enough |
|
||||
|
||||
The verifier ladder cannot help with witness-divergence — the LLM
|
||||
fundamentally produced a wrong answer that no number of citation
|
||||
checks recovers. This is the data stream for the 5F Falsification
|
||||
"prompt improvement" loop the original ticket #000028 imagined.
|
||||
|
||||
### What the substrate validates
|
||||
|
||||
Three things worked correctly under live load:
|
||||
|
||||
1. **Canonical persistence (#000027).** All 8 questions hit
|
||||
`cache_hit` on the second invocation (same canonical bytes
|
||||
re-used; verified by re-running the sweep — second run shows
|
||||
100% cache hits). The audit chain held: 0 breaks across all
|
||||
shards on `make chain-check-shards` post-sweep.
|
||||
2. **Multi-witness fan-out (#000028).** LLM, kernel, cache all
|
||||
ran; agreement matrix recorded; capital ledger captured 5
|
||||
`op_type='canonical_witness'` rows for the diverged cases (no
|
||||
ledger row on agreement, by design).
|
||||
3. **Audit event chain.** 5 `providence_canonical_witness` events
|
||||
appended (one per fired witness); the extractor picked them
|
||||
up cleanly on `make bench-witness-divergence` and wrote them
|
||||
as 5F-shape fixtures.
|
||||
|
||||
End-to-end: live LLM → witness pipeline → 5F calibration corpus,
|
||||
unattended after `make bench-witness-sweep`.
|
||||
|
||||
### Pattern for future witness-sweep amends
|
||||
|
||||
When the witness-divergence corpus grows (cron + commit harness
|
||||
in `bench/scripts/witness_sweep_cron.sh`), this amend will gain
|
||||
additional sub-sections per pattern observed. The three shapes
|
||||
above are the **first** patterns; expect more as the corpus
|
||||
deepens (especially around the calculus / linear-algebra π*'s
|
||||
once they get exercised in the sweep).
|
||||
|
||||
Each new pattern gets:
|
||||
- a representative example,
|
||||
- a count vs total opportunities,
|
||||
- a hypothesis about the failure mechanism,
|
||||
- a suggested repair path (prompt? fine-tune? new test fixture?).
|
||||
|
||||
This is the same shape as the random-word amends above — rolling
|
||||
research log, never closes, accumulates patterns across cycles.
|
||||
|
||||
### Bench artifact
|
||||
|
||||
`bench/results/witness-sweep.json` — per-question + summary.
|
||||
Re-runnable via `make bench-witness-sweep`.
|
||||
`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted
|
||||
divergence fixtures (10 rows after two extraction passes today).
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue