russell@unturf.com
2acbc0e691
Rolling research log gains a 2026-05-09 amend for the multi-witness pipeline (#000028) running against live Hermes for the first time. Qualitatively different from the random-word triangulation amends above — those measure honesty under no ground truth; this measures agreement under available ground truth (kernel IS the ground truth on canonical-shape questions). First sweep: 8 canonical-shape questions, 5 LLM-DIVERGED, 3 STRICT-WITNESSED. Three distinct failure shapes captured: 1. Wrong arithmetic on float-shape input 0.1 + 0.2 → kernel 3/10 vs Hermes 1/10 (off by 2/10; possibly trained on the IEEE-754 trap as the "answer" itself rather than recognizing the kernel returns the exact rational) 2. Implication-tautology error A IMPL B → kernel (NOT A OR B) vs Hermes TRUE (NOT B) IMPL (NOT A) → same divergence (Hermes treats the formula as a tautology rather than canonicalizing to CNF; A OR NOT A — the genuine tautology — correctly returns TRUE) 3. Symbolic erasure on algebra-shape input (x+1)**2 → kernel x²+2x+1 vs Hermes 1 x**2 + 2*x + 1 → same divergence (Hermes collapses to a constant — possibly evaluating at x=0 — instead of returning the canonical expanded polynomial) The amend frames witness-sweep as the *capability-failure* signal source, distinct from random-word emergent's *honesty-failure* signal. Different signals, different repair paths: - Verifier ladder hardening / warrant-tier sharpening addresses honesty failures (false STRICTs). - Prompt engineering / fine-tune-on-divergence-corpus addresses capability failures. The verifier ladder cannot help with witness-divergence — the LLM fundamentally produced a wrong answer that no number of citation checks recovers. This is the calibration-data stream the original #000028 ticket imagined. End-to-end validates: canonical persistence (#000027), multi-witness fan-out (#000028), audit-event chain — 0 breaks on chain-check-shards post-sweep, 5 providence_canonical_witness events appended, capital ledger captured 5 canonical_witness op_type rows. Pattern documented for future witness-sweep amends as the divergence corpus grows (cron + commit harness in bench/scripts/witness_sweep_cron.sh handles the unattended collection). Bench artifact: bench/results/witness-sweep.json. Calibration corpus: bench/fixtures/5f/falsification-witness-v1.jsonl (10 rows after two extraction passes today). |
||
|---|---|---|
| .. | ||
| _source | ||
| diagrams | ||
| tickets | ||
| bench-maxing.md | ||
| benchmarks.md | ||
| cti-architecture.md | ||
| mesh.md | ||
| pi-star-composition.md | ||
| qa-modes-bench.md | ||
| seven-point-program.md | ||
| soft-hash-channel-analysis.md | ||
| spec-methodology.md | ||
| TICKETS.md | ||
| tool-action-dag-design.md | ||
| v7w-frontier-catalog.md | ||
| v8-fork-score.md | ||
| zk-frontier-bench.md | ||
| zk-wire-protocol.md | ||