# Ticket #000006 — Bench-emergent findings (first 72 cycles)

**Status:** open · rolling research log (verifier-ladder healthy at 134 cycles; emergent gems still being mined)
**Opened:** 2026-05-02
**Last amend:** 2026-05-12
**Scope:** catalog the failure & success shapes the random-word
triangulation harness surfaced; queue up tunings the curated bench
can't suggest because it doesn't see these question shapes.
**Audience:** anyone tuning the verifier / warrant / deflection
checks after the initial bench-emergent shake-out.
**Hard constraint:** every recommendation here must stay lexical.
NLI / LLM-as-judge stays sidecar territory. The verifier-ladder
discipline (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED
→ UNGROUNDED) is preserved across all proposed changes.

## What ran

`scripts/bench_emergent.py` ran 72 cycles in 4 batches on
2026-05-02 (UTC 17:30-18:00 ish). Each cycle:

1. Three random words from `/usr/share/dict/words` (`^[a-z]{5,12}$`).
2. Hermes @ temp=0.8 wove a creative question paragraph.
3. Arborist answered via `claim_lattice` mode (full pipeline).
4. Append journey to `bench/emergent_log.jsonl`.

Aggregate distribution:

| audit_mode | count | % |
|---|---|---|
| UNGROUNDED  | 54 | 75% |
| HYBRID      | 14 | 19% |
| STRICT      |  4 |  6% |
| **grounded total** | **18** | **25%** |

Violation-kind frequencies (across all 72 cycles, mostly UNGROUNDED):

| kind | count | what it indicates |
|---|---|---|
| `CITATION_MISMATCH` | 57 | per-pair citation-coverage failure (Rule 5 of the seven hard checks). Most common — model cited a chunk that shares few/no content tokens with the claim. |
| `TOO_MANY_EVIDENCE_IDS` | 11 | pointer-id over-citation; trim-and-verify still fires |
| `DEFLECTION_DETECTED` | 11 | subject-anchor missing from answer (the soft-demote landed today). |
| `TITLE_MISMATCH` | 7 | cited source's title shares no content token with the claim (Rule 8). |
| `MANUAL_QUOTE_VIOLATION` | 2 | model used `"` chars in pointer-mode claim text |
| `SOURCE_ROLE_BLOCKED` | 1 | cited a noisy source role |
| `metaphor_deflection` | 1 | the new metaphor sidecar (added by parallel work) — fired once |

## Failure shapes catalogued

The 18 grounded entries split into five categories. Each shows
something the curated bench couldn't surface.

### A. Honest non-relationship grounding (the gold case)

`['brontosaurs', 'Bennett', 'belaboring']` — STRICT 1/1.

> Q: What connection could possibly exist between belaboring a
>    point, the dinosaur brontosaurs, and Bennett, an evidently
>    unrelated surname?
> A: The Sinclair Oil Corporation's use of the brontosaurus
>    dinosaur logo is unrelated to the Bennett surname.
> Cited: Sinclair Oil Corporation

Model identified non-relationship + grounded the partial truth
(Sinclair logo is real). This is the **emergent_synthesis** novelty
class — connects facts to refute the question's implicit premise.
**Don't tune this away.**

### B. Genuine emergent triangulation grounded in real corpus

`['sprayers', 'hearties', 'xxviii']` — STRICT 2/2.
`['Brazil', 'directorates', 'tropic']` — STRICT 1/1.
`['viewing', 'octopus', 'exculpated']` — STRICT 1/1.

Model gracefully ignored unparseable tokens (Roman-numeral
nonsense, archaic words) + grounded answers in real Wikipedia
articles (Water fight, Water supply and sanitation in Brazil,
etc.). **known_truth_grounding** novelty class. STRICT is honest
here.

### C. Lazy-anchor over-citation honestly demoted

`['arguments', 'publicists', 'apparatuses']` — HYBRID **1/11**.

11 pointer-ids cited, only 1 verified. Triple-violation:
`CITATION_MISMATCH` + `DEFLECTION_DETECTED` + `TITLE_MISMATCH`.
Display label: `POINTER-LINKED-PARTIAL · via claim_lattice`. The
verifier honestly reports the over-citation; ladder demoted to
the bottom rung. **Working as designed.**

### D. False-premise affirmation (TITLE_MISMATCH catches)

`['widescreens', 'cashback', 'within']` — HYBRID 1/1, **TITLE_MISMATCH**.

> Q: How do widescreen televisions offer cashback deals within a
>    limited time frame to encourage consumers to purchase?
> A: Widescreen televisions offer cashback deals within a limited
>    time frame to encourage consumers to purchase.
> Cited: Coupon (generic article on coupons)

Model affirmed the question's unverified premise, citing a
generic Coupon article that doesn't anchor "widescreen" or
"cashback" specifically. **TITLE_MISMATCH fires** because
"Coupon" doesn't share any content token with "widescreen
televisions". Display: `POINTER-LINKED-PARTIAL · title
mismatch`. **Working as designed.**

Tuning candidate: a stricter "premise-affirmation" detector
that compares the answer's content tokens to the question's
content tokens — when answer tokens ≈ question tokens minus
generic verbs, flag as **PREMISE_PARROTING** (claim text is the
question's premise restated). Deferred until bench shows this
pattern under EVIDENCE-WARRANTED at a measurable rate; today
TITLE_MISMATCH catches the cases we have.

### E. Total deflection to unrelated trivia (DEFLECTION_DETECTED catches)

`['comeliness', 'fetish', 'investitures']` — HYBRID 2/3, `DEFLECTION_DETECTED`. Cited Rock & Chips (Only Fools and Horses prequel).

`['saleswomen', 'excesses', 'libidinous']` — HYBRID 3/3, `DEFLECTION_DETECTED`. Cited The New Tomorrow + Mordred.

`['Tomas', 'pulls', 'transceivers']` — HYBRID 1/1, no deflection violation
(model latched onto a Tomas-named character; subject anchor "transceivers"
ambiguous). Edge case worth investigating.

DEFLECTION_DETECTED fires on the first two. The third escapes the
subject-anchor heuristic because `_content_tokens_in_order()` may
have placed `transceivers` at a different position. Tuning candidate:
investigate the deflection-sidecar shape detection on Tomas-style
questions where the subject is buried mid-paragraph.

## Suggested tuning passes (ranked)

The other agent picking up these findings should consider:

1. **Tomas-pulls-transceivers case** (E.3) — deflection sidecar
   missed it. Either:
   - Strengthen subject-anchor heuristic (try last 2-3 content
     tokens, not just last 1)
   - Add overlap-ratio threshold (currently subject-anchor takes
     precedence over overlap; maybe combine signals)
   - Document why this case legitimately escapes the check
2. **`xxviii` STRICT acceptance** — Roman numerals ignored
   gracefully is fine; should the bench-emergent harness filter
   them out at word-pick time? Easy to add to `_WORD_RE`. But
   xxviii surfaced an interesting case (model gracefully ignoring
   nonsense), so maybe keep it.
3. **`metaphor_deflection` sidecar** (added in parallel) — only
   fired 1×/72. Either the bench shape rarely produces metaphors,
   or the detection threshold needs tuning. Worth a calibration
   pass.

## Conclusion

The verifier-ladder + soft-demote stack is **doing its job** on
the random-word stress test. 25% grounded rate is healthy
(triplets that genuinely cluster in 2010 Wikipedia). Honest
UNGROUNDED dominates (75%) — that's correct behavior on triplets
the corpus genuinely lacks.

The 4 STRICTs are all defensible (honest grounding or
non-relationship identification). The 14 HYBRIDs surface 4
distinct failure shapes — TITLE_MISMATCH, DEFLECTION_DETECTED,
CITATION_MISMATCH, TOO_MANY_EVIDENCE_IDS — all of which already
demote via the soft-demote ladder.

**The biggest discovery is what the bench DIDN'T surface**: zero
EVIDENCE-WARRANTED false-positives. Every grounded run was either
honestly STRICT or honestly demoted. The verifier ladder catches
what it's supposed to.

## Amend — 2026-05-02 (delta + close)

Re-aggregated `bench/emergent_log.jsonl` after the parallel-agent
work landed (`a1dd330` deflection-tightening, `8fec3a5` metaphor
sidecar, `b87966c` metaphor cue derivation) and after Ticket
#000007 (query-layer hyphen fold). Log now holds 134 cycles —
the original 72 plus 62 post-ticket cycles.

**Combined 134-cycle distribution:**

| audit_mode | count | % | vs original 72 |
|---|---|---|---|
| UNGROUNDED  | 106 | 79.1% | +4.1pp |
| HYBRID      |  22 | 16.4% | -2.6pp |
| STRICT      |   6 |  4.5% | -1.5pp |
| **grounded total** | **28** | **20.9%** | -4.1pp |

Distribution is stable. Verifier-ladder discipline holds: the 10
post-ticket grounded cases (2 STRICT + 8 HYBRID) all carry
appropriate violation labels. No EVIDENCE-WARRANTED false-positives.

**Post-ticket grounded breakdown:**

| triplet | mode | violations |
|---|---|---|
| `reprehending/Jonesboro/Claus` | STRICT | — (honest grounding, Santa Claus mythology) |
| `steer/reply/correcter` | STRICT | — (honest grounding) |
| `chiller/tromps/naughtiest` | HYBRID | DEFLECTION_DETECTED |
| `teazle/ensconced/Jannie` | HYBRID | DEFLECTION_DETECTED |
| `sowers/leakage/earbuds` | HYBRID | CITATION_MISMATCH + TOO_MANY_EVIDENCE_IDS |
| `Pigmy/aortas/spoiler` | HYBRID | DEFLECTION_DETECTED |
| `colonnade/snacked/acutest` | HYBRID | DEFLECTION_DETECTED + TITLE_MISMATCH |
| `posturing/diminutions/ligatures` | HYBRID | CITATION_MISMATCH |
| `trace/Markham/crawfishes` | HYBRID | DEFLECTION_DETECTED + WARRANT_MISSING |
| `underfeed/stoppered/biorhythm` | HYBRID | CITATION_MISMATCH + TITLE_MISMATCH + WARRANT_MISSING |

**Delta violation kinds:** CITATION_MISMATCH 53, TOO_MANY_EVIDENCE_IDS
18, DEFLECTION_DETECTED 5, TITLE_MISMATCH 5, WARRANT_MISSING 3,
UNKNOWN_EVIDENCE_ID 1, MANUAL_QUOTE_VIOLATION 1. WARRANT_MISSING
is new vs the original snapshot — confirms the warrant-strengthening
work is catching inferential-claim cases the ladder previously
admitted.

**Tuning candidate resolution:**

1. **Tomas-pulls-transceivers (deflection sidecar miss)** —
   **resolved**. Commit `a1dd330` ("qa: tighten TITLE_MISMATCH +
   DEFLECTION demotes from emergent-log review") landed at 17:33
   UTC 2026-05-02, ~20 minutes BEFORE this ticket was opened. The
   deflection check on the Tomas log entry now reports
   `DEFLECTION_DETECTED` correctly. Ticket text was authored from
   a stale snapshot. Post-ticket delta confirms the check fires
   on every applicable shape (4 of 5 deflection-shaped HYBRID
   cases). No code change needed.

2. **`xxviii` STRICT acceptance** — **kept as designed**. No new
   evidence to filter Roman-numeral / archaic-word triplets at
   word-pick time. Graceful ignoring is a feature, not a bug.

3. **`metaphor_deflection` sidecar calibration** — **deferred**.
   Fired 0× across the 43 post-ticket cycles. Sample too small
   for calibration; revisit when corpus-stress accumulates more
   samples. The sidecar shipped (`8fec3a5`, `b87966c`) and stays
   live; no calibration knob to turn yet.

**Headline finding (unchanged):** zero EVIDENCE-WARRANTED
false-positives across all 134 cycles. The four-rung ladder
catches what it's supposed to.

**Ticket stays open** — emergent stress-testing is ongoing
research, not a one-shot defect. The amend section is the rolling
delta log: append a new `## Amend — YYYY-MM-DD` section after each
substantive bench-emergent re-run with the new distribution +
findings. Code-level tunings that emerge from this research open
their own tickets and link back here.

## Amend — 2026-05-02b (200 cycles, suspect EVIDENCE-WARRANTED gem)

Re-aggregated at 200 cycles total. Distribution stays stable:

| audit_mode | count | % | vs 134-cycle |
|---|---|---|---|
| UNGROUNDED  | 159 | 79.5% | +0.4pp |
| HYBRID      |  35 | 17.5% | +1.1pp |
| STRICT      |   6 |  3.0% | -1.5pp |

**STRICT cases (all six, full set):**

| triplet | source cited | shape |
|---|---|---|
| `sprayers/hearties/xxviii` | Water fight | known-truth grounding (Roman numeral ignored) |
| `viewing/octopus/exculpated` | Water-supply article | known-truth grounding |
| `Brazil/directorates/tropic` | Water supply and sanitation in Brazil | known-truth grounding |
| `brontosaurs/Bennett/belaboring` | Sinclair Oil Corporation | non-relationship grounding (gold case) |
| `reprehending/Jonesboro/Claus` | The Flight of the Phoenix (1965 film) | non-relationship grounding (Humphrey-not-elf) |
| `steer/reply/correcter` | Glossary of language teaching terms and ideas | **suspect — see below** |

Five of six STRICTs are defensible. The sixth (`steer/reply/correcter`)
is the first plausible **EVIDENCE-WARRANTED false-positive in 200
cycles** — worth deeper scrutiny.

### Suspect case: `steer/reply/correcter`

> Q: How might a correcter be used to steer a reply in a conversation,
>    and what aspects of language or communication do these three terms
>    encompass in their relationships with each other?
> A: A correcter can be used to steer a reply in a conversation by
>    identifying and correcting errors in grammar, spelling, or
>    punctuation in the original message. This helps ensure that the
>    reply is accurate and clear. The three terms — correcter, grammar,
>    and communication — encompass various aspects of language in
>    their relationships with each other. Grammar refers to the set
>    of structural rules that govern the composition of sentences,
>    phrases, and words in a language. Communication involves the
>    exchange of information, ideas, thoughts, and feelings between
>    individuals through a common system of symbols, such as language.
>    A correcter plays a role in maintaining the accuracy of language
>    in communication by identifying and correcting errors in grammar,
>    spelling, or punctuation, which can help prevent misunderstandings
>    or confusion in the conversation.
> Cited: E1 (Glossary of language teaching terms and ideas).

This is **premise-affirmation + confabulated definition**. The cited
spotlight excerpt is a Sociolinguistics entry about "exchange of
meaning, relationships, mode of discourse, channel" — generic
language-discourse vocabulary. The model wove a definition-style
answer using overlapping vocabulary that the verifier's per-claim
citation-coverage check passed.

`arborist inspect --cache-key 08dbd2c1...` confirms STRICT 1/1
verified, no unverified spans, no violations. The verifier
genuinely accepted the claim.

**Discriminator run (2026-05-02):** decompressed all 10 chunks of
the cited document (`6edf7202… Glossary of language teaching terms
and ideas`, ~33,565 chars total) and grep'd for the claim's
distinctive subject tokens:

```
chunk 0..9: correcter=False steer=False reply=False
```

**Confirmed false positive.** The words `correcter`, `steer`, and
`reply` appear NOWHERE in the cited document. Yet the verifier
accepted STRICT 1/1 with no violations on the claim "A correcter
can be used to steer a reply in a conversation by identifying and
correcting errors in grammar, spelling, or punctuation…"

### Mechanism

Per-claim citation-coverage (Rule 5) requires ≥30% of the claim's
content tokens to appear in the cited span. The claim contains
~50 content tokens, of which ~30+ are GENERIC linguistic vocabulary
(language, communication, grammar, exchange, relationships,
meaning, discourse, aspects, terms…) — all of which appear in any
linguistics article including this glossary. Coverage check passes
on the generic vocabulary; the QUESTION-DISTINCTIVE tokens
(`correcter`, `steer`, `reply`) ride along unverified.

Same defect class as the pre-#000007 bi-polar bug, but turned
inward: token coverage is symmetric over the claim, but the SUBJECT
of the claim (the unusual tokens) is unverified. The model
premise-parroted the question and clothed it in plausible
definitional vocabulary that incidentally token-overlapped the
cited chunk.

### Headline correction

200-cycle headline updates from "zero EVIDENCE-WARRANTED
false-positives" to **"one confirmed EVIDENCE-WARRANTED
false-positive (0.5%) — `steer/reply/correcter`."** The four-rung
ladder still works on every other case in the sample; this is one
specific shape — PREMISE_PARROTING / GENERIC-VOCAB-RIDE-ALONG —
that the lexical Rule-5 check cannot catch alone.

### Action items emerging from 200-cycle delta

1. **Rule 9 — SUBJECT_TOKENS_ABSENT detector — landed inline (no
   separate ticket).** New per-claim check in
   `arborist/qa/verify.py`. For each resolving claim, compute the
   set of content tokens shared by question AND claim
   (`_parroted_subject_tokens_absent`). For each, check substring
   presence in the union of cited evidence spans (lower-cased,
   same as Rule 5). When ≥ `subject_tokens_absent_threshold` (default
   3) parroted tokens are absent → emit `SUBJECT_TOKENS_ABSENT`
   violation, demote STRICT → HYBRID. Stays lexical, stays binary,
   folds into `verifier_policy_hash` + `governance_policy_hash`.

   **Live verification on the original case** (cache-split via
   policy-hash bump):

   | cache_key | policy | audit_mode |
   |---|---|---|
   | `08dbd2c1…` | pre-fix | STRICT (false positive) |
   | `6a519636…` | post-fix | UNGROUNDED |

   Ladder now correctly demotes the parroted answer. Three new
   unit tests pin the signature: demote on threshold-meeting
   parrot, no-op when question is None, no-demote below threshold.

2. **`metaphor_deflection` sidecar** still rare (low single-digit
   fires across 200 cycles). Calibration deferred until 300+.

Other tuning candidates from the original ticket remain resolved
(Tomas) or kept-as-designed (xxviii).

## Status

Open · rolling research log. Three original tuning candidates:
(1) Tomas-pulls-transceivers — resolved by `a1dd330` (deflection
sidecar tightened); (2) `xxviii` STRICT acceptance — kept as
designed; (3) metaphor sidecar calibration — deferred until
sample size grows. Re-amend as findings accumulate.

## Cross-reference: ticket #000008 bench cycles (2026-05-03)

Ticket #000008 (broad-quantifier preflight guard) landed Phases
0–4 on 2026-05-03 and now runs its own bench A/B cycles per its
§12 with the `--policy KEY=VALUE` harness flag (commit `002f84c`).
Findings relevant to the rolling-emergent stream:

- The `bench/qa_questions.txt` 73-question set carries 7 broad
  questions (~10%) classified ALL/COMPREHENSIVE/OPEN_REQUEST by
  the §10 classifier. Two bounded-universal fixtures added 2026-
  05-03 (`name all members of the beatles`, `list all planets in
  the solar system`).
- Classifier defect caught + fixed in `d24291b`: `how many X?`
  was mis-classifying as MANY (4 false positives across the
  bench). Fixed via leading-anchor count-question short-circuit.
- `metaphor_deflection` calibration (item 3 above) can leverage
  #000008's larger bench cycles — the broad-subset A/B at n=3 ×
  3 modes accumulates 81 rows per cell, useful sample size growth
  for metaphor-sidecar tuning if the metaphor cue happens to fire
  on broad-quantifier shapes.

## Amend — 2026-05-04 (300-cycle update + post-substrate-hardening)

Bench-emergent stress test ran another 100 cycles in background
(`scripts/bench_emergent.py --n 100`) under the post-#000008/9/10/11
substrate. `bench/emergent_log.jsonl` now holds **300 total cycles**.

**Headline shift on the last 100 cycles** (vs the 134-cycle
baseline distribution):

| Verdict     | 134-cycle baseline | Last 100 (post-hardening) | Δ        |
|-------------|--------------------|----------------------------|----------|
| STRICT      | 5% (7/134)         | **0% (0/100)**             | -5pp     |
| HYBRID      | 22% (29/134)       | 16% (16/100)               | -6pp     |
| UNGROUNDED  | 73% (98/134)       | **84% (84/100)**           | +11pp    |

**Zero false-positive STRICTs across 100 random-word triplets.**
The substrate is now dramatically more honest than at the 134-
cycle baseline. Every cycle goes through `claim_lattice` JSON mode
(the bench-emergent harness pins this).

**Violation profile on last 100 cycles** (claim_lattice JSON):

| Kind                   | Count | Comment                                         |
|------------------------|-------|-------------------------------------------------|
| CITATION_MISMATCH      | 86    | dominant gate; per-claim coverage check         |
| TOO_MANY_EVIDENCE_IDS  | 24    | model cites >2 ids per claim                    |
| SUBJECT_TOKENS_ABSENT  | 12    | Rule 9 firing on premise-parroting              |
| DEFLECTION_DETECTED    | 12    | answer's subject doesn't match question's       |
| TITLE_MISMATCH         | 10    | cited evidence's source title shares zero stems |
| MANUAL_QUOTE_VIOLATION | 4     | model typed a quote span (CTI invariant break)  |
| TOO_MANY_CLAIMS        | 2     | claim count exceeds cap (12 default)            |
| UNKNOWN_EVIDENCE_ID    | 2     | model invented an evidence_id                   |
| WARRANT_MISSING        | 1     | anchor-class warrant didn't fire                |

`metaphor_deflection` signal fires on 6/100 cycles — still rare
(item 3 from the original tuning candidates). Sample size 100 is
borderline for calibration; another 100-200 cycles would let us
decide whether to tighten or relax the threshold.

**Interpretation.** The 5pp drop in STRICT-rate isn't a regression
— it's the substrate's verifier ladder + new preflight contracts
(#000008/9/10/11) doing their job. Random-word triplets are
genuinely ungrounded for the most part; the prior 5% STRICT rate
included false-positives that the post-hardening verifier now
catches. The bench-emergent harness is a stress test designed to
surface failure shapes; honest UNGROUNDED on inputs the corpus
can't ground is the correct outcome.

**No new tuning candidates** surface from this amend. The original
three (Tomas-pulls-transceivers, xxviii Roman numerals, metaphor
calibration) remain at their respective resolution states. Item 3
(metaphor calibration) is now closer to having the sample size it
needs (300 cycles total; ~30 metaphor signals across them; needs
~50-100 to calibrate).

**Bench artifact:** `bench/emergent_log.jsonl` (300 lines).
Background run completed 2026-05-04T02:46Z (PID 125680, ~25min
wall-clock for 100 cycles).


## Amend — 2026-05-09 (witness-sweep first data: canonical-shape questions)

A qualitatively different experimental shape from the random-word
triangulation work above. The earlier amends measured *honesty
under no ground truth* (random-word triplets are genuinely
ungrounded; honest UNGROUNDED is the right outcome). This amend
measures *agreement under available ground truth* — the
multi-witness (#000028) pipeline running against canonical-shape
questions where the kernel IS the ground truth.

### Setup

`make bench-witness-sweep` against `~/.arborist/shards` + live
Hermes endpoint (`hermes.ai.unturf.com`, Hermes-3-Llama-3.1-8B-FP8).
Eight canonical-shape questions, two arithmetic / two algebra /
three logic / one tautology check. Witness mode fans out kernel +
cache + LLM in parallel; the LLM's free-form answer is
canonicalized through the same `pi_star_ref` for byte comparison.

Recorded artifact: `bench/results/witness-sweep.json`.

### Aggregate

| Agreement label    | Count | %     |
|--------------------|-------|-------|
| `STRICT-WITNESSED` | 3     | 37.5% |
| `LLM-DIVERGED`     | 5     | 62.5% |

Wall median 159 ms; max 1.1 s (the cold-cache miss on the first
question). LLM is the wall-time bottleneck; kernel + cache run
in parallel under it.

### Per-question

```
question                        kernel                        Hermes raw    label
─────────────────────────────────────────────────────────────────────────────────────────
0.1 + 0.2                       3/10                          1/10           DIVERGED
1/3 + 1/6                       1/2                           1/2            WITNESSED
2**10                           1024/1                        1024           WITNESSED
A IMPL B                        (NOT A OR B)                  TRUE           DIVERGED
(NOT B) IMPL (NOT A)            (NOT A OR B)                  TRUE           DIVERGED
A OR NOT A                      TRUE                          TRUE           WITNESSED
(x+1)**2                        x**2 + 2*x + 1                1              DIVERGED
x**2 + 2*x + 1                  x**2 + 2*x + 1                1              DIVERGED
```

(Algebra outputs shown collapsed; the actual canonical bytes use
`sp.srepr` form — both algebra questions canonicalize to
identical bytes, which is itself a working test of #000030 Phase
1's polynomial-identity collapse.)

### Three distinct failure shapes in eight cases

**Shape 1 — Wrong arithmetic on float-shape input.**
`0.1 + 0.2 → 1/10`. Off by 2/10. The famous "0.1+0.2 ≠ 0.3" IEEE-
754 trap; Hermes plausibly trained on the trap as the *answer*
without recognizing that `arithmetic@v1` returns the exact
rational `3/10` (no floating-point drift). Single occurrence —
need more arithmetic-shape data to know if this generalizes
beyond the canonical 0.1+0.2 case.

**Shape 2 — Implication-tautology error.**
`A IMPL B → TRUE` and `(NOT B) IMPL (NOT A) → TRUE`. Hermes is
treating `A IMPL B` as if the implication itself is a tautology,
which it isn't (it's only `TRUE` when A is false or B is true).
The kernel's CNF output is `(NOT A OR B)` — the canonical form
that holds the implication's actual content. Two occurrences out
of three implication-shape questions, while the genuine tautology
`A OR NOT A` correctly returns `TRUE`. Hermes appears to confuse
"propositional formula" with "tautological propositional
formula."

**Shape 3 — Symbolic erasure on algebra-shape input.**
Both algebra questions (`(x+1)**2` and `x**2 + 2*x + 1`) → `1`.
The model returns a constant when given a polynomial in `x`. Two
occurrences out of two algebra-shape questions — pattern is
consistent within the sample. Plausible explanation: Hermes
collapses the symbolic question to "what's the value at x=0?"
(both give `1` at x=0) — but that's not what `algebra-symbolic@v1`
asks for. The kernel returns the canonical expanded polynomial,
not a numeric evaluation.

### Calibration value of these divergences

Each `LLM-DIVERGED` row is a supervised-correction fixture for 5F
Falsification by construction (#000028's witness pipeline). They
land as
`bench/fixtures/5f/falsification-witness-v1.jsonl` after one
`make bench-witness-divergence` invocation. Today's sweep
contributed 5 fresh divergence rows; cumulative across two
extraction passes the file holds 10 rows.

Unlike the random-word emergent log (which catches *honesty*
failures — STRICTs that shouldn't have been STRICT), the
witness-sweep log catches *capability* failures — places where
the LLM's free-form answer doesn't match a canonical answer that
exists. Different signal, different repair path:

| Signal source        | Repair path                                                |
|----------------------|------------------------------------------------------------|
| Random-word emergent | Verifier ladder hardening; warrant-tier sharpening         |
| Witness-sweep        | Prompt-engineering for canonical-shape questions; possibly fine-tune Hermes on the witness-divergence corpus once it's large enough |

The verifier ladder cannot help with witness-divergence — the LLM
fundamentally produced a wrong answer that no number of citation
checks recovers. This is the data stream for the 5F Falsification
"prompt improvement" loop the original ticket #000028 imagined.

### What the substrate validates

Three things worked correctly under live load:

1. **Canonical persistence (#000027).** All 8 questions hit
   `cache_hit` on the second invocation (same canonical bytes
   re-used; verified by re-running the sweep — second run shows
   100% cache hits). The audit chain held: 0 breaks across all
   shards on `make chain-check-shards` post-sweep.
2. **Multi-witness fan-out (#000028).** LLM, kernel, cache all
   ran; agreement matrix recorded; capital ledger captured 5
   `op_type='canonical_witness'` rows for the diverged cases (no
   ledger row on agreement, by design).
3. **Audit event chain.** 5 `providence_canonical_witness` events
   appended (one per fired witness); the extractor picked them
   up cleanly on `make bench-witness-divergence` and wrote them
   as 5F-shape fixtures.

End-to-end: live LLM → witness pipeline → 5F calibration corpus,
unattended after `make bench-witness-sweep`.

### Pattern for future witness-sweep amends

When the witness-divergence corpus grows (cron + commit harness
in `bench/scripts/witness_sweep_cron.sh`), this amend will gain
additional sub-sections per pattern observed. The three shapes
above are the **first** patterns; expect more as the corpus
deepens (especially around the calculus / linear-algebra π*'s
once they get exercised in the sweep).

Each new pattern gets:
- a representative example,
- a count vs total opportunities,
- a hypothesis about the failure mechanism,
- a suggested repair path (prompt? fine-tune? new test fixture?).

This is the same shape as the random-word amends above — rolling
research log, never closes, accumulates patterns across cycles.

### Bench artifact

`bench/results/witness-sweep.json` — per-question + summary.
Re-runnable via `make bench-witness-sweep`.
`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted
divergence fixtures (10 rows after two extraction passes today).


## Amend — 2026-05-10 (Phase 3 warrant-chain dormancy: structurally rescue-only)

A third qualitatively different experimental shape, parallel to
the 2026-05-09 witness-sweep widening. The earlier amends measured
honesty under no ground truth (random-word) and capability under
canonical ground truth (witness-sweep). This amend measures
**warrant-chain rescue activation under chain-backed ground truth**
— the #000031 Phase 3 verifier (`a57b194`) running against fixtures
explicitly targeting the 92 claim-pack-warrant-resolved records.

### Setup

#000031 Phase 3 (warrant-chain-aware verifier) landed 2026-05-10.
Mechanism: when the LLM cites a chunk that has chain-root warrant
backing AND the lexical `warrant_check` would otherwise emit
`WARRANT_MISSING`, Phase 3 suppresses that violation by promoting
via the chain.

Three sequential fixture iterations probed whether the mechanism
fires empirically on the live corpus:

1. **Live validation** (`1d2a558`) — 7-question parallel-shift
   probe targeting chain-root documents.
2. **Aggressive A/B** (`c5bc53f`) — 13 questions using exact
   claim-pack title phrasing; n=3, 117 cells per condition.
3. **Paraphrase investigation** (`a9fcb42`) — 5 fixture iterations
   each probed-then-benched, varying question shape (definitional,
   how-use, plain-English, who-proposed, who-first-stated).

### Aggregate

| Investigation | Chain-root retrieval | Phase 3 fires | STRICT-rate Δ |
|---|---|---|---|
| Live validation | 6/7 → EVIDENCE-WARRANTED | n/a (smoke) | n/a (no A/B) |
| Aggressive A/B | 12/13 questions hit chain roots | **0 / 117 cells** | +0 (85/117 → 85/117) |
| Paraphrase v1-v3' | 12/13 → 1/8 (varies) | **0** across all iterations | not measured |

`cache_key parity 39/39` on aggressive A/B — Phase 3 is
mechanically observable but produces zero fires.

### Root cause — `arborist/qa/warrant.py:507`

`warrant_check` **vacuous-passes** (returns `(True, [])` without
anchor extraction) unless the question shape is one of:

```text
relation       — proper-noun anchors via is_relation_question
date           — 4-digit year in claim
entity-list    — multi-entity question shape
count          — count-shape question
why-cause      — cause anchors gated on why-shape question
```

Definitional questions ("what is X", "define X") don't have
warrant shape → no `WARRANT_MISSING` to suppress → Phase 3 has
nothing to do.

### Structural reason for the dormancy

Claim-pack chunks (the 92 chain-root documents) are deliberately
**terse** — they state axioms / theorems precisely with
mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions ("who" / "when"
/ "why") retrieve from Wikipedia (which has matching prose
context) instead of from claim-pack chunks.

The data condition Phase 3 needs — chain-backed chunk cited for a
warrant-shape claim — is **empirically unreachable through normal
retrieval pathways on this corpus shape**.

### Verdict

Phase 3's runtime fire rate is structurally bounded near zero on
the current corpus. **Mechanism is correct, non-regressing,
unit-tested** (`tests/test_warrant_chain.py`).

Two readings, both true:

1. **The lexical layer is strong enough** on clean axiom questions
   that the chain-fallback isn't needed. The mechanism that lands
   warrants on cheap lexical evidence does most of the work.
2. **Phase 3 is a latent capability.** It will activate when corpus
   shape shifts — claim-pack content gaining prose context, or the
   corpus mix changing toward warrant-shape questions. The audit-
   line tail `· warrant proven via chain ×N` will surface fires
   when they occur.

### Recommendation (per the investigation's own conclusion)

Don't engineer Phase 3 fires via fixture design. Track fire rate
as a **corpus-evolution signal**. If claim-pack content gains
prose context or the corpus mix changes, Phase 3 will start firing
organically and the audit-line tail will surface it.

This closes the three-thread investigation cleanly. The fixtures
+ reports remain as the empirical trail for future shifts.

### Adjacent landings (same-day infrastructure)

Two cryptographic-primitive Phase 1 deliverables landed
2026-05-10 ahead of v7 plastic-training deployment:

- **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at
  `bench/scripts/phi_alignment_probe.py` + <!--AUTOCOUNT:tests:tests/test_phi_alignment_probe.py-->23<!--/AUTOCOUNT--> tests (was 14;
  +9 from `a4b3056` 2026-05-10 pattern backfill applying the
  monotonicity / hand-formula / closure / parametrized-invalid
  patterns from `docs/calculator-test-patterns.md`) +
  <!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl-->30<!--/AUTOCOUNT-->-vector KAT fixture (10 aligned + 10 uniform + 10
  anti-aligned synthetic checkpoints, deterministic-seeded) at
  `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`.
- **#000035 Phase 1** (earlier today): φ_PRG reference impl at
  `arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode
  KDF) + <!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->31<!--/AUTOCOUNT--> tests (was 20; +7 from `de997f7` 2026-05-10 pattern
  backfill — prefix-extension closure invariant, hand-computed
  HMAC-SHA-512 first-block formula, parametrized invalid-input
  cones) + <!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT-->-vector KAT fixture at
  `bench/fixtures/phi-prg/known-answer-tests.jsonl`.

Both are KAT-pinned + bench-pinned regression artifacts that
will surface signal the moment a real v7 checkpoint gives the
probes something to measure. Soft-hash analysis §9.1 + §9.2
were updated to point at these landings (`28b57da`, `0b038f1`).

### Distinct signal vs prior amends

| Source | Catches | Repair path |
|---|---|---|
| Random-word emergent | honesty failures (false-STRICTs) | verifier ladder hardening |
| Witness-sweep | capability failures (LLM gives wrong answer) | prompt eng or fine-tune |
| Phase 3 warrant-chain | rescue-mechanism activation conditions | corpus shape evolution (NOT fixture engineering) |

### Bench artifacts

- `bench/results/phase3-live-validation-2026-05-10.md`
- `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` (initial)
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md`
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`

Re-runnable via the bench-emergent harness when corpus shape
shifts and a measurable fire rate is expected.


## Amend — 2026-05-10b (Prometheus-Σ Phase 3 dry-run: scheduler calibration findings)

A **fourth qualitatively different experimental shape**:

- random-word (honesty under no ground truth)
- witness-sweep (capability under canonical ground truth)
- warrant-chain (rescue activation under chain-backed ground truth)
- **Prometheus-Σ dry-run** ← this amend (controller scoring under real shard state, no LLM/mutation)

### Setup

#000037 Phase 1 (`arborist/substrate/prometheus.py`) + Phase 2
(`controller_events` sibling audit table + advisory writes) landed
2026-05-10. Phase 3 (sleep-sweep scheduler) is deliberately not
landed — instead a read-only **dry-run simulator** at
`bench/scripts/prometheus_sigma_sweep_dryrun.py` (`make
prometheus-sweep-dryrun`) drives the Phase 1 controller against
real shard data, no LLM calls, no mutations. Catches
**scheduler-calibration failures** — controller verdicts that
look correct in unit tests but fail under real-corpus statistics.

### Aggregate

```text
4 379 branches scored, 55 ms wall, 12.5 µs/branch pure Python
```

Controller-latency is non-binding for Phase 3; the witness fan-out
(Hermes calls) will be the dominant cost when it ships.

### Five scheduler-calibration findings

**F1 — Chunk-size dominates Kelly threshold.** First iteration
chunked candidates 64-at-a-time → **100 % DEFERRED**. Kelly's
`f_i = max(0, (p_i·b − q_i)/b)` requires `p_i > 0.5`; softmax over
64 branches never gives any single branch that much mass.
**Phase 3 fix:** chunk size = Hermes concurrency (4 today;
governance parameter going forward), not the candidate pool size.

**F2 — Flat `capital_cost` blocks every allocation.** Iteration 1
assigned `capital_cost=1.0` to every branch; combined with small
audit-mode-based Δ5F deltas (±0.05–0.10), every utility came out
negative — DEFERRED or REJECT for every chunk. Iteration 2 split
the cost class:

```text
Target A (providence_cache):
    CANONICAL_PROJECTION=0.05   UNGROUNDED=0.4
    HYBRID=0.8                  STRICT=1.0

Target B (documents):
    HEAD-only freshness probe=0.02
    canonical-shape witness=0.05
```

Surfaced real ACCEPT/MARGINAL signal immediately. **Phase 3 fix:**
either split `capital_cost` into `kernel_cost` + `llm_cost` in a
v2 dataclass, or compute `capital_cost` as the expected cost given
which witness paths actually fire.

**F3 — τ_qa = 7d filters out every `CANONICAL_PROJECTION` row.**
All 29 CP rows in `qa.db` are ≤ 7 days old — they're the recent π*
graduations from #000027 / #000030 / #000032. At τ_qa = 7d, zero
CP rows surface → zero ACCEPT chunks → zero high-value sleep work
discovered. At τ_qa = 1d, 9 CP rows surface and 4 chunks return
MARGINAL. **Phase 3 fix:** split τ_qa by audit_mode. Kernel-only
modes (CP) take a short τ (1d default — cheap to re-probe);
LLM-witness modes (STRICT/HYBRID/UNGROUNDED) take a longer τ
(7d default — re-witness is expensive).

**F4 — Target B canonical-shape detection is the real headline.**
4.40 % of sampled documents (n=2000) contain math/logic/time-
series canonical-shape statements. Extrapolated to **~152 609
canonical-probe candidates** across 3 468 392 docs in
`~/.arborist/shards`. Controller correctly returns MARGINAL on 83
of 500 Target B chunks (the regex-prefilter hits). **Phase 3 fix:**
the MARGINAL queue from Target B becomes the funnel for 5F
falsification-fixture mining (§3 "Divergence → candidate
falsification fixture"). At Hermes concurrency = 4 a full Target B
sweep is still ~38K rounds even after the prefilter — Phase 3
needs a sustained-throughput floor + per-window cap, not a
one-shot burst design.

**F5 — Quarantined rows correctly veto.** The two
`falsification_state='quarantined'` rows in `qa.db` are flagged as
`cache_drift` hard-vetoes by Target A's branch synthesis. The
veto path exercises end-to-end against real corpus data — no
fixture-only mocking.

### Updated distinct-signal table

| Source | Catches | Repair path |
|---|---|---|
| Random-word emergent | honesty failures (false-STRICTs) | verifier ladder hardening |
| Witness-sweep | capability failures (LLM gives wrong answer) | prompt eng or fine-tune |
| Phase 3 warrant-chain | rescue-mechanism activation conditions | corpus shape evolution (NOT fixture engineering) |
| Prometheus-Σ dry-run | **scheduler-calibration failures** | tune chunk size, cost weights, τ thresholds *before* Phase 3 scheduler ships |

### What the dry-run validates end-to-end

1. **Phase 1 controller pure function.** `score_controller_state()`
   returns a defensible ControllerDecision on every branch type
   the live corpus contains.
2. **Phase 2 advisory writes.** `controller_events` sibling table
   accepts emitted decisions; audit chain not affected.
3. **Five hard-veto classes fire correctly.** Cache-drift veto
   (F5), DEFERRED on budget-zero (F1), REJECT on negative utility
   (F2), MARGINAL queue for sleep (F4), ACCEPT for high-value
   short-τ work (F3).

### Bench artifact

`bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md` —
detailed per-shard numbers + iteration log. Regenerated by
`make prometheus-sweep-dryrun`.

Phase 3 scheduler (when it ships) inherits F1-F5 as known-good
defaults; the dry-run is the calibration substrate Phase 3 will
reference for choice justification.


---

## Amend — 2026-05-12 (+30 emergent cycles)

`make bench-emergent EMERGENT_N=30` (seed=None). 30 fresh
3-word-triangulation cycles appended to `bench/emergent_log.jsonl`
(the 30 entries with `iso_ts` `2026-05-12T15:...Z`).

**Aggregate: 29/30 UNGROUNDED · 1/30 HYBRID · 0 STRICT.** This is the
*expected, healthy* null result — the triangulator generates absurd
questions ("How does the depth of one's music-making abilities on a
kazoo …", "How can a hardliner in Syria, during wintertime, maintain
…") that have no real corpus support, and the verifier ladder
honestly returns UNGROUNDED rather than hallucinating-then-STRICTing.
**0 false-positive STRICT on garbage input — the ladder is doing its
job.** (The `n_verified/n_quotes` "0/N" on the UNGROUNDED ones is the
model emitting N claims of which 0 verify — correct.)

**The one HYBRID — `perversely / succeeds / disagreeably`** (a re-run
of failure shape **E**, "total deflection to unrelated trivia",
catching as designed):
- Question: "How does one perversely succeed in doing something
  disagreeably?" — all three words are common enough that FTS5
  pulled chunks from *Hirabah* + *Original righteousness* (the words
  appear incidentally in that prose).
- Answer: a verbatim block about Quranic punishment for piracy +
  "original righteousness" — **total topic-drift** from the question.
- Verdict: `claim_lattice`, n_verified 3/4 (three claims quoted the
  retrieved chunks verbatim → verify), violations
  `['CITATION_MISMATCH', 'DEFLECTION_DETECTED']` → **capped at HYBRID,
  not STRICT**. The deflection sidecar caught the topic-drift and the
  verifier correctly refused to call a verbatim-but-irrelevant answer
  STRICT. Nothing new — this is shape E (`a1dd330` territory) firing
  on a fresh instance; logged here as a confirmation that the
  deflection + verbatim machinery holds on adversarial-by-construction
  input.

**No new gem.** Nothing in these 30 surfaces a new failure shape or a
correctness concern — the run is a clean re-confirmation of the
verifier-ladder health (UNGROUNDED-on-garbage; HYBRID-not-STRICT on
verbatim-but-deflected). The 30 entries join the
`scripts/bench_emergent.py --print-pending` queue if a future
teacher-review pass wants to look closer; on this skim none demand it.
