arborist/docs/tickets/ticket-000006-bench-emergent-findings.md

Ticket #000006 — Bench-emergent findings (first 72 cycles)

Status: open · rolling research log (verifier-ladder healthy at 134 cycles; emergent gems still being mined) Opened: 2026-05-02 Last amend: 2026-05-02 Scope: catalog the failure & success shapes the random-word triangulation harness surfaced; queue up tunings the curated bench can't suggest because it doesn't see these question shapes. Audience: anyone tuning the verifier / warrant / deflection checks after the initial bench-emergent shake-out. Hard constraint: every recommendation here must stay lexical. NLI / LLM-as-judge stays sidecar territory. The verifier-ladder discipline (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED) is preserved across all proposed changes.

What ran

scripts/bench_emergent.py ran 72 cycles in 4 batches on 2026-05-02 (UTC 17:30-18:00 ish). Each cycle:

1. Three random words from /usr/share/dict/words (^[a-z]{5,12}$).
2. Hermes @ temp=0.8 wove a creative question paragraph.
3. Arborist answered via claim_lattice mode (full pipeline).
4. Append journey to bench/emergent_log.jsonl.

Aggregate distribution:

audit_mode	count	%
UNGROUNDED	54	75%
HYBRID	14	19%
STRICT	4	6%
grounded total	18	25%

Violation-kind frequencies (across all 72 cycles, mostly UNGROUNDED):

kind	count	what it indicates
CITATION_MISMATCH	57	per-pair citation-coverage failure (Rule 5 of the seven hard checks). Most common — model cited a chunk that shares few/no content tokens with the claim.
TOO_MANY_EVIDENCE_IDS	11	pointer-id over-citation; trim-and-verify still fires
DEFLECTION_DETECTED	11	subject-anchor missing from answer (the soft-demote landed today).
TITLE_MISMATCH	7	cited source's title shares no content token with the claim (Rule 8).
MANUAL_QUOTE_VIOLATION	2	model used " chars in pointer-mode claim text
SOURCE_ROLE_BLOCKED	1	cited a noisy source role
metaphor_deflection	1	the new metaphor sidecar (added by parallel work) — fired once

Failure shapes catalogued

The 18 grounded entries split into five categories. Each shows something the curated bench couldn't surface.

A. Honest non-relationship grounding (the gold case)

['brontosaurs', 'Bennett', 'belaboring'] — STRICT 1/1.

Q: What connection could possibly exist between belaboring a point, the dinosaur brontosaurs, and Bennett, an evidently unrelated surname? A: The Sinclair Oil Corporation's use of the brontosaurus dinosaur logo is unrelated to the Bennett surname. Cited: Sinclair Oil Corporation

Model identified non-relationship + grounded the partial truth (Sinclair logo is real). This is the emergent_synthesis novelty class — connects facts to refute the question's implicit premise. Don't tune this away.

B. Genuine emergent triangulation grounded in real corpus

['sprayers', 'hearties', 'xxviii'] — STRICT 2/2. ['Brazil', 'directorates', 'tropic'] — STRICT 1/1. ['viewing', 'octopus', 'exculpated'] — STRICT 1/1.

Model gracefully ignored unparseable tokens (Roman-numeral nonsense, archaic words) + grounded answers in real Wikipedia articles (Water fight, Water supply and sanitation in Brazil, etc.). known_truth_grounding novelty class. STRICT is honest here.

C. Lazy-anchor over-citation honestly demoted

['arguments', 'publicists', 'apparatuses'] — HYBRID 1/11.

11 pointer-ids cited, only 1 verified. Triple-violation: CITATION_MISMATCH + DEFLECTION_DETECTED + TITLE_MISMATCH. Display label: POINTER-LINKED-PARTIAL · via claim_lattice. The verifier honestly reports the over-citation; ladder demoted to the bottom rung. Working as designed.

D. False-premise affirmation (TITLE_MISMATCH catches)

['widescreens', 'cashback', 'within'] — HYBRID 1/1, TITLE_MISMATCH.

Q: How do widescreen televisions offer cashback deals within a limited time frame to encourage consumers to purchase? A: Widescreen televisions offer cashback deals within a limited time frame to encourage consumers to purchase. Cited: Coupon (generic article on coupons)

Model affirmed the question's unverified premise, citing a generic Coupon article that doesn't anchor "widescreen" or "cashback" specifically. TITLE_MISMATCH fires because "Coupon" doesn't share any content token with "widescreen televisions". Display: POINTER-LINKED-PARTIAL · title mismatch. Working as designed.

Tuning candidate: a stricter "premise-affirmation" detector that compares the answer's content tokens to the question's content tokens — when answer tokens ≈ question tokens minus generic verbs, flag as PREMISE_PARROTING (claim text is the question's premise restated). Deferred until bench shows this pattern under EVIDENCE-WARRANTED at a measurable rate; today TITLE_MISMATCH catches the cases we have.

E. Total deflection to unrelated trivia (DEFLECTION_DETECTED catches)

['comeliness', 'fetish', 'investitures'] — HYBRID 2/3, DEFLECTION_DETECTED. Cited Rock & Chips (Only Fools and Horses prequel).

['saleswomen', 'excesses', 'libidinous'] — HYBRID 3/3, DEFLECTION_DETECTED. Cited The New Tomorrow + Mordred.

['Tomas', 'pulls', 'transceivers'] — HYBRID 1/1, no deflection violation (model latched onto a Tomas-named character; subject anchor "transceivers" ambiguous). Edge case worth investigating.

DEFLECTION_DETECTED fires on the first two. The third escapes the subject-anchor heuristic because _content_tokens_in_order() may have placed transceivers at a different position. Tuning candidate: investigate the deflection-sidecar shape detection on Tomas-style questions where the subject is buried mid-paragraph.

Suggested tuning passes (ranked)

The other agent picking up these findings should consider:

1. Tomas-pulls-transceivers case (E.3) — deflection sidecar missed it. Either:
   • Strengthen subject-anchor heuristic (try last 2-3 content tokens, not just last 1)
   • Add overlap-ratio threshold (currently subject-anchor takes precedence over overlap; maybe combine signals)
   • Document why this case legitimately escapes the check
2. xxviii STRICT acceptance — Roman numerals ignored gracefully is fine; should the bench-emergent harness filter them out at word-pick time? Easy to add to _WORD_RE. But xxviii surfaced an interesting case (model gracefully ignoring nonsense), so maybe keep it.
3. metaphor_deflection sidecar (added in parallel) — only fired 1×/72. Either the bench shape rarely produces metaphors, or the detection threshold needs tuning. Worth a calibration pass.

Conclusion

The verifier-ladder + soft-demote stack is doing its job on the random-word stress test. 25% grounded rate is healthy (triplets that genuinely cluster in 2010 Wikipedia). Honest UNGROUNDED dominates (75%) — that's correct behavior on triplets the corpus genuinely lacks.

The 4 STRICTs are all defensible (honest grounding or non-relationship identification). The 14 HYBRIDs surface 4 distinct failure shapes — TITLE_MISMATCH, DEFLECTION_DETECTED, CITATION_MISMATCH, TOO_MANY_EVIDENCE_IDS — all of which already demote via the soft-demote ladder.

The biggest discovery is what the bench DIDN'T surface: zero EVIDENCE-WARRANTED false-positives. Every grounded run was either honestly STRICT or honestly demoted. The verifier ladder catches what it's supposed to.

Amend — 2026-05-02 (delta + close)

Re-aggregated bench/emergent_log.jsonl after the parallel-agent work landed (a1dd330 deflection-tightening, 8fec3a5 metaphor sidecar, b87966c metaphor cue derivation) and after Ticket #000007 (query-layer hyphen fold). Log now holds 134 cycles — the original 72 plus 62 post-ticket cycles.

Combined 134-cycle distribution:

audit_mode	count	%	vs original 72
UNGROUNDED	106	79.1%	+4.1pp
HYBRID	22	16.4%	-2.6pp
STRICT	6	4.5%	-1.5pp
grounded total	28	20.9%	-4.1pp

Distribution is stable. Verifier-ladder discipline holds: the 10 post-ticket grounded cases (2 STRICT + 8 HYBRID) all carry appropriate violation labels. No EVIDENCE-WARRANTED false-positives.

Post-ticket grounded breakdown:

triplet	mode	violations
reprehending/Jonesboro/Claus	STRICT	— (honest grounding, Santa Claus mythology)
steer/reply/correcter	STRICT	— (honest grounding)
chiller/tromps/naughtiest	HYBRID	DEFLECTION_DETECTED
teazle/ensconced/Jannie	HYBRID	DEFLECTION_DETECTED
sowers/leakage/earbuds	HYBRID	CITATION_MISMATCH + TOO_MANY_EVIDENCE_IDS
Pigmy/aortas/spoiler	HYBRID	DEFLECTION_DETECTED
colonnade/snacked/acutest	HYBRID	DEFLECTION_DETECTED + TITLE_MISMATCH
posturing/diminutions/ligatures	HYBRID	CITATION_MISMATCH
trace/Markham/crawfishes	HYBRID	DEFLECTION_DETECTED + WARRANT_MISSING
underfeed/stoppered/biorhythm	HYBRID	CITATION_MISMATCH + TITLE_MISMATCH + WARRANT_MISSING

Delta violation kinds: CITATION_MISMATCH 53, TOO_MANY_EVIDENCE_IDS 18, DEFLECTION_DETECTED 5, TITLE_MISMATCH 5, WARRANT_MISSING 3, UNKNOWN_EVIDENCE_ID 1, MANUAL_QUOTE_VIOLATION 1. WARRANT_MISSING is new vs the original snapshot — confirms the warrant-strengthening work is catching inferential-claim cases the ladder previously admitted.

Tuning candidate resolution:

1. Tomas-pulls-transceivers (deflection sidecar miss) — resolved. Commit a1dd330 ("qa: tighten TITLE_MISMATCH + DEFLECTION demotes from emergent-log review") landed at 17:33 UTC 2026-05-02, ~20 minutes BEFORE this ticket was opened. The deflection check on the Tomas log entry now reports DEFLECTION_DETECTED correctly. Ticket text was authored from a stale snapshot. Post-ticket delta confirms the check fires on every applicable shape (4 of 5 deflection-shaped HYBRID cases). No code change needed.

2. xxviii STRICT acceptance — kept as designed. No new evidence to filter Roman-numeral / archaic-word triplets at word-pick time. Graceful ignoring is a feature, not a bug.

3. metaphor_deflection sidecar calibration — deferred. Fired 0× across the 43 post-ticket cycles. Sample too small for calibration; revisit when corpus-stress accumulates more samples. The sidecar shipped (8fec3a5, b87966c) and stays live; no calibration knob to turn yet.

Headline finding (unchanged): zero EVIDENCE-WARRANTED false-positives across all 134 cycles. The four-rung ladder catches what it's supposed to.

Ticket stays open — emergent stress-testing is ongoing research, not a one-shot defect. The amend section is the rolling delta log: append a new ## Amend — YYYY-MM-DD section after each substantive bench-emergent re-run with the new distribution + findings. Code-level tunings that emerge from this research open their own tickets and link back here.

Amend — 2026-05-02b (200 cycles, suspect EVIDENCE-WARRANTED gem)

Re-aggregated at 200 cycles total. Distribution stays stable:

audit_mode	count	%	vs 134-cycle
UNGROUNDED	159	79.5%	+0.4pp
HYBRID	35	17.5%	+1.1pp
STRICT	6	3.0%	-1.5pp

STRICT cases (all six, full set):

triplet	source cited	shape
sprayers/hearties/xxviii	Water fight	known-truth grounding (Roman numeral ignored)
viewing/octopus/exculpated	Water-supply article	known-truth grounding
Brazil/directorates/tropic	Water supply and sanitation in Brazil	known-truth grounding
brontosaurs/Bennett/belaboring	Sinclair Oil Corporation	non-relationship grounding (gold case)
reprehending/Jonesboro/Claus	The Flight of the Phoenix (1965 film)	non-relationship grounding (Humphrey-not-elf)
steer/reply/correcter	Glossary of language teaching terms and ideas	suspect — see below

Five of six STRICTs are defensible. The sixth (steer/reply/correcter) is the first plausible EVIDENCE-WARRANTED false-positive in 200 cycles — worth deeper scrutiny.

Suspect case: steer/reply/correcter

Q: How might a correcter be used to steer a reply in a conversation, and what aspects of language or communication do these three terms encompass in their relationships with each other? A: A correcter can be used to steer a reply in a conversation by identifying and correcting errors in grammar, spelling, or punctuation in the original message. This helps ensure that the reply is accurate and clear. The three terms — correcter, grammar, and communication — encompass various aspects of language in their relationships with each other. Grammar refers to the set of structural rules that govern the composition of sentences, phrases, and words in a language. Communication involves the exchange of information, ideas, thoughts, and feelings between individuals through a common system of symbols, such as language. A correcter plays a role in maintaining the accuracy of language in communication by identifying and correcting errors in grammar, spelling, or punctuation, which can help prevent misunderstandings or confusion in the conversation. Cited: E1 (Glossary of language teaching terms and ideas).

This is premise-affirmation + confabulated definition. The cited spotlight excerpt is a Sociolinguistics entry about "exchange of meaning, relationships, mode of discourse, channel" — generic language-discourse vocabulary. The model wove a definition-style answer using overlapping vocabulary that the verifier's per-claim citation-coverage check passed.

arborist inspect --cache-key 08dbd2c1... confirms STRICT 1/1 verified, no unverified spans, no violations. The verifier genuinely accepted the claim.

Discriminator run (2026-05-02): decompressed all 10 chunks of the cited document (6edf7202… Glossary of language teaching terms and ideas, ~33,565 chars total) and grep'd for the claim's distinctive subject tokens:

chunk 0..9: correcter=False steer=False reply=False

Confirmed false positive. The words correcter, steer, and reply appear NOWHERE in the cited document. Yet the verifier accepted STRICT 1/1 with no violations on the claim "A correcter can be used to steer a reply in a conversation by identifying and correcting errors in grammar, spelling, or punctuation…"

Mechanism

Per-claim citation-coverage (Rule 5) requires ≥30% of the claim's content tokens to appear in the cited span. The claim contains ~50 content tokens, of which ~30+ are GENERIC linguistic vocabulary (language, communication, grammar, exchange, relationships, meaning, discourse, aspects, terms…) — all of which appear in any linguistics article including this glossary. Coverage check passes on the generic vocabulary; the QUESTION-DISTINCTIVE tokens (correcter, steer, reply) ride along unverified.

Same defect class as the pre-#000007 bi-polar bug, but turned inward: token coverage is symmetric over the claim, but the SUBJECT of the claim (the unusual tokens) is unverified. The model premise-parroted the question and clothed it in plausible definitional vocabulary that incidentally token-overlapped the cited chunk.

Headline correction

200-cycle headline updates from "zero EVIDENCE-WARRANTED false-positives" to "one confirmed EVIDENCE-WARRANTED false-positive (0.5%) — steer/reply/correcter." The four-rung ladder still works on every other case in the sample; this is one specific shape — PREMISE_PARROTING / GENERIC-VOCAB-RIDE-ALONG — that the lexical Rule-5 check cannot catch alone.

Action items emerging from 200-cycle delta

1. Rule 9 — SUBJECT_TOKENS_ABSENT detector — landed inline (no separate ticket). New per-claim check in arborist/qa/verify.py. For each resolving claim, compute the set of content tokens shared by question AND claim (_parroted_subject_tokens_absent). For each, check substring presence in the union of cited evidence spans (lower-cased, same as Rule 5). When ≥ subject_tokens_absent_threshold (default 3) parroted tokens are absent → emit SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays lexical, stays binary, folds into verifier_policy_hash + governance_policy_hash.

   Live verification on the original case (cache-split via policy-hash bump):

   cache_key	policy	audit_mode
   08dbd2c1…	pre-fix	STRICT (false positive)
   6a519636…	post-fix	UNGROUNDED

   Ladder now correctly demotes the parroted answer. Three new unit tests pin the signature: demote on threshold-meeting parrot, no-op when question is None, no-demote below threshold.

2. metaphor_deflection sidecar still rare (low single-digit fires across 200 cycles). Calibration deferred until 300+.

Other tuning candidates from the original ticket remain resolved (Tomas) or kept-as-designed (xxviii).

Status

Open · rolling research log. Three original tuning candidates: (1) Tomas-pulls-transceivers — resolved by a1dd330 (deflection sidecar tightened); (2) xxviii STRICT acceptance — kept as designed; (3) metaphor sidecar calibration — deferred until sample size grows. Re-amend as findings accumulate.

Cross-reference: ticket #000008 bench cycles (2026-05-03)

Ticket #000008 (broad-quantifier preflight guard) landed Phases 0–4 on 2026-05-03 and now runs its own bench A/B cycles per its §12 with the --policy KEY=VALUE harness flag (commit 002f84c). Findings relevant to the rolling-emergent stream:

• The bench/qa_questions.txt 73-question set carries 7 broad questions (~10%) classified ALL/COMPREHENSIVE/OPEN_REQUEST by the §10 classifier. Two bounded-universal fixtures added 2026- 05-03 (name all members of the beatles, list all planets in the solar system).
• Classifier defect caught + fixed in d24291b: how many X? was mis-classifying as MANY (4 false positives across the bench). Fixed via leading-anchor count-question short-circuit.
• metaphor_deflection calibration (item 3 above) can leverage #000008's larger bench cycles — the broad-subset A/B at n=3 × 3 modes accumulates 81 rows per cell, useful sample size growth for metaphor-sidecar tuning if the metaphor cue happens to fire on broad-quantifier shapes.

Amend — 2026-05-04 (300-cycle update + post-substrate-hardening)

Bench-emergent stress test ran another 100 cycles in background (scripts/bench_emergent.py --n 100) under the post-#000008/9/10/11 substrate. bench/emergent_log.jsonl now holds 300 total cycles.

Headline shift on the last 100 cycles (vs the 134-cycle baseline distribution):

Verdict	134-cycle baseline	Last 100 (post-hardening)	Δ
STRICT	5% (7/134)	0% (0/100)	-5pp
HYBRID	22% (29/134)	16% (16/100)	-6pp
UNGROUNDED	73% (98/134)	84% (84/100)	+11pp

Zero false-positive STRICTs across 100 random-word triplets. The substrate is now dramatically more honest than at the 134- cycle baseline. Every cycle goes through claim_lattice JSON mode (the bench-emergent harness pins this).

Violation profile on last 100 cycles (claim_lattice JSON):

Kind	Count	Comment
CITATION_MISMATCH	86	dominant gate; per-claim coverage check
TOO_MANY_EVIDENCE_IDS	24	model cites >2 ids per claim
SUBJECT_TOKENS_ABSENT	12	Rule 9 firing on premise-parroting
DEFLECTION_DETECTED	12	answer's subject doesn't match question's
TITLE_MISMATCH	10	cited evidence's source title shares zero stems
MANUAL_QUOTE_VIOLATION	4	model typed a quote span (CTI invariant break)
TOO_MANY_CLAIMS	2	claim count exceeds cap (12 default)
UNKNOWN_EVIDENCE_ID	2	model invented an evidence_id
WARRANT_MISSING	1	anchor-class warrant didn't fire

metaphor_deflection signal fires on 6/100 cycles — still rare (item 3 from the original tuning candidates). Sample size 100 is borderline for calibration; another 100-200 cycles would let us decide whether to tighten or relax the threshold.

Interpretation. The 5pp drop in STRICT-rate isn't a regression — it's the substrate's verifier ladder + new preflight contracts (#000008/9/10/11) doing their job. Random-word triplets are genuinely ungrounded for the most part; the prior 5% STRICT rate included false-positives that the post-hardening verifier now catches. The bench-emergent harness is a stress test designed to surface failure shapes; honest UNGROUNDED on inputs the corpus can't ground is the correct outcome.

No new tuning candidates surface from this amend. The original three (Tomas-pulls-transceivers, xxviii Roman numerals, metaphor calibration) remain at their respective resolution states. Item 3 (metaphor calibration) is now closer to having the sample size it needs (300 cycles total; ~30 metaphor signals across them; needs ~50-100 to calibrate).

Bench artifact: bench/emergent_log.jsonl (300 lines). Background run completed 2026-05-04T02:46Z (PID 125680, ~25min wall-clock for 100 cycles).