engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ticket(#000006): 300-cycle update — zero false-positive STRICT post-hardening

Browse source 
Bench-emergent stress test ran another 100 cycles under the
post-#000008/9/10/11 substrate. Total accumulated: 300 cycles.

Verdict distribution shift on last 100 vs 134-cycle baseline:

  STRICT      5% (7/134)  →  0% (0/100)   -5pp
  HYBRID      22% (29/134) →  16% (16/100) -6pp
  UNGROUNDED  73% (98/134) →  84% (84/100) +11pp

Zero false-positive STRICTs across 100 random-word triplets.

The 5pp drop in STRICT-rate isn't a regression — it's the
verifier ladder + new preflight contracts doing their job.
Random-word triplets are genuinely ungrounded for the most
part; the prior 5% STRICT rate included false-positives that
the post-hardening verifier now catches.

Violation profile (last 100 cycles, claim_lattice JSON):
  CITATION_MISMATCH:      86  dominant gate
  TOO_MANY_EVIDENCE_IDS:  24
  SUBJECT_TOKENS_ABSENT:  12  Rule 9 firing on parroting
  DEFLECTION_DETECTED:    12
  TITLE_MISMATCH:         10
  ...

metaphor_deflection fires 6/100 — still rare. Item 3
(calibration) is now closer to sample-size threshold (~30
signals across 300 cycles; needs ~50-100 to calibrate).

No new tuning candidates surface. Original three remain at
their resolution states.
This commit is contained in:
russell@unturf.com 2026-05-03 23:16:45 -04:00
parent a94d6a3244
commit 453e340e08
No known key found for this signature in database
2 changed files with 112 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

53
bench/emergent_log.jsonl
View file

File diff suppressed because one or more lines are too long

59
docs/tickets/ticket-000006-bench-emergent-findings.md
View file

@ -402,3 +402,62 @@ Findings relevant to the rolling-emergent stream:
3 modes accumulates 81 rows per cell, useful sample size growth 3 modes accumulates 81 rows per cell, useful sample size growth
for metaphor-sidecar tuning if the metaphor cue happens to fire for metaphor-sidecar tuning if the metaphor cue happens to fire
on broad-quantifier shapes. on broad-quantifier shapes.
## Amend — 2026-05-04 (300-cycle update + post-substrate-hardening)
Bench-emergent stress test ran another 100 cycles in background
(`scripts/bench_emergent.py --n 100`) under the post-#000008/9/10/11
substrate. `bench/emergent_log.jsonl` now holds **300 total cycles**.
**Headline shift on the last 100 cycles** (vs the 134-cycle
baseline distribution):
| Verdict | 134-cycle baseline | Last 100 (post-hardening) | Δ |
|-------------|--------------------|----------------------------|----------|
| STRICT | 5% (7/134) | **0% (0/100)** | -5pp |
| HYBRID | 22% (29/134) | 16% (16/100) | -6pp |
| UNGROUNDED | 73% (98/134) | **84% (84/100)** | +11pp |
**Zero false-positive STRICTs across 100 random-word triplets.**
The substrate is now dramatically more honest than at the 134-
cycle baseline. Every cycle goes through `claim_lattice` JSON mode
(the bench-emergent harness pins this).
**Violation profile on last 100 cycles** (claim_lattice JSON):
| Kind | Count | Comment |
|------------------------|-------|-------------------------------------------------|
| CITATION_MISMATCH | 86 | dominant gate; per-claim coverage check |
| TOO_MANY_EVIDENCE_IDS | 24 | model cites >2 ids per claim |
| SUBJECT_TOKENS_ABSENT | 12 | Rule 9 firing on premise-parroting |
| DEFLECTION_DETECTED | 12 | answer's subject doesn't match question's |
| TITLE_MISMATCH | 10 | cited evidence's source title shares zero stems |
| MANUAL_QUOTE_VIOLATION | 4 | model typed a quote span (CTI invariant break) |
| TOO_MANY_CLAIMS | 2 | claim count exceeds cap (12 default) |
| UNKNOWN_EVIDENCE_ID | 2 | model invented an evidence_id |
| WARRANT_MISSING | 1 | anchor-class warrant didn't fire |
`metaphor_deflection` signal fires on 6/100 cycles — still rare
(item 3 from the original tuning candidates). Sample size 100 is
borderline for calibration; another 100-200 cycles would let us
decide whether to tighten or relax the threshold.
**Interpretation.** The 5pp drop in STRICT-rate isn't a regression
— it's the substrate's verifier ladder + new preflight contracts
(#000008/9/10/11) doing their job. Random-word triplets are
genuinely ungrounded for the most part; the prior 5% STRICT rate
included false-positives that the post-hardening verifier now
catches. The bench-emergent harness is a stress test designed to
surface failure shapes; honest UNGROUNDED on inputs the corpus
can't ground is the correct outcome.
**No new tuning candidates** surface from this amend. The original
three (Tomas-pulls-transceivers, xxviii Roman numerals, metaphor
calibration) remain at their respective resolution states. Item 3
(metaphor calibration) is now closer to having the sample size it
needs (300 cycles total; ~30 metaphor signals across them; needs
~50-100 to calibrate).
**Bench artifact:** `bench/emergent_log.jsonl` (300 lines).
Background run completed 2026-05-04T02:46Z (PID 125680, ~25min
wall-clock for 100 cycles).