Bench-emergent stress test ran another 100 cycles under the
post-#000008/9/10/11 substrate. Total accumulated: 300 cycles.
Verdict distribution shift on last 100 vs 134-cycle baseline:
STRICT 5% (7/134) → 0% (0/100) -5pp
HYBRID 22% (29/134) → 16% (16/100) -6pp
UNGROUNDED 73% (98/134) → 84% (84/100) +11pp
Zero false-positive STRICTs across 100 random-word triplets.
The 5pp drop in STRICT-rate isn't a regression — it's the
verifier ladder + new preflight contracts doing their job.
Random-word triplets are genuinely ungrounded for the most
part; the prior 5% STRICT rate included false-positives that
the post-hardening verifier now catches.
Violation profile (last 100 cycles, claim_lattice JSON):
CITATION_MISMATCH: 86 dominant gate
TOO_MANY_EVIDENCE_IDS: 24
SUBJECT_TOKENS_ABSENT: 12 Rule 9 firing on parroting
DEFLECTION_DETECTED: 12
TITLE_MISMATCH: 10
...
metaphor_deflection fires 6/100 — still rare. Item 3
(calibration) is now closer to sample-size threshold (~30
signals across 300 cycles; needs ~50-100 to calibrate).
No new tuning candidates surface. Original three remain at
their resolution states.
Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.
#000025 — Metacog test fixture expansion:
bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
covering edge cases per detector kind: temporal (4 cases),
contradiction (4), false-premise (5), out-of-corpus (3), multi-
trigger (2), well-formed controls (5). Documents two known
detector ceilings: Q11 over-fires on past-tense factoid
("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
NASA-fake) miss false premises that lack a presupposition
pattern match. Fixture now serves as long-term regression suite.
#000026 — --show-preflight full clause render:
build_run_dag() and build_reject_run_dag() gain optional
preflight_payload kwarg. When supplied, the canonical 5-clause
CTI payload (classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs + question_state + node_version)
persists alongside the leaf hash in run_dag_blob.
aborist providence --show-preflight CACHE_KEY now renders the
full payload + verifies the persisted hash matches the
recomputed canonical hash (audit-replay tamper detection).
Legacy rows fall through cleanly: payload_hash_check reports
"unavailable: legacy row predates preflight_payload persistence".
#000027 — Latency profile:
Microbenched preflight: 0.46ms/question (negligible). Single
fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
33-35s in Addendum 3 was vLLM concurrency contention at c=4
(per qa-modes-bench.md saturation note), not substrate
overhead. Added preflight_ms + soft_preflight_ms to timings
dict for explicit confirmation in future cycles.
#000028 — Auto-quality-check sweep revival:
scripts/bench_emergent.py running with EMERGENT_N=100 in
background (PID 125680). Will accumulate cycles into
bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
Async — not blocking on completion.
#000029 — #000011 SOFT_PREFLIGHT_HINT implementation:
aborist/qa/soft_preflight.py — new module. SoftPreflightHint
dataclass + soft_preflight_question() pure function. 9
canonical labels mapping to soft analogues of #000010 hard
detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
Constrained-generation prompt (max_tokens=128, temp=0.0) asks
the model to pick ONE label + one-line rationale.
Fail-closed across every parse path:
- chat_client raises → SOFT_PARSE_FAIL
- response unparseable → SOFT_PARSE_FAIL
- label outside enum → SOFT_PARSE_FAIL
Sidecar enforces SOFT_ prefix at the normalize step so a
model that drops the prefix still gets caught.
Wired into query() between preflight & retrieval. Default
OFF (`soft_preflight_enabled: False`). NOT folded into
_VERIFIER_POLICY_FIELDS — soft hints don't gate cache
identity (#000011 §4). Audit-line tail renders as
"· soft: <label>" (e.g. "· soft: time sensitive") so the
signal is visually distinct from hard tails.
--soft-preflight CLI flag opts in per-call. End-to-end
live-verified on "When did Mr. Burns become Homer's biological
father?" — produces:
EVIDENCE-WARRANTED · via claim_lattice
· false premise · soft: time sensitive
1/1 16.4s
Hard `· false premise` (from #000010 deterministic detector)
composed with soft `· soft: time sensitive` (from #000011
sidecar). The model classified a different shape than the hard
detector — by design; soft hints are independent advisory
signals, not redundant with the hard layer.
25 new tests pin: default-OFF behavior, parse-failure modes,
label normalization (SOFT_ prefix enforced), all 8 actionable
labels round-trip, fail-closed on client exceptions, dataclass
JSON round-trip, rationale-length cap.
Other:
- #000010 §13.3 documents 2/5 metacog-trigger questions return
STRICT despite hard-detector warning — direct empirical
motivation for #000011 design.
- tests/test_dag.py extends with 3 _extract_preflight_hash_*
helper tests (cleaning #000009 §7.2 unfinished state).
- bench/emergent_log.jsonl adds new cycles from background run.
#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
Re-aggregated bench/emergent_log.jsonl at 200 cycles. Distribution
holds (79.5% UNGROUNDED, 17.5% HYBRID, 3.0% STRICT). Six STRICTs
total; five defensible (known-truth or non-relationship grounding).
The sixth — `steer/reply/correcter` — is the FIRST CONFIRMED
EVIDENCE-WARRANTED false-positive in 200 cycles. Decompressed all
10 chunks of the cited document (Glossary of language teaching
terms and ideas, 33.5K chars) and confirmed the words `correcter`,
`steer`, and `reply` appear NOWHERE in the source. Yet STRICT 1/1
verified, no violations.
Mechanism: per-claim citation-coverage Rule 5 (≥30% claim tokens
in cited span) passed on GENERIC linguistic vocabulary (language,
communication, grammar, exchange, relationships) which appears in
any linguistics article. The question-distinctive tokens
(correcter, steer, reply) rode along unverified. PREMISE_PARROTING
shape predicted in original ticket §D, exhibit A.
Headline correction: from "zero false-positives" to "one confirmed
false-positive (0.5%)."
Action item: opens design space for #000008 — PREMISE_PARROTING
detector. Lexical signal: question-distinctive tokens absent from
cited chunk while Rule 5 still passes on generic vocabulary →
SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays
binary, stays lexical, folds into verifier_policy_hash.
The ticket file was at docs/ticket-000006-... at top-level; the
tickets/ subdir convention landed before #6 was opened. Move into
docs/tickets/ to match every other ticket.
Amend with the 62-entry post-ticket delta. bench/emergent_log.jsonl
now holds 134 cycles total. Distribution stable vs original 72:
106 UNGROUNDED (79.1%), 22 HYBRID (16.4%), 6 STRICT (4.5%). Ten
new grounded cases all carry appropriate violation labels — zero
EVIDENCE-WARRANTED false-positives across all 134 cycles. Original
tuning candidates: Tomas-deflection resolved by a1dd330,
xxviii-STRICT kept as designed, metaphor sidecar calibration
deferred (still rare).
Reframe ticket as a rolling research log. Emergent stress-testing
is an ongoing thread, not a one-shot defect-fix; future
bench-emergent re-runs append new ## Amend sections here, and any
code-level tunings open their own tickets linking back.
TICKETS.md row reflects the rolling-research framing. Stale
"## Next ID" footer scrubbed from the ticket body (TICKETS.md is
canonical for the next-id counter).
72 random-word triangulation cycles ran on 2026-05-02. Aggregate:
- 54 UNGROUNDED (75%)
- 14 HYBRID (19%)
- 4 STRICT (6%)
- 18 grounded total (25%)
Catalogs five failure/success shapes the curated bench-qa fixture
set can't surface, with one-line per category. Key finding: the
verifier-ladder + soft-demote stack is doing its job — every
HYBRID demoted via TITLE_MISMATCH / DEFLECTION_DETECTED /
CITATION_MISMATCH / TOO_MANY_EVIDENCE_IDS appropriately. Zero
EVIDENCE-WARRANTED false-positives in the sample.
Tuning candidates queued for the parallel-agent work:
1. Tomas-pulls-transceivers case escaped DEFLECTION_DETECTED
(subject anchor buried mid-paragraph; consider last 2-3
content tokens vs just last 1)
2. Roman-numeral / archaic-word triplets — accept gracefully
(system already does)
3. metaphor_deflection sidecar (added in parallel work) only
fired 1×/72; calibration pass when more samples land
Ticket is OPEN and awaiting follow-up tunings; re-run
`make bench-emergent EMERGENT_N=50` after parallel work commits
land and amend with the delta.
Status table + Next ID bumped to 000007.
Emergent stress-test log surfaced two failure shapes the curated
bench can't reach. Reviewer (claude-opus-4-7 + fox) appended
teacher: blocks to four log entries; two of them named real
defects in the demote chain. This commit ships those fixes plus
the teacher-block annotations.
Cases reviewed (2026-05-02 emergent_log.jsonl):
✅ Menkar/sterilizers/trifle — working as designed
🟡 swallowtail/upbraided/rockiest — metaphorical deflection
🔴 cashback/widescreens/within — warrant gap on inferential claims
🔴 comeliness/fetish/investitures — topic shift earned ANCHOR-WARRANTED unfairly
Two code changes:
1) verify_claim_lattice + verify_claim_lattice_json: when EVERY
resolving claim has TITLE_MISMATCH, demote audit_mode →
UNGROUNDED. Previously HYBRID. Catches the cashback case
('widescreens offer cashback' cited to a generic Coupon
article — n_verified=1 overclaimed when the citation was
meaningless).
2) cli._ladder_rung_for_lattice: DEFLECTION_DETECTED moves from
soft-demote (cap at ANCHOR-WARRANTED) to hard-demote (cap at
POINTER-LINKED) alongside WARRANT_MISSING and TITLE_MISMATCH.
Catches the comeliness case (model fully shifted topic to
'Rock & Chips'; pre-fix the verified Rock & Chips claims
landed at ANCHOR-WARRANTED unfairly).
Test updates:
- tests/test_verify_json.py:test_verify_json_title_mismatch_demotes_to_ungrounded
(renamed; expected outcome flipped HYBRID → UNGROUNDED)
- 4 fixtures in tests/test_claim_lattice.py and 1 in
tests/test_verify_json.py: claim text augmented with 'in the
film' so the Jurassic-Park-titled fixture's title shares a
stem with the claim. These tests were testing other paths
(NO_EVIDENCE_POINTER, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID)
and the title-mismatch was incidental to the fixture choice.
Teacher blocks on the four log entries record the verdict +
failure_class + recommendation + notes, with reviewer + date.
The two ✅/🟡 entries have no recommended code change; the two 🔴
entries reference this commit as the fix.
751/34 tests pass.
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
