Bench-emergent stress test ran another 100 cycles under the
post-#000008/9/10/11 substrate. Total accumulated: 300 cycles.
Verdict distribution shift on last 100 vs 134-cycle baseline:
STRICT 5% (7/134) → 0% (0/100) -5pp
HYBRID 22% (29/134) → 16% (16/100) -6pp
UNGROUNDED 73% (98/134) → 84% (84/100) +11pp
Zero false-positive STRICTs across 100 random-word triplets.
The 5pp drop in STRICT-rate isn't a regression — it's the
verifier ladder + new preflight contracts doing their job.
Random-word triplets are genuinely ungrounded for the most
part; the prior 5% STRICT rate included false-positives that
the post-hardening verifier now catches.
Violation profile (last 100 cycles, claim_lattice JSON):
CITATION_MISMATCH: 86 dominant gate
TOO_MANY_EVIDENCE_IDS: 24
SUBJECT_TOKENS_ABSENT: 12 Rule 9 firing on parroting
DEFLECTION_DETECTED: 12
TITLE_MISMATCH: 10
...
metaphor_deflection fires 6/100 — still rare. Item 3
(calibration) is now closer to sample-size threshold (~30
signals across 300 cycles; needs ~50-100 to calibrate).
No new tuning candidates surface. Original three remain at
their resolution states.
Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.
#000025 — Metacog test fixture expansion:
bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
covering edge cases per detector kind: temporal (4 cases),
contradiction (4), false-premise (5), out-of-corpus (3), multi-
trigger (2), well-formed controls (5). Documents two known
detector ceilings: Q11 over-fires on past-tense factoid
("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
NASA-fake) miss false premises that lack a presupposition
pattern match. Fixture now serves as long-term regression suite.
#000026 — --show-preflight full clause render:
build_run_dag() and build_reject_run_dag() gain optional
preflight_payload kwarg. When supplied, the canonical 5-clause
CTI payload (classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs + question_state + node_version)
persists alongside the leaf hash in run_dag_blob.
aborist providence --show-preflight CACHE_KEY now renders the
full payload + verifies the persisted hash matches the
recomputed canonical hash (audit-replay tamper detection).
Legacy rows fall through cleanly: payload_hash_check reports
"unavailable: legacy row predates preflight_payload persistence".
#000027 — Latency profile:
Microbenched preflight: 0.46ms/question (negligible). Single
fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
33-35s in Addendum 3 was vLLM concurrency contention at c=4
(per qa-modes-bench.md saturation note), not substrate
overhead. Added preflight_ms + soft_preflight_ms to timings
dict for explicit confirmation in future cycles.
#000028 — Auto-quality-check sweep revival:
scripts/bench_emergent.py running with EMERGENT_N=100 in
background (PID 125680). Will accumulate cycles into
bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
Async — not blocking on completion.
#000029 — #000011 SOFT_PREFLIGHT_HINT implementation:
aborist/qa/soft_preflight.py — new module. SoftPreflightHint
dataclass + soft_preflight_question() pure function. 9
canonical labels mapping to soft analogues of #000010 hard
detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
Constrained-generation prompt (max_tokens=128, temp=0.0) asks
the model to pick ONE label + one-line rationale.
Fail-closed across every parse path:
- chat_client raises → SOFT_PARSE_FAIL
- response unparseable → SOFT_PARSE_FAIL
- label outside enum → SOFT_PARSE_FAIL
Sidecar enforces SOFT_ prefix at the normalize step so a
model that drops the prefix still gets caught.
Wired into query() between preflight & retrieval. Default
OFF (`soft_preflight_enabled: False`). NOT folded into
_VERIFIER_POLICY_FIELDS — soft hints don't gate cache
identity (#000011 §4). Audit-line tail renders as
"· soft: <label>" (e.g. "· soft: time sensitive") so the
signal is visually distinct from hard tails.
--soft-preflight CLI flag opts in per-call. End-to-end
live-verified on "When did Mr. Burns become Homer's biological
father?" — produces:
EVIDENCE-WARRANTED · via claim_lattice
· false premise · soft: time sensitive
1/1 16.4s
Hard `· false premise` (from #000010 deterministic detector)
composed with soft `· soft: time sensitive` (from #000011
sidecar). The model classified a different shape than the hard
detector — by design; soft hints are independent advisory
signals, not redundant with the hard layer.
25 new tests pin: default-OFF behavior, parse-failure modes,
label normalization (SOFT_ prefix enforced), all 8 actionable
labels round-trip, fail-closed on client exceptions, dataclass
JSON round-trip, rationale-length cap.
Other:
- #000010 §13.3 documents 2/5 metacog-trigger questions return
STRICT despite hard-detector warning — direct empirical
motivation for #000011 design.
- tests/test_dag.py extends with 3 _extract_preflight_hash_*
helper tests (cleaning #000009 §7.2 unfinished state).
- bench/emergent_log.jsonl adds new cycles from background run.
#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
Fan-out execution of the deferred-but-not-blocking pile from
prior status reports.
#000009 §7.2 — bench harness preflight_hash field:
- aborist/qa/query.py surfaces `preflight_hash` on result dict
(miss path, reject path, and cache-hit path via new helper
`_extract_preflight_hash_from_blob` that pulls the stage hash
out of persisted run_dag_blob).
- bench/qa_sweep.py adds 12-char preflight_hash prefix to bench
rows. Mirrors cache_key truncation pattern. Operators can
grep / SQL-filter bench JSONL by preflight policy state.
- 3 new tests in tests/test_dag.py for the extract helper.
#000009 §7.2 — `aborist providence --show-preflight CACHE_KEY_PREFIX`:
- New CLI flag pulls the preflight stage payload from a row's
run_dag_blob. Match by 12-char prefix. Renders preflight stage
hash + run-DAG stage list. Operator tool for inspecting which
policy state governed a cached row.
- Live verified on a real cache row (8a212fecb2a9 — current CEO
of OpenAI question, 10-stage CTI shape with preflight at idx 1).
- Legacy rows (predating #000009) report a clean fall-through
message: "run_dag has no preflight stage (predates #000009)".
#000010 §12.6 — reference-frame plumbing into QuestionState:
- Pre-retrieval preflight runs with reference_frames=()
(frame_detection needs source titles, not available yet).
Post-retrieval, query.py re-runs preflight_question() with
the detected frames so the result-dict + run-DAG QuestionState
carry frame-aware logical_statuses (specifically
`reference_frame_ambiguous` when 2+ frames match).
- Live verified on Orwell-style question; logical_statuses now
correctly includes `reference_frame_ambiguous` in the result.
Metacog-trigger bench fixture (#000010 §13.3):
- bench/qa_questions_metacog_subset.txt — 6 questions, one per
detector kind plus a well-formed control.
- Bench artifact 2026-05-04T02-18-42Z. Detector accuracy 6/6
on fixture; 2 of 5 trigger questions return STRICT on lattice
mode despite metacog warning (JSON STRICT on
George-Washington-stop-being-president-of-France false-premise
+ uploaded-contract out-of-corpus questions). Audit-line tails
correctly surface the warnings.
- qa-modes-bench.md Addendum 4 captures the per-question matrix
+ interpretation. #000010 §13.3 cross-references with bench
artifact stamp.
#000011 SOFT_PREFLIGHT_HINT design ticket opened:
- docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md
captures the design proposal per #000010 §18 / source doc.
Implementation deferred — design only.
- Sidecar would add model-assisted preflight as a soft signal
(`SOFT_FALSE_PREMISE_SUSPECTED` etc.) that NEVER enters the
verifier hard path. Strict guardrail: cannot create
PREFLIGHT_OK or PREFLIGHT_BLOCKED without deterministic
support.
- Validated by §13.3 finding: deterministic detectors flag
correctly; corpus-accidental grounding produces 2/5 STRICT
on trigger questions; soft sidecar would add independent
semantic skepticism.
- TICKETS.md index row added; Next ID bumped to 000012.
996 tests passing (3 new for the extract helper).
Cross-doc consistency:
- qa-modes-bench.md Addenda 1+2+3+4 chronological
- #000010 §13.1 (broad subset) + §13.2 (full bench) + §13.3
(metacog trigger subset)
- #000011 design captured but not implemented
Closes ticket #000010 (Meta-Cognition Preflight Guard). Mechanism
complete; defaults preserve the dry-run discipline pattern from
#000008.
Phase 2 — wire preflight into query() and runner.ask():
- preflight_question() runs after policy resolution + quantifier
classification, before retrieval.
- QuestionState surfaces on miss path, cache-hit path, AND
reject-broad early-return path of query() — schema column-
aligned across all four returns.
- runner.ask() carries the same fields for `aborist ask` parity.
Phase 3 — policy fields + governance hash + CLI flags:
- 6 new policy fields, all default-on except
metacognition_block_on_contradiction (default False — label-
only by default; opt-in via --block-on-contradiction).
- All 6 folded into _VERIFIER_POLICY_FIELDS so flipping any
invalidates prior cache records on lookup.
- 2 new CLI flags on `aborist query`:
--no-preflight Level 2 master kill
--block-on-contradiction strict mode (hard-block on
lexical contradictions)
Phase 4 — audit-line labels + bench fields + tests:
- _render_warrant_tail extended with 5 metacog tail tokens:
· false premise
· contradictory
· stale risk
· out of corpus
· frame ambiguous
- Bench rows in qa_sweep.py gain 7 new bounded-size projection
fields (logical_statuses, question_shape, preflight_result,
temporal_sensitivity, has_false_premise, has_contradiction,
corpus_requirement). Full QuestionState stays on result dict
for CLI render only.
- tests/test_metacognition.py grew from 42 → 68 tests
(16 new: 6 governance + 6 audit-line tail + 4 default-policy
pinning).
Live verified end-to-end:
$ make query-dry Q="Who is the current CEO of OpenAI?" BURN=1
UNGROUNDED · via claim_lattice · stale risk
$ make query-dry Q="When did Mr. Burns become Homer's biological
father?" BURN=1
UNGROUNDED · via claim_lattice · false premise
978 tests passing; 36 skipped.
What's NOT shipped (deferred):
- Run-DAG node binding for metacognition_preflight stage —
joins ticket #000009 Phase 5 (same audit-replay gap; both
nodes can land together).
- Reference-frame plumbing — frame_detection runs post-retrieval,
preflight here is pre-retrieval; deferred until two-pass
or post-classification update lands.
- SOFT_PREFLIGHT_HINT (model-assisted sidecar) — source doc §18
reserves this label; hard rule preserved (no LLM in preflight
hard path).
- Bench A/B measuring preflight on vs off — quick to run once
stack settles.
Ticket #000010 status: closed · landed 2026-05-03.
Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):
intensity pre-fix post-fix
SINGULAR 61 (84%) 65 (89%)
MANY 4 ( 5%) 0 ( 0%) ← all 4 were `how many X?`
ALL 1 ( 1%) 1 ( 1%)
COMPREHENSIVE 1 ( 1%) 1 ( 1%)
OPEN_REQUEST 5 ( 7%) 5 ( 7%)
SMALL_NUM 1 ( 1%) 1 ( 1%)
Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).
Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.
9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).
Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:
name all members of the beatles
list all planets in the solar system
Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.
915 tests passing (9 new); 36 skipped.
Adds the §10.6 + §9.6 Phase-0.x bench-row fields. Pure additive —
no row-schema renames, no policy effects yet. Phase 1 classifier
will fill the quantifier_* slots; Phase 2 cap-table will fill
claim_cap_applied. Keeping the keys present here makes the JSONL
schema stable across the rollout so post-Phase-1 markdown can
re-render against pre-Phase-1 rows without column-misalignment.
New per-row fields:
- answer_pointer_count distinct E\d+ ids in raw_answer
- answer_chars_with_brackets chars inside [E\d+,...] regions
- raw_meaningful_line_count >20-char lines in raw_answer
(matches verifier FORMAT_COLLAPSED
denominator)
- quantifier_intensity slot for Phase 1 classifier
- quantifier_matched_token slot for Phase 1 classifier
- scope_bound_hint slot for Phase 1 classifier
- claim_cap_applied slot for Phase 2 cap-table
- model_profile_id configured model id verbatim
New helper _bracket_diagnostics() bundles the bracket/pointer/line
extraction in one place; module-level regexes (_BRACKET_RE,
_BRACKET_REGION_RE, _POINTER_ID_RE) avoid per-row recompilation.
5 new tests cover the helper: empty input, single pointer, multi-
pointer-in-one-bracket, separate brackets with shared id, format-
collapsed shape (5+ meaningful lines, 0 brackets — the
2026-05-02 winners-of-all-major-sports case).
Closes the bench-side gap surfaced in §5.2: JSONL was carrying summary
numbers only, blinding the harness to FORMAT_COLLAPSED rate and per-
violation-kind distributions. Without these, A/B/D bench measurements
on the broad-quantifier preflight guard would be guesses.
- query() result dict surfaces format_collapsed + raw_answer (lattice
modes only) so the bench can read them directly instead of re-deriving
from cache rows that --burn overwrites.
- Each bench row gains format_collapsed, violation_kinds (sorted unique
list — full payloads stay off the row to keep size bounded), and
answer_brackets (count of [E\d+] in raw_answer for lattice modes).
- _summarize aggregates per-mode FC count (only explicit True; None
means check didn't apply), per-kind tallies (each kind once per row),
and lattice-only bracket sum/n.
- Markdown renderer adds a `## format-collapse + violation kinds`
section with per-mode FC rate, mean raw brackets, and one column per
observed violation kind. Degrades gracefully when the sweep produces
no violations.
- 5 new bench-harness tests pin the aggregation rules.
Re-baseline (2026-05-02T20-58-57Z) sharpens §5.1 analysis dramatically:
NO_EVIDENCE_POINTER fires 3/3 in pointer mode and is the dominant gate,
not TITLE_MISMATCH (1/3) as §5.1 inferred from JSONL alone. FORMAT_
COLLAPSED actually fires 1/3 — not the rare corner the first baseline
called it. Implies Option B (prompt reminder) is the load-bearing fix
for the verdict; Option A (cap reduction) only moves secondary kinds.
§5.3 sub-investigation closed on first read — SCHEMA_INVALID:1 in
pointer mode is a legitimate kind emitted by verify_claim_lattice for
empty-claim-text (verify.py:1242) and bare-name-claim (verify.py:1270),
not a JSON-mode leak.
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.
Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.
verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict
Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
into verifier_policy_hash
- threaded through ask() and query() call sites
CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail
Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.
Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).
Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.
Re-aggregated bench/emergent_log.jsonl at 200 cycles. Distribution
holds (79.5% UNGROUNDED, 17.5% HYBRID, 3.0% STRICT). Six STRICTs
total; five defensible (known-truth or non-relationship grounding).
The sixth — `steer/reply/correcter` — is the FIRST CONFIRMED
EVIDENCE-WARRANTED false-positive in 200 cycles. Decompressed all
10 chunks of the cited document (Glossary of language teaching
terms and ideas, 33.5K chars) and confirmed the words `correcter`,
`steer`, and `reply` appear NOWHERE in the source. Yet STRICT 1/1
verified, no violations.
Mechanism: per-claim citation-coverage Rule 5 (≥30% claim tokens
in cited span) passed on GENERIC linguistic vocabulary (language,
communication, grammar, exchange, relationships) which appears in
any linguistics article. The question-distinctive tokens
(correcter, steer, reply) rode along unverified. PREMISE_PARROTING
shape predicted in original ticket §D, exhibit A.
Headline correction: from "zero false-positives" to "one confirmed
false-positive (0.5%)."
Action item: opens design space for #000008 — PREMISE_PARROTING
detector. Lexical signal: question-distinctive tokens absent from
cited chunk while Rule 5 still passes on generic vocabulary →
SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays
binary, stays lexical, folds into verifier_policy_hash.
The ticket file was at docs/ticket-000006-... at top-level; the
tickets/ subdir convention landed before #6 was opened. Move into
docs/tickets/ to match every other ticket.
Amend with the 62-entry post-ticket delta. bench/emergent_log.jsonl
now holds 134 cycles total. Distribution stable vs original 72:
106 UNGROUNDED (79.1%), 22 HYBRID (16.4%), 6 STRICT (4.5%). Ten
new grounded cases all carry appropriate violation labels — zero
EVIDENCE-WARRANTED false-positives across all 134 cycles. Original
tuning candidates: Tomas-deflection resolved by a1dd330,
xxviii-STRICT kept as designed, metaphor sidecar calibration
deferred (still rare).
Reframe ticket as a rolling research log. Emergent stress-testing
is an ongoing thread, not a one-shot defect-fix; future
bench-emergent re-runs append new ## Amend sections here, and any
code-level tunings open their own tickets linking back.
TICKETS.md row reflects the rolling-research framing. Stale
"## Next ID" footer scrubbed from the ticket body (TICKETS.md is
canonical for the next-id counter).
Pins the Ticket #000007 reproduction case as a permanent bench
fixture. Stresses the FTS5 hyphen-tokenization asymmetry where
query orthography (`bi-polar`) and indexed orthography (`Bipolar
disorder`) use different hyphen conventions for the same concept.
Pre-fix the medical-condition cluster never surfaced; post-fix
EVIDENCE-WARRANTED 2/2 grounded in `Bipolar disorder` + `Bipolar`
disambiguation.
72 random-word triangulation cycles ran on 2026-05-02. Aggregate:
- 54 UNGROUNDED (75%)
- 14 HYBRID (19%)
- 4 STRICT (6%)
- 18 grounded total (25%)
Catalogs five failure/success shapes the curated bench-qa fixture
set can't surface, with one-line per category. Key finding: the
verifier-ladder + soft-demote stack is doing its job — every
HYBRID demoted via TITLE_MISMATCH / DEFLECTION_DETECTED /
CITATION_MISMATCH / TOO_MANY_EVIDENCE_IDS appropriately. Zero
EVIDENCE-WARRANTED false-positives in the sample.
Tuning candidates queued for the parallel-agent work:
1. Tomas-pulls-transceivers case escaped DEFLECTION_DETECTED
(subject anchor buried mid-paragraph; consider last 2-3
content tokens vs just last 1)
2. Roman-numeral / archaic-word triplets — accept gracefully
(system already does)
3. metaphor_deflection sidecar (added in parallel work) only
fired 1×/72; calibration pass when more samples land
Ticket is OPEN and awaiting follow-up tunings; re-run
`make bench-emergent EMERGENT_N=50` after parallel work commits
land and amend with the delta.
Status table + Next ID bumped to 000007.
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
Emergent stress-test log surfaced two failure shapes the curated
bench can't reach. Reviewer (claude-opus-4-7 + fox) appended
teacher: blocks to four log entries; two of them named real
defects in the demote chain. This commit ships those fixes plus
the teacher-block annotations.
Cases reviewed (2026-05-02 emergent_log.jsonl):
✅ Menkar/sterilizers/trifle — working as designed
🟡 swallowtail/upbraided/rockiest — metaphorical deflection
🔴 cashback/widescreens/within — warrant gap on inferential claims
🔴 comeliness/fetish/investitures — topic shift earned ANCHOR-WARRANTED unfairly
Two code changes:
1) verify_claim_lattice + verify_claim_lattice_json: when EVERY
resolving claim has TITLE_MISMATCH, demote audit_mode →
UNGROUNDED. Previously HYBRID. Catches the cashback case
('widescreens offer cashback' cited to a generic Coupon
article — n_verified=1 overclaimed when the citation was
meaningless).
2) cli._ladder_rung_for_lattice: DEFLECTION_DETECTED moves from
soft-demote (cap at ANCHOR-WARRANTED) to hard-demote (cap at
POINTER-LINKED) alongside WARRANT_MISSING and TITLE_MISMATCH.
Catches the comeliness case (model fully shifted topic to
'Rock & Chips'; pre-fix the verified Rock & Chips claims
landed at ANCHOR-WARRANTED unfairly).
Test updates:
- tests/test_verify_json.py:test_verify_json_title_mismatch_demotes_to_ungrounded
(renamed; expected outcome flipped HYBRID → UNGROUNDED)
- 4 fixtures in tests/test_claim_lattice.py and 1 in
tests/test_verify_json.py: claim text augmented with 'in the
film' so the Jurassic-Park-titled fixture's title shares a
stem with the claim. These tests were testing other paths
(NO_EVIDENCE_POINTER, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID)
and the title-mismatch was incidental to the fixture choice.
Teacher blocks on the four log entries record the verdict +
failure_class + recommendation + notes, with reviewer + date.
The two ✅/🟡 entries have no recommended code change; the two 🔴
entries reference this commit as the fix.
751/34 tests pass.
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
Adds --resume <jsonl-path>: read the existing JSONL, build a set
of (question, mode, sample_idx) tasks already done, skip those
in the shuffled task list, and append fresh rows to the same
file. The markdown rollup uses the union of pre-existing + new
rows. Same --seed required for the order to align across the
resumed run.
Why: a 60-90 min full bench is annoying to re-run from zero when
something interrupts (network blip, kill, kernel panic). The
JSONL has been the durable artifact for ages; the bench just
didn't know how to read it.
Implementation:
- args.resume: Path | None
- if given: parse JSONL line-by-line, populate done_tasks set,
reuse the resume_path's stem as the run stamp, set jsonl_path
+ md_path to the resumed paths
- file_mode = 'a' if resuming else 'w'
- tasks filter: drop entries already in done_tasks
- rows initialized with existing_rows so the markdown summary
sees the union
- done counter starts at len(existing_rows) so the print
countdown reflects total progress
Empirical context: the c=6 bench killed at 142/639 left a
recoverable JSONL. Future kills can resume via:
.venv/bin/python bench/qa_sweep.py \
--resume bench/qa_results/<stamp>.jsonl \
--seed 0 --concurrency 4
(Same seed reproduces the shuffle so the remaining tasks come
out in the original order.)
Concurrency sweep on the smoke (15 tasks at each c):
c=3 102s (peak throughput)
c=4 106s (4% slower; chosen for full bench — more
forgiving on single-call hiccups)
c=5 119s (12% slower than c=3)
c=6 185s (45% slower; vLLM batching ceiling)
Lesson: more concurrent ≠ faster. vLLM continuous batching has
a sweet spot around c=3-4 for this endpoint.
Previously the bench scheduled at the (question, mode) cell level —
each cell ran its n samples sequentially before the next cell
started. With n=3 + concurrency=6, that meant 6 workers each
chained 3 sequential calls, vLLM's continuous batcher saw
correlated requests, and the n=3 variance signal was confounded
with 'what was vLLM doing in this same-cell batch.'
Two coupled defects, one fix:
(1) Statistical: samples-of-the-same-cell are NOT i.i.d. when run
back-to-back. Adjacent samples share vLLM batch composition,
KV-cache locality, and queue load. The 'variance' you see
across n=3 is partly that batch's particular weather, not true
model nondeterminism.
(2) Throughput: vLLM batches diverse requests well; correlated
requests fill the batch with similar work and starve.
Cell-grouped scheduling correlated by cache_key.
Refactor: build sample-level tasks ((q, mode, sample_idx)),
shuffle with deterministic seed (default 0), submit all to
ThreadPoolExecutor. Per-cell Lock dict (defaultdict(Lock))
serializes burn+insert against the shared cache_key — two samples
of the same cell that happen to land in adjacent worker slots
will queue on the cell's lock instead of racing. Lock contention
is rare under shuffle (samples of one cell are spread across
time) so throughput cost is near-zero.
New CLI flag --seed N for reproducible task ordering.
Changes:
- tasks list at sample granularity
- rng = random.Random(seed); rng.shuffle(tasks)
- cell_locks = defaultdict(Lock); held over _run_one + sample_idx assignment
- print line shows '#i/n' so order-of-completion is visible
Smoke at --n 1 --concurrency 6: clean, no errors, ~3 min for 15
tasks (essentially same wall-clock as cell-level scheduling at
n=1, as expected — the real win comes at n=3 where sample-level
concurrency unlocks parallelism within cells).
Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.
UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.
Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.
Touched:
aborist/qa/dag.py:_sha256_hex (the loud one)
aborist/qa/keys.py:_sha256
aborist/qa/evidence.py:_sha256_hex
aborist/store.py: chain_audit_events + append_audit
Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).
Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.
Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.
Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.
pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.
Bench-max scoreboard (predicted lift from this commit alone):
+1pp lattice modes (UTF fix)
+cycle-time enabler (smoke fixture, xdist)
no calibration cost — none of the verifier checks moved.
zstandard.ZstdCompressor and ZstdDecompressor instances each carry an
internal libzstd context that is NOT thread-safe. Calling .compress()
or .decompress() on a single shared instance from multiple threads
corrupts the context and raises:
ZstdError: decompression error: Data corruption detected
Surfaced when bench/qa_sweep.py learned a --concurrency flag and ran
4 (question, mode) cells in parallel. ~19% of retrievals failed on
zstd corruption before the fix. The prior comment claiming the
singletons were 'stateless across calls — safe to share across
threads' was wrong.
Replace the module-level singletons with threading.local() caches.
Each thread reuses its own ZstdCompressor / ZstdDecompressor; no
contention across threads. Init cost is negligible vs decompression.
bench/qa_sweep.py:
--concurrency N (default 1) parallelizes (question, mode) CELLS
using a ThreadPoolExecutor. Samples within a cell stay sequential
so burn-then-insert against a single cache_key never races itself.
Lock-protected JSONL writes & progress prints. Exception in any
worker surfaces via fut.result().
vLLM handles concurrent requests well; 4-8 is a reasonable starting
point. With --n 2 --concurrency 4, expect ~10 min wall-clock for the
full 426-run sweep against hermes.ai.unturf.com (vs ~2h sequential).
Two paired changes addressing fox's "learn this hyperparameter
from model use, not hard coding" + the 1M-context-window
caveat:
(1) Log-scale prompt-size buckets extend from 8KB through 1M+:
<8KB / 8-16KB / 16-32KB / 32-64KB / 64-128KB / 128-256KB /
256-512KB / 512K-1M / >=1M
The same bench harness now covers 8B-class models (Hermes 82K
context, max useful prompt ~32-64KB) through 1M-context models
(Gemini 1.5 Pro, Claude with extended context, Llama 4) without
code change. A model whose context window stops at 82K simply
never populates the giant buckets; a 1M-context model fills
them and finds its own sweet spot.
(2) New "recommended context budget (learned from this bench)"
section — per-mode peak-strict-rate bucket. Operator-driven
landing per the five-step algorithm step 5: surfaced, not
auto-applied. Minimum sample size of 5 runs per bucket so
statistical noise doesn't masquerade as signal. Tie-break on
smaller-bucket-wins so equivalent strict-rates favor the
cheaper choice.
The substrate is now self-tuning at the OBSERVATION layer: bench
records what budget actually grades best per model. Per-model
profile JSON (storing the recommended budget back into
~/.aborist/model_profiles/<model>.json) is the next beat once
this surface is observable in real bench runs.
3 new bench tests cover the recommended-budget section, the
sample-size floor, and the giant-context bucket coverage.
Full suite: 746 passed (was 738, +8).
Connects to D8 (automate after test-pinning): the bench tells us;
we don't guess.
Wires the seven-point program (docs/seven-point-program.md) into
the bench harness as a per-mode coverage table.
Per-row computation in _directive_compliance(answer_mode, result, err):
D2_pointer_clauses — answer_mode is lattice variant
D3_cti_substrate_ready — lattice mode + run_dag_root populated
(full coverage pending ticket #000002)
D4_evidence_map_bound — run_dag_root populated
(retrieval-plan binding pending #000001)
D6_warrant_fired — warrant ran (lattice mode today;
per-shape gating pending #000003)
D7_honest_label — audit_mode in canonical enum
(renderer transformation pinned in
test_cli_render.py)
D1 (no LLM in verifier), D5 (verifier_method enum), D8 (test-pinning
discipline) are global properties of the substrate and don't appear
per-row — they get tracked once in CLAUDE.md / the program doc.
_summarize aggregates per-row booleans into per-mode pass counts.
_render_markdown emits a 'directive coverage (seven-point program)'
section with per-mode per-directive coverage as 'count/N (pct%)'.
Discipline rationale per CLAUDE.md's five-step algorithm step 5:
the bench harness IS the automation substrate, so it must enforce
the directives before any feature gets layered on top. Bench scores
that climb without directive coverage climbing are graveyard-digging
(step 4: don't dig the grave faster).
5 new bench tests cover: per-mode pass aggregation, the markdown
section presence and content, the helper's behavior on quote mode
(D2 fail by design), lattice mode (all directives pass), and error
rows (empty dict — no signal). Full suite 669 passed.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Per-call accounting of prompt capacity so we can answer 'does
context size affect strict-rate?' at aggregate scale.
Per-call result dict gains:
"prompt_chars": {
"system_prompt": <int>, # system message chars
"grounding_reminder": <int>, # restated rule message (if any)
"user_question": <int>, # the original question text
"evidence_or_context":<int>, # rendered evidence map / wikitext context
"messages_total": <int>, # sum of all messages[].content
}
"answer_chars": <int> # rendered answer length
Char-level for now: model-agnostic, fast, and a workable proxy
for prompt tokens (~4 chars/token English prose, ~2.5 for JSON-
evidence-block heavy contexts). Token-exact accounting can layer
on later if a tokenizer is cheap enough to wire in per-call.
Surfaces in three places:
(1) result["prompt_chars"] — for programmatic callers. Cache hits
return prompt_chars too (computed before lookup).
(2) `aborist query` human render — one-liner showing the breakdown
and the answer length, so an operator can tell at a glance
whether STRICT came from a 5KB tight prompt or a 60KB stuffed
context.
(3) bench/qa_sweep.py — per-row prompt_chars_{total,evidence,system,
question} + answer_chars columns in the JSONL, plus a new
"strict-rate by prompt size" section in the markdown summary
bucketing runs by capacity (<8KB, 8-16KB, 16-32KB, 32-64KB,
>=64KB) so the bench reveals whether input size correlates
with verdict quality across modes.
Never enters cache_key — runtime measurements, not policy.
Two paired changes:
(1) Retrieval template-phrase stopwords. Bench finding 2026-04-30:
'tell me all there is to know about york england?' missed the
en.wikipedia.org/wiki/York article entirely — corpus has it but the
FTS5 AND-mode MATCH expanded to "all" AND "there" AND "know" AND
"york" AND "england", which filters to only docs containing every
template word. Articles like "Charles I of England" happen to mention
"all" + "there" + "know" + "england" enough to win; the bare "York"
stub doesn't, so it falls out of the top-K.
Added `all there know everything anything something` to both
_FTS5_STOPWORDS (search/fts5.py) and _TITLE_STOPWORDS (qa/query.py)
so they stay in sync. After the change "York" article ranks #1 on
both 'tell me all there is to know about' and 'tell me everything
about' phrasings.
These are retrieval-time stopwords only — they don't enter the
cache_key or governance_policy_hash (cache_key hashes the original
question text via question_hash). Existing cached records remain
valid; new queries get better retrieval.
(2) bench/qa_sweep.py: per-row deflection_kind + subject_anchor +
subject_in_answer (from qa.inspect.diagnose_deflection); summary
table grows a 'deflections' column tracking STRICT/HYBRID rows
where the question's subject anchor is missing from the answer.
Surfaces the Mars-BDFL pattern at aggregate scale.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Question-set churn (bench/qa_questions.txt):
- narrow factoid: drop 'who founded apple computer?'; add
'a bridge between new london & groton?' (single-answer factoid
with named-entity grounding via Gold Star Memorial Bridge)
- broad descriptive: drop 'tell me about the apollo program'
- entity list: drop 'list the founders of microsoft'; add
'name simpsons family members including pets?',
'list of obelisk in connecticut'
- leading/adversarial: add 'whales are endangered from over hunting'
(declarative-shape, tests verifier under premise-seeking phrasing)
- out-of-corpus: rephrase Mars no-such-entity question to
'who is a benevolent dictator for life for mars?' — sharper
adversarial shape since BDFL is a real OSS-governance term
the model is tempted to project onto Mars
- minor edits: 'GNU linux', 'C programming language', 'method man',
'cold fusion breakthrough' phrasing
tests/test_qa_quality_live.py: rename test_mars_prime_minister_*
to test_mars_benevolent_dictator_*; refresh refutation markers
to match the new BDFL phrasing; expand to cover 'no one' /
'no person' negations the model uses on this shape.
Adds _repair_truncated_json() to recover from mid-output truncation
where Hermes-3-8B writes one long claim text and runs out of
max_tokens before closing the structure. Pre-fix the lenient parser
saw something like:
{"claims":[{"text":"The Apollo program was the United States
spaceflight effort which landed... <CUT>
with no closing `"`, no closing `}`, no closing `]`, no closing
outer `}`. Couldn't parse, lost the whole answer.
Post-fix the repair walks the input once tracking string state
and bracket stack:
- close any open string (append `"`)
- drop dangling structural artifacts:
`"key":` (partial key with colon)
`"key"` after `,` (partial key without colon)
trailing `,` before close
- close the bracket stack in reverse (`{` → `}`, `[` → `]`)
Conservative: never inserts content (no key names, no values, no
commas) — only closes / drops. Worst case the repair is a no-op
or makes parsing fail in a different way; never silently
fabricates data.
Apollo live test result: UNGROUNDED 0/0 (lost answer) →
UNGROUNDED 0/1 (partial first claim recovered with evidence_ids
intact, partial second claim dropped). The user sees the model's
attempted answer in the unverified bucket instead of empty space.
The audit chain captures the repair via fixups labels
(close_string, close_brace, close_bracket, drop_partial_key,
strip_trailing_comma) so an auditor can see what the parser had
to peel.
Pipeline ordering in _lenient_json_parse: strict parse first,
then peel (fence/prose-trim/curly-quotes/trailing-comma), then
strict parse again, then truncation repair as last resort. Each
step's fixups append to the same list so the verdict's
`json_fixups` field is comprehensive.
Bench question set picked up "name simpsons family members
including pets?" — a multi-part list with adjacent-relationship
shape, addresses a gap in the corpus probe.
460 unit tests pass; 12 skipped (live fixtures unchanged).
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Adds bench/qa_sweep.py + bench/qa_questions.txt + `make bench-qa` target.
Sweeps a fixed question set through quote and claim_lattice_pointer modes,
N samples per cell (default 3), each sample burns the cached record so
Hermes nondeterminism becomes the variance source.
Outputs:
bench/qa_results/<utc>.jsonl one row per (mode, question, sample)
bench/qa_results/<utc>.md summary + per-question vote counts
Question set spans the failure-mode shapes the verifier needs to handle:
narrow factoid, broad descriptive ("tell me about X"), entity list,
relationship, and out-of-corpus (should land UNGROUNDED honestly).
Methodology gap this closes: at n=1 the strict-rate swings ±20pp on a
fixed prompt purely from sampling noise. n=3 makes 5pp deltas legible.
aborist/progress.py — stdlib-only rate-limited stderr reporter.
Periodic lines (default every 2s) showing:
elapsed | seen | inserted | docs/s now | docs/s avg | percent | ETA
Wired into ingest_source via a `progress` parameter; CLI default-on
with --quiet to suppress and --total-estimate N to enable percent/ETA.
Demo on a 5k-doc ingest with --progress-interval 0.8:
[ 1s] | 200 seen | 200 new | 193 now | ( 193 avg) docs/s | 4.0% | ETA 24s
[ 7s] | 3,200 seen | 3,200 new | 458 now | ( 424 avg) docs/s | 64.0% | ETA 3s
[ 12s] | 5,001 seen | 5,000 new | 392 now | ( 413 avg) docs/s | 100.0% | ETA 0s
bench/run.sh + `make bench` — reproducible 5000-doc workload through
three configs:
serial single process, single SQLite
parallel-shared N shards, one shared SQLite (WAL serialized)
attached N shards, per-shard SQLite (true parallel writes)
Output is a one-shot table plus CSV at /tmp/aborist-bench/results.csv
so the ratchet is visible as we keep optimizing. Override via
BENCH_DOCS=N and SHARDS=N.
Latest baseline (this commit, on this machine):
config wall_s docs docs/s
serial 11.74 5000 425.7
parallel-shared 10.39 5000 481.1
attached 7.99 5000 625.5
53 tests still passing.
