CLI flags on `aborist query`:
--no-quantifier-guard Level 2 disable: kills the guard for
one call. Telemetry → None.
--allow-broad Emergent-search: classifier on, caps
off. For exploratory enumeration.
--reject-broad Strict reject: ALL/COMPREHENSIVE/
OPEN_REQUEST + scope_bound_hint==
"unbounded" returns UNGROUNDED before
the LLM call (saves ~10-15s). Bounded
universals (Beatles, year-anchored)
are NOT rejected per §10.1.
--apply-quantifier-caps Flip Phase 2 dry-run gate per-call.
Path from dry-run to live cap.
Three new soft-demote violation kinds (§10.3) — no new audit_mode
token; tails on the existing audit-line:
BROAD_QUANTIFIER_RUNAWAY "broad runaway"
BROAD_QUANTIFIER_CAP_APPLIED "broad cap N" (cap value rendered)
BROAD_QUANTIFIER_SCOPE_UNBOUND "broad unbounded"
All three cap the ladder at ANCHOR-WARRANTED. Plus one HARD demote
(early-return UNGROUNDED):
BROAD_QUANTIFIER_REJECTED "broad rejected" (preflight rejection)
The reject-broad path early-returns from query() before the LLM
call when policy enables quantifier_reject_broad AND the question
is broad-unbounded. Result schema mirrors a normal UNGROUNDED row
(answer_text carries the rejection rationale + actionable narrowing
hints). _render_query_human gets a dedicated branch for the new
status so operators see the rejection without --json.
Live verification (post-commit):
$ aborist query --reject-broad "Winners of all major sports?"
UNGROUNDED · via BROAD_QUANTIFIER_REJECTED · ALL ("all") · cap was 8
0/0 0.0s (preflight)
BROAD-QUANTIFIER PREFLIGHT REJECTED · scope unbounded
Question matched ALL intensity ("all") with an under-specified
universe. Narrow ... or run with --allow-broad for exploratory
enumeration.
$ aborist query --reject-broad "name all members of the Beatles"
UNGROUNDED · via claim_lattice · title mismatch 4/4 20.9s
[Beatles enumerated, scope_bound_hint=bounded → not rejected]
`quantifier_reject_broad` folded into _VERIFIER_POLICY_FIELDS so
flipping reject default invalidates prior cache records.
16 new tests cover: soft-demote registration, hard-demote NOT in
soft-demote set, ladder rung mapping for each kind, tail rendering
(including cap value interpolation), tail combination with
existing kinds, end-to-end render through _render_query_human,
governance-hash binding. Two skipped placeholders mark the
integration paths exercised by live bench.
Lands aborist/qa/quantifier_reminder.py with broad_quantifier_
reminder(): one-line user-turn message restating the cap and the
[E\d+] citation rule for broad-intensity questions. Two templates:
bounded universe:
"This is a broad-quantifier query with a bounded universe.
Return at most N pointer-linked claim lines. Each claim must
cite an evidence id like [E5]; do not write claim lines
without bracket citations."
unbounded universe:
"This is a broad-quantifier query with an under-specified scope.
Return at most N pointer-linked claim lines. If you cannot
ground N claims with evidence IDs, return fewer grounded
claims. Do not enumerate from training prior. Each claim must
cite an evidence id like [E5]; do not write claim lines
without bracket citations."
The bounded template skips the "do not enumerate from training
prior" clause — the corpus has the answer set. Unknown scope falls
through to the stricter unbounded template (over-warn rather than
under-warn).
Wired into both query() and runner.ask() at the same insertion
point as the existing grounding_reminder — between
grounding_reminder and the evidence/question payload, where
Hermes-3-8B's most-recent-token attention catches it.
Default OFF (`quantifier_reminder_enabled: false`). Empirical
justification: ticket §3 Option B con notes Hermes already ignores
parts of the existing reminder under enumeration pressure. The
mechanism lands so an operator can A/B test cap-only vs cap+reminder
without code changes; default flips on after bench shows ≥5pp delta
on FORMAT_COLLAPSED or pointer-loss rate per §10.8.
`quantifier_reminder_enabled` folded into _VERIFIER_POLICY_FIELDS
so flipping the switch invalidates prior cache records.
19 new tests cover: gating (non-broad → None, missing cap → None,
None intensity → None), bounded-vs-unbounded template selection,
unknown scope falls back to unbounded, cap interpolation,
[E\d+] citation rule restatement, governance-hash invalidation.
Lands aborist/qa/model_profiles.py with two profiles:
- adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic
tight caps for broad intensities (ALL=8, COMPREHENSIVE=5,
OPEN_REQUEST=5) reflecting the 2026-05-02 runaway case.
- default
large-reasoner-class fallback (ALL=12, COMPREHENSIVE=15,
OPEN_REQUEST=12). Used when model_profile_id has no entry.
cap_for_intensity() resolves caps via three-source chain:
1. policy_overrides (per-call dict, highest priority)
2. per-model profile from PROFILES
3. "default" profile fallback
EXPLICIT_COUNT sentinel handles SMALL_NUM_EXPLICIT and
COMPARATIVE_BOUND — cap is the question's explicit count, not a
profile-set value. Defensive fallback to MANY cap if classifier
fired the rung without extracting a count.
Four new policy fields, all folded into governance_policy_hash via
_VERIFIER_POLICY_FIELDS:
- quantifier_guard_enabled master kill (default True)
- quantifier_guard_apply_caps dry-run gate (default False per
§10.11.3 — cap LOOKED UP and reported
on result, but NOT applied to the
verifier until operator flips True)
- quantifier_caps_by_intensity per-call override dict
- quantifier_guard_modes per-mode opt-in list (default
["claim_lattice_pointer",
"claim_lattice"]; quote opts out)
Six-level disable hierarchy (§10.11.2) implemented:
- Per-test: policy={"quantifier_guard_enabled": False}
- Per-call: --no-quantifier-guard (Phase 4)
- Per-phase: each policy switch is independent
- Per-mode: quantifier_guard_modes filter
- Per-model: model_profiles.py lookup
- Master: governance_policy_hash invalidation on flip
Wired through both query() and runner.ask() — both compute
effective_max_claims from the (classifier_intensity, model_profile,
policy_overrides) triple and pass it as max_claims_per_answer to
the verifier. Dry-run mode keeps effective_max_claims at the policy
default (12) until apply_caps flips True.
Result dict surfaces claim_cap_applied (the LOOKED-UP cap, even in
dry-run) plus all Phase-1 quantifier fields on miss-path AND
cache-hit path so bench rows stay column-aligned.
19 new tests pin: per-model selection, EXPLICIT_COUNT sentinel,
override precedence, governance-hash invalidation on every cap
field, profile shape (all ten rungs covered), default profile
presence.
Lands aborist/qa/quantifier.py with classify_question_quantifier(),
a pure function mapping a question string onto the ten-rung
intensity ladder (ticket #000008 §2):
ABSENT < SINGULAR < PROPORTIONAL < SMALL_NUM_EXPLICIT
< COMPARATIVE_BOUND < FEW < MANY < ABSENT < ALL
< OPEN_REQUEST < COMPREHENSIVE
Returns intensity, matched_token, explicit_count, is_broad,
operational_shape, scope_bound_hint, classifier_version. Pure: no
I/O, no model call, no retrieval call.
Highest-intensity-wins arbitration: COMPREHENSIVE strictly stronger
than OPEN_REQUEST (both > ALL). Catches "tell me everything about
all wars" → COMPREHENSIVE rather than dropping to one of the softer
shape detectors.
Scope_bound_hint heuristic (§10.1): bounded vs unbounded universals.
"All members of the Beatles" → bounded (corpus-known finite set).
"Winners of all major sports" → unbounded (scope undefined). Year-
anchored questions ("…in 2024") bound the universe to one event.
Heuristic only — corpus-arity check left for future refinement.
Wired into query() right after policy resolution. Both miss and
cache-hit paths surface quantifier_intensity, quantifier_matched
_token, scope_bound_hint, quantifier_explicit_count on the result
dict. claim_cap_applied is None until Phase 2 lands the cap-
application gate (default-off per §10.11.3 dry-run discipline).
61 new tests cover every rung, scope-bound detection (bounded /
unbounded / unknown), highest-wins arbitration, and regression
fixtures (factoid/wh-questions don't over-classify as broad).
Adds the §10.6 + §9.6 Phase-0.x bench-row fields. Pure additive —
no row-schema renames, no policy effects yet. Phase 1 classifier
will fill the quantifier_* slots; Phase 2 cap-table will fill
claim_cap_applied. Keeping the keys present here makes the JSONL
schema stable across the rollout so post-Phase-1 markdown can
re-render against pre-Phase-1 rows without column-misalignment.
New per-row fields:
- answer_pointer_count distinct E\d+ ids in raw_answer
- answer_chars_with_brackets chars inside [E\d+,...] regions
- raw_meaningful_line_count >20-char lines in raw_answer
(matches verifier FORMAT_COLLAPSED
denominator)
- quantifier_intensity slot for Phase 1 classifier
- quantifier_matched_token slot for Phase 1 classifier
- scope_bound_hint slot for Phase 1 classifier
- claim_cap_applied slot for Phase 2 cap-table
- model_profile_id configured model id verbatim
New helper _bracket_diagnostics() bundles the bracket/pointer/line
extraction in one place; module-level regexes (_BRACKET_RE,
_BRACKET_REGION_RE, _POINTER_ID_RE) avoid per-row recompilation.
5 new tests cover the helper: empty input, single pointer, multi-
pointer-in-one-bracket, separate brackets with shared id, format-
collapsed shape (5+ meaningful lines, 0 brackets — the
2026-05-02 winners-of-all-major-sports case).
Closes the bench-side gap surfaced in §5.2: JSONL was carrying summary
numbers only, blinding the harness to FORMAT_COLLAPSED rate and per-
violation-kind distributions. Without these, A/B/D bench measurements
on the broad-quantifier preflight guard would be guesses.
- query() result dict surfaces format_collapsed + raw_answer (lattice
modes only) so the bench can read them directly instead of re-deriving
from cache rows that --burn overwrites.
- Each bench row gains format_collapsed, violation_kinds (sorted unique
list — full payloads stay off the row to keep size bounded), and
answer_brackets (count of [E\d+] in raw_answer for lattice modes).
- _summarize aggregates per-mode FC count (only explicit True; None
means check didn't apply), per-kind tallies (each kind once per row),
and lattice-only bracket sum/n.
- Markdown renderer adds a `## format-collapse + violation kinds`
section with per-mode FC rate, mean raw brackets, and one column per
observed violation kind. Degrades gracefully when the sweep produces
no violations.
- 5 new bench-harness tests pin the aggregation rules.
Re-baseline (2026-05-02T20-58-57Z) sharpens §5.1 analysis dramatically:
NO_EVIDENCE_POINTER fires 3/3 in pointer mode and is the dominant gate,
not TITLE_MISMATCH (1/3) as §5.1 inferred from JSONL alone. FORMAT_
COLLAPSED actually fires 1/3 — not the rare corner the first baseline
called it. Implies Option B (prompt reminder) is the load-bearing fix
for the verdict; Option A (cap reduction) only moves secondary kinds.
§5.3 sub-investigation closed on first read — SCHEMA_INVALID:1 in
pointer mode is a legitimate kind emitted by verify_claim_lattice for
empty-claim-text (verify.py:1242) and bare-name-claim (verify.py:1270),
not a JSON-mode leak.
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.
Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.
verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict
Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
into verifier_policy_hash
- threaded through ask() and query() call sites
CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail
Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.
Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).
Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.
Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.
New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.
Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
gain `subject_tokens_absent_threshold` kwarg + per-claim check
block (mirrors TITLE_MISMATCH plumbing, sits right after it
in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
`verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value
Live verification (cache-split cleanly via policy-hash bump):
pre-fix cache_key 08dbd2c1… : STRICT (false positive)
post-fix cache_key 6a519636… : UNGROUNDED
Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT
Full suite: 776 passed, 34 skipped.
aborist now writes one JSONL session per `make query` invocation
and per `bench-emergent` cycle to:
~/.aborist/unfirehose/{project-slug}/{session-uuid}.jsonl
Unfirehose's native-harness auto-discovery picks up any
~/.{name}/unfirehose/ directory (see ingest.ts:discoverNativeHarnesses)
without registration — once a session lands, the unfirehose watcher
debounces, ingests, and exposes it in the dashboard alongside
Claude Code / Fetch / uncloseai sessions.
Schema: unfirehose/1.0 (per ~/git/unfirehose-nextjs-logger/docs/
unfirehose-schema.md). Each session file:
line 1 type=session (header — id, projectId, firstPrompt,
harness="aborist", harnessVersion)
line 2 type=message role=user
line 3 type=message role=assistant
content=[text]
model=hermes-3-llama-3.1-8b-fp8-dynamic
provider=hermes
durationMs=<wall>
aborist_meta={audit_mode, n_verified/n_quotes,
cache_key, cache_status, lookup_path,
violations, sources, timings_ms, answer_mode}
line 4 type=message role=system subtype=session_end durationMs
aborist-specific extras (verifier verdict, sources, timings) ride
under namespaced ``aborist_meta`` so the canonical fields stay clean
for off-the-shelf consumers; per the spec, unknown fields are
ignored downstream.
Bench-emergent cycles emit an additional system init message at
the start of each session noting the 3 random words, marking the
session as a generator-driven cycle vs a normal user query.
Failure-isolation: journal write is wrapped in a broad try/except
at every call site. A journaling bug must NEVER break the query
or bench loop.
Tests: 10 new in tests/test_journal.py (slug encoding, session
header, parent-id chain, session_end on close, aborist_meta
passthrough, usage block, idempotent close). Full suite: 663 passed.
Live verified: `make query Q="what is photosynthesis?"` produced
a 4-line JSONL with STRICT 3/3, all sources + timings populated,
ready for unfirehose ingestion.
Re-aggregated bench/emergent_log.jsonl at 200 cycles. Distribution
holds (79.5% UNGROUNDED, 17.5% HYBRID, 3.0% STRICT). Six STRICTs
total; five defensible (known-truth or non-relationship grounding).
The sixth — `steer/reply/correcter` — is the FIRST CONFIRMED
EVIDENCE-WARRANTED false-positive in 200 cycles. Decompressed all
10 chunks of the cited document (Glossary of language teaching
terms and ideas, 33.5K chars) and confirmed the words `correcter`,
`steer`, and `reply` appear NOWHERE in the source. Yet STRICT 1/1
verified, no violations.
Mechanism: per-claim citation-coverage Rule 5 (≥30% claim tokens
in cited span) passed on GENERIC linguistic vocabulary (language,
communication, grammar, exchange, relationships) which appears in
any linguistics article. The question-distinctive tokens
(correcter, steer, reply) rode along unverified. PREMISE_PARROTING
shape predicted in original ticket §D, exhibit A.
Headline correction: from "zero false-positives" to "one confirmed
false-positive (0.5%)."
Action item: opens design space for #000008 — PREMISE_PARROTING
detector. Lexical signal: question-distinctive tokens absent from
cited chunk while Rule 5 still passes on generic vocabulary →
SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays
binary, stays lexical, folds into verifier_policy_hash.
The ticket file was at docs/ticket-000006-... at top-level; the
tickets/ subdir convention landed before #6 was opened. Move into
docs/tickets/ to match every other ticket.
Amend with the 62-entry post-ticket delta. bench/emergent_log.jsonl
now holds 134 cycles total. Distribution stable vs original 72:
106 UNGROUNDED (79.1%), 22 HYBRID (16.4%), 6 STRICT (4.5%). Ten
new grounded cases all carry appropriate violation labels — zero
EVIDENCE-WARRANTED false-positives across all 134 cycles. Original
tuning candidates: Tomas-deflection resolved by a1dd330,
xxviii-STRICT kept as designed, metaphor sidecar calibration
deferred (still rare).
Reframe ticket as a rolling research log. Emergent stress-testing
is an ongoing thread, not a one-shot defect-fix; future
bench-emergent re-runs append new ## Amend sections here, and any
code-level tunings open their own tickets linking back.
TICKETS.md row reflects the rolling-research framing. Stale
"## Next ID" footer scrubbed from the ticket body (TICKETS.md is
canonical for the next-id counter).
Three changes that shape the same lever:
(1) The metaphor-cue wordlist now unions /usr/share/dict/words +
/usr/share/dict/american-english + /usr/share/dict/british-english.
The Debian split made the prior 'just symlink to american-english'
miss British spellings (colour, organisation, realise) which
silently became false negatives on British-speaker questions.
Union: 102,485 → 104,305 entries on this machine. ~1,820 added
British-specific entries.
(2) Supplemental dictionary support: operators can layer
domain-specific vocabulary into the morphological substrate.
Two paths:
- Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b
- Programmatic: register_metaphor_dictionary(path)
Each supplemental dict is one word per line. The cue suffix
tests (-ly stem, -ing stem, -est stem) then resolve domain
stems automatically — adding 'aerodynamic' to a custom dict
makes 'aerodynamically' classify as adverbial without code
changes.
Use case: 'a tree with its own vocabulary' — an aviation
forest, a medical corpus, a legal-domain shard each carries
jargon the standard wordlist doesn't cover. Register once,
suffix tests pick up domain stems forever.
(3) README gains a 'Sidecar diagnostics' section with a table of
the three sidecars (deflection, title-relevance, metaphor-
deflection) plus a 'Metaphor-deflection cue dictionary' subsection
explaining the derivation rule, the load order, and the per-
forest vocabulary configurability. Architecturally documents
why the rule is *derived* from the union (Phase-2 lesson) and
not hand-curated.
3 new tests in tests/test_inspect.py:
- register_metaphor_dictionary unions a custom path's words
- ABORIST_METAPHOR_DICTS env var supplements with two paths
- re-registering same path is idempotent
763/34 tests pass.
Pins the Ticket #000007 reproduction case as a permanent bench
fixture. Stresses the FTS5 hyphen-tokenization asymmetry where
query orthography (`bi-polar`) and indexed orthography (`Bipolar
disorder`) use different hyphen conventions for the same concept.
Pre-fix the medical-condition cluster never surfaced; post-fix
EVIDENCE-WARRANTED 2/2 grounded in `Bipolar disorder` + `Bipolar`
disambiguation.
Closes the FTS5 hyphen-tokenization asymmetry: `bi-polar is rare?`
retrieved only the Bi-Polar album/disambiguation cluster while the
medical-condition cluster (Bipolar disorder, Bipolar I/II disorder,
etc.) sat in the same shards untouched. `unicode61` splits hyphens
at index AND query time; `Bi-Polar Blues` indexes as [bi, polar, ...]
while `Bipolar disorder` indexes as [bipolar] — non-overlapping
token sets that never met.
Fix is query-layer only — no canonicalization_version bump, no
re-index, existing cache_keys stay valid:
- _hyphen_fold_variants(s): emit joined-no-hyphen variants for
every hyphenated run.
- _title_query_tokens(s): additively merges variants symmetrically
(queries AND titles when called on either).
- _filter_by_title_relevance: accept-path 5 — title stem-overlap
with hyphen-fold anchors passes the breadth gate. Rescues
`Bipolar disorder` (1-of-N qtoken match) without disrupting
non-hyphen queries (anchors empty → zero side effect).
- DEFAULT_QUERY_POLICY / DEFAULT_POLICY: hyphen_fold_v1: True
marker folds into governance_policy_hash; new records
cache-split cleanly from pre-fold records.
Live verification on /home/fox/.aborist/shards: same query now
retrieves `Bipolar disorder` (#5) and `Bipolar` disambiguation
(#7); model cites both, answer reads "Bi-polar disorder is not
rare; it affects approximately 2.8% of the U.S. population".
EVIDENCE-WARRANTED 2/2, properly grounded.
Tests: 4 new (3 unit, 1 integration with regression-pinned
Bipolar-disorder retrieval). Full suite 760 passed, 34 skipped.
Also: CLAUDE.md gains a close-when-complete hint for tickets — an
open ticket whose code already shipped is a stale map.
72 random-word triangulation cycles ran on 2026-05-02. Aggregate:
- 54 UNGROUNDED (75%)
- 14 HYBRID (19%)
- 4 STRICT (6%)
- 18 grounded total (25%)
Catalogs five failure/success shapes the curated bench-qa fixture
set can't surface, with one-line per category. Key finding: the
verifier-ladder + soft-demote stack is doing its job — every
HYBRID demoted via TITLE_MISMATCH / DEFLECTION_DETECTED /
CITATION_MISMATCH / TOO_MANY_EVIDENCE_IDS appropriately. Zero
EVIDENCE-WARRANTED false-positives in the sample.
Tuning candidates queued for the parallel-agent work:
1. Tomas-pulls-transceivers case escaped DEFLECTION_DETECTED
(subject anchor buried mid-paragraph; consider last 2-3
content tokens vs just last 1)
2. Roman-numeral / archaic-word triplets — accept gracefully
(system already does)
3. metaphor_deflection sidecar (added in parallel work) only
fired 1×/72; calibration pass when more samples land
Ticket is OPEN and awaiting follow-up tunings; re-run
`make bench-emergent EMERGENT_N=50` after parallel work commits
land and amend with the delta.
Status table + Next ID bumped to 000007.
Phase-2 lesson applied. The previous metaphor-deflection sidecar
shipped two hand-curated frozensets (_METAPHOR_LY_STOPWORDS,
_METAPHOR_ING_BLACKLIST) listing English -ly nouns and common -ing
verb forms. Same trap as concept_relations Phase 1: hand-curating
exception lists for English suffix patterns has an infinite tail
(every newly-encountered -ly noun earns one more git commit).
Replace with derivation. The system English wordlist
/usr/share/dict/words is already on the bench-emergent picker's
critical path; reuse it as the morphological substrate.
Cue rule (purely lexical, derived):
- -ly word is adverbial iff stem (or stem+e, or stem-i+y) is in
the wordlist:
gracefully → graceful ✓
truly → true ✓
happily → happy ✓
butterfly → butterf ✗ (noun, filtered automatically)
italy → ital ✗ (proper noun)
family → famil ✗ (noun)
- -ing word is participle iff stem (or stem+e, or de-doubled
consonant stem) is a verb in the wordlist:
fluttering → flutter ✓
running → run ✓ (consonant de-doubled)
making → make ✓
sterling → sterl ✗ (noun, filtered)
during → dur ✗ (preposition, filtered)
- -est word is superlative iff stem (or stem+e, or stem-i+y) is
an adjective in the wordlist:
rockiest → rocky ✓
longest → long ✓
safest → safe ✓
The closed-class prepositional cue set stays hand-listed (~10
words) — these are a finite grammatical class, not a vocabulary
tail. _METAPHOR_LY_STOPWORDS and _METAPHOR_ING_BLACKLIST deleted
entirely.
Graceful degradation: if /usr/share/dict/words doesn't exist
(BSD, some containers, Windows), the wordlist function returns
empty; suffix tests all return False; the sidecar quietly returns
no_signal. The closed prepositional set still works.
Verified on swallowtail canary: 5 cues found (amidst, fluttering,
gracefully, rockiest, upbraiding), threshold met, fires
metaphor_deflection.
Minor residual noise (holly → 'hol' in dict; interest → 'inter'
in dict) accepted as soft-signal cost. The signal is read by
human reviewers; over-flagging at the +1-cue level rarely
crosses the 3-cue threshold for a real question.
756/34 tests pass.
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
Emergent stress-test log surfaced two failure shapes the curated
bench can't reach. Reviewer (claude-opus-4-7 + fox) appended
teacher: blocks to four log entries; two of them named real
defects in the demote chain. This commit ships those fixes plus
the teacher-block annotations.
Cases reviewed (2026-05-02 emergent_log.jsonl):
✅ Menkar/sterilizers/trifle — working as designed
🟡 swallowtail/upbraided/rockiest — metaphorical deflection
🔴 cashback/widescreens/within — warrant gap on inferential claims
🔴 comeliness/fetish/investitures — topic shift earned ANCHOR-WARRANTED unfairly
Two code changes:
1) verify_claim_lattice + verify_claim_lattice_json: when EVERY
resolving claim has TITLE_MISMATCH, demote audit_mode →
UNGROUNDED. Previously HYBRID. Catches the cashback case
('widescreens offer cashback' cited to a generic Coupon
article — n_verified=1 overclaimed when the citation was
meaningless).
2) cli._ladder_rung_for_lattice: DEFLECTION_DETECTED moves from
soft-demote (cap at ANCHOR-WARRANTED) to hard-demote (cap at
POINTER-LINKED) alongside WARRANT_MISSING and TITLE_MISMATCH.
Catches the comeliness case (model fully shifted topic to
'Rock & Chips'; pre-fix the verified Rock & Chips claims
landed at ANCHOR-WARRANTED unfairly).
Test updates:
- tests/test_verify_json.py:test_verify_json_title_mismatch_demotes_to_ungrounded
(renamed; expected outcome flipped HYBRID → UNGROUNDED)
- 4 fixtures in tests/test_claim_lattice.py and 1 in
tests/test_verify_json.py: claim text augmented with 'in the
film' so the Jurassic-Park-titled fixture's title shares a
stem with the claim. These tests were testing other paths
(NO_EVIDENCE_POINTER, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID)
and the title-mismatch was incidental to the fixture choice.
Teacher blocks on the four log entries record the verdict +
failure_class + recommendation + notes, with reviewer + date.
The two ✅/🟡 entries have no recommended code change; the two 🔴
entries reference this commit as the fix.
751/34 tests pass.
12 per-module files + index = 13 files of 50-185 lines each =
1,064 lines of API reference scattered across a directory.
Each per-module file had real meat (API examples, ASCII tree,
conventions) but the cognitive cost of 'which file is this in?'
outweighed the navigation benefit.
Built via concatenation + patch-fix:
- cat index.md + per-module files in topological order
- rewrite ../diagrams/ -> diagrams/ (relative to docs/modules.md)
- rewrite ../../aborist/ -> ../aborist/
- rewrite ../TICKETS.md -> TICKETS.md, ../mesh.md -> mesh.md, etc.
- inter-module links (./<name>.md, <name>.md) -> #<name>-py anchors
- demote per-module H1 -> H2, H2 -> H3, etc., so the wrapper H1
is the only top-level heading
- de-dup the index.md's (now-H2) 'Aborist module reference'
header against the wrapper, replace with 'Diagrams index'
- inject explicit <a id="<name>-py"></a> anchors after each
module's H2 so the TOC links resolve regardless of GitHub's
auto-slug rules
- polish TOC link text: '[merkle.md](#merkle-py)' -> '[↓](#...)'
(the '.md' suffix made no sense once it's an in-doc anchor)
References updated:
- README.md (×2)
Net: 1,124 single-file lines vs 1,064 across 13 files. Slightly
larger because of the patch-fix scaffolding (anchors + section
markers), but one Cmd-F covers everything.
751/34 tests still pass.
Two cleanup operations bundled (separate scopes, single commit
since they share the doc-tree settle):
1. Move ticket-NNNNNN-<slug>.md files into docs/tickets/. The
directory makes browsing the design log easier; the index
stays at docs/TICKETS.md (top-level pointer). Convention text
in TICKETS.md updated to spell the new path.
2. Delete three docs whose load-bearing content has either been
absorbed into the codebase or distilled into closed tickets:
- docs/naming-deferral.md (147 lines) — explained why we
don't rename claim_lattice → CTI/PROMETHEUS-Σ. Decision
stays in place; the rationale is no longer worth a
dedicated doc. Inline citations removed from
cti-architecture.md (4 refs), warrant.py, ticket-000003
(closed-ticket internal ref).
- docs/reference-frame-failure-class.md (169 lines) — Orwell
case journal that motivated the phrase-pattern retrieval
route. The route shipped; the analysis is now duplicate
with the closed Ticket #000002. Inline citation removed
from CLAUDE.md retrieval pipeline section + frame.py.
- docs/test-coverage-audit-2026-05-01.md (46 lines) —
point-in-time audit checking 16/16 of fox's §11 list. Tests
themselves live in tests/; the audit was a one-shot
checkmark exercise.
References updated:
CLAUDE.md, aborist/qa/frame.py, aborist/qa/retrieval_plan.py,
aborist/qa/warrant.py, docs/cti-architecture.md, docs/TICKETS.md,
docs/tickets/ticket-000003 + ticket-000004 (internal links).
Net: -362 lines + tickets/ subdir. 751/34 tests still pass.
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
The journal had two stapled sections — the morning 11:31Z bench
(n=2 cell-grouped) followed by an 'Update' divider and the
afternoon 15:07Z bench (n=3 sample-shuffled). Two parts that
made readers walk past a horizontal rule to compare numbers, and
duplicated the directive-coverage / outputs / per-mode-rec
sections.
Rewrote as one coherent narrative:
- Frontmatter mentions both stamps in one table (when/what/wall)
- 'Hardening' section walks the chronology in two beats:
* pre-11:31Z (Rule 8, warrant gen, ladder, etc.)
* 11:31Z → 15:07Z (Sprint 1b, Sprint 2, DRY, keep-alive,
sample-shuffle, --resume, concurrency-sweep, surrogate v2)
- 'Aggregate' = authoritative 15:07Z table
- 'Δ across the day' = 3-column comparison
(2026-04-30 → 11:31Z → 15:07Z) with net deltas
- 'Per-bucket strict-rate' = 15:07Z bucket data
- 'Recommended context budget' = final
- 'Pointer-mode signal' kept (lazy-anchor analysis still valid)
- 'Wall-clock & throughput evolution' = both benches in one
table + concurrency sweep table
- 'Errors' = surrogate story across both benches
- 'Verdict' = final recommendation
- 'Outputs' = both jsonl files
212 lines → 175 lines (-36). One source of truth, one read.
bench-maxing.md gains a 'Bench harness — the speed playbook'
section capturing the 2026-05-02 speed wins as durable doctrine:
- Sample-level shuffled scheduling vs cell-grouped (+58%
throughput, true i.i.d. variance for n>=3)
- Persistent httpx client (TLS handshake amortization)
- Concurrency tuning (vLLM peak at c=3-4, brutal past c=4)
- Per-mode max_context_chars from bench feedback (the bench
is the substrate's voice; let it drive policy)
- --seed for reproducibility
- --resume for stop/start-able bench
- Smoke fixture for inner-loop iteration
- pytest -n auto (3.6× speedup on test suite)
README:
- Updated whitepaper section refs (§6/7/8/9/13) — old refs
pointed at §13.4.11/13.8/13.9 which no longer exist after
the whitepaper rewrite landed.
- 'Tests' section renamed 'Tests & bench' with make targets
for bench-qa, bench-qa-smoke, test-live. Resume + concurrency
semantics surfaced.
The c=4 sample-shuffled bench at 15:07Z lands the post-Sprint-1b
+ post-Sprint-2 + post-DRY + post-keep-alive + post-shuffle
state.
Headlines:
quote 0.50 → 0.54 (+4pp)
claim_lattice_pointer 0.23 → 0.20 (-3pp)
claim_lattice (JSON) 0.44 → 0.42 (-2pp)
Quote's +4pp is the cleanest lift of the sprint set: the per-mode
24KB cap (Sprint 1b) surfaces tighter retrievals that quote can
ground verbatim, and the bucket data confirms quote peaks at
8-16KB (0.58 strict-rate). JSON's peak migrated to its targeted
32-64KB bucket (0.48 strict-rate, vs 0.38 at 16-32KB) — Sprint
1b's intent confirmed at the per-bucket level even though the
aggregate slipped 2pp.
Pointer's slight drop is consistent with Sprint 2's smoke result
— the chunk-specificity Rule 9 didn't lift Hermes-3-8B's
lazy-anchoring at n=3. The structural fix will need a stronger
intervention than a prompt nudge.
Wall-clock & throughput:
11:31Z: cell-grouped, c=4, n=2, 426 tasks, 51 min, 8.4/min
15:07Z: sample-shuffled, c=4, n=3, 639 tasks, 48 min, 13.3/min
Sample-shuffled scheduling delivers +58% throughput at same
concurrency. n=3 (50% more work) ran in 6% LESS wall-clock.
Per-call mean latency dropped 35-42% across all modes — vLLM's
continuous batcher fills better when fed a diverse request
stream instead of cache_key-correlated cells.
Concurrency sweep: c=3 peak, c=4 within 4% (chosen), c=5 12%
slower, c=6 brutal (45% slower). vLLM saturates at c=3-4 on
this endpoint.
Errors: 6, all on 'tell me about the roman empire' question.
Root cause traced & fixed in 41d1d9b (lone UTF-16 surrogates
in Wikipedia chunk content broke httpx's outbound JSON encode
— different path from the 3b91223 SHA-256 hashers fix which
hardened the OUTPUT side). Next bench: 0 errors.
Six bench runs aborted with:
UnicodeEncodeError: 'utf-8' codec can't encode characters
in position N-M: surrogates not allowed
The error fires inside httpx's json-encode path: when the
request body's JSON contains lone surrogates (from Wikipedia
chunks ingested with invalid-UTF-8 source bytes), httpx's
.encode('utf-8') raises before the request even leaves the
client.
Earlier surrogate fixes (3b91223) hardened the OUTPUT side —
sha256 hashers now use errors='surrogatepass' so the run-DAG
roots survive surrogate-bearing model output. But the INPUT
side (corpus text injected into the prompt) was still
vulnerable: the LLM never sees the surrogate but the HTTP
client tries to send it.
Fix: scrub message content via WTF-8 → UTF-8-with-replace
roundtrip in OpenAICompatibleClient.chat_completion. Lone
surrogates become U+FFFD (REPLACEMENT CHARACTER); the prompt
serializes cleanly. Verified: 'tell me about the roman empire'
under claim_lattice mode now classifies HYBRID 5/7 instead of
erroring out (this question was 6/6 lattice runs failing on
the 2026-05-02 c=4 bench).
The scrub lives in the client because the hot path needs to
guarantee the outbound HTTP body is valid UTF-8, regardless of
what upstream code injected. Defense-in-depth: ingest-time
sanitization would be cleaner but the existing corpus already
has surrogates baked in, and re-ingest would invalidate every
document_root in 6 GB of shards.
Tests: 751/34 still pass clean in 11s with pytest -n auto.
Surfaces where the per-call cost lands. The audit-line `7.5s`
already shows total_ms but it's hidden inline with the verdict
& cache_status. The new `timings:` line (after `capacity:`) breaks
total wall-clock into its components so an operator can see at a
glance whether a slow query was Hermes-bound, search-bound, or
something else.
Format:
timings: cache 0.00s · search 0.30s · context 0.16s ·
llm 6.50s · persist 0.20s · **total 7.16s**
Skipped phases (cache=0 on cache-miss; persist=0 on cache-hit;
context=0 when retrieval pre-loaded) are filtered so the line
stays compact. **total** stays bold to match the audit-line
elapsed value already on screen.
JSON mode (`--json`) already exposes `timings` — this just lifts
the same data into the human render.
Tests pass (641 / -n auto / 11s).
Adds --resume <jsonl-path>: read the existing JSONL, build a set
of (question, mode, sample_idx) tasks already done, skip those
in the shuffled task list, and append fresh rows to the same
file. The markdown rollup uses the union of pre-existing + new
rows. Same --seed required for the order to align across the
resumed run.
Why: a 60-90 min full bench is annoying to re-run from zero when
something interrupts (network blip, kill, kernel panic). The
JSONL has been the durable artifact for ages; the bench just
didn't know how to read it.
Implementation:
- args.resume: Path | None
- if given: parse JSONL line-by-line, populate done_tasks set,
reuse the resume_path's stem as the run stamp, set jsonl_path
+ md_path to the resumed paths
- file_mode = 'a' if resuming else 'w'
- tasks filter: drop entries already in done_tasks
- rows initialized with existing_rows so the markdown summary
sees the union
- done counter starts at len(existing_rows) so the print
countdown reflects total progress
Empirical context: the c=6 bench killed at 142/639 left a
recoverable JSONL. Future kills can resume via:
.venv/bin/python bench/qa_sweep.py \
--resume bench/qa_results/<stamp>.jsonl \
--seed 0 --concurrency 4
(Same seed reproduces the shuffle so the remaining tasks come
out in the original order.)
Concurrency sweep on the smoke (15 tasks at each c):
c=3 102s (peak throughput)
c=4 106s (4% slower; chosen for full bench — more
forgiving on single-call hiccups)
c=5 119s (12% slower than c=3)
c=6 185s (45% slower; vLLM batching ceiling)
Lesson: more concurrent ≠ faster. vLLM continuous batching has
a sweet spot around c=3-4 for this endpoint.
Adds `workflow:rules: - when: never` at the top of .gitlab-ci.yml
so GitLab refuses to create any pipeline on push. The job
definitions stay in place — re-enable is a 4-line revert (delete
the workflow block) when bench infrastructure & CI runners are
decoupled.
Reason: aborist CI runners overlap with the same in-house pool
fox uses for `make bench-qa` against the live Hermes endpoint.
Auto-triggered pipelines on every push contend for the runner
& risk skewing bench latencies (already saw concurrency=4 push
mean per-call latency from 6-8s to 29s — adding background CI
jobs makes the signal noisier).
Local `make test` is unaffected — still 11s with -n auto.
Previously the bench scheduled at the (question, mode) cell level —
each cell ran its n samples sequentially before the next cell
started. With n=3 + concurrency=6, that meant 6 workers each
chained 3 sequential calls, vLLM's continuous batcher saw
correlated requests, and the n=3 variance signal was confounded
with 'what was vLLM doing in this same-cell batch.'
Two coupled defects, one fix:
(1) Statistical: samples-of-the-same-cell are NOT i.i.d. when run
back-to-back. Adjacent samples share vLLM batch composition,
KV-cache locality, and queue load. The 'variance' you see
across n=3 is partly that batch's particular weather, not true
model nondeterminism.
(2) Throughput: vLLM batches diverse requests well; correlated
requests fill the batch with similar work and starve.
Cell-grouped scheduling correlated by cache_key.
Refactor: build sample-level tasks ((q, mode, sample_idx)),
shuffle with deterministic seed (default 0), submit all to
ThreadPoolExecutor. Per-cell Lock dict (defaultdict(Lock))
serializes burn+insert against the shared cache_key — two samples
of the same cell that happen to land in adjacent worker slots
will queue on the cell's lock instead of racing. Lock contention
is rare under shuffle (samples of one cell are spread across
time) so throughput cost is near-zero.
New CLI flag --seed N for reproducible task ordering.
Changes:
- tasks list at sample granularity
- rng = random.Random(seed); rng.shuffle(tasks)
- cell_locks = defaultdict(Lock); held over _run_one + sample_idx assignment
- print line shows '#i/n' so order-of-completion is visible
Smoke at --n 1 --concurrency 6: clean, no errors, ~3 min for 15
tasks (essentially same wall-clock as cell-level scheduling at
n=1, as expected — the real win comes at n=3 where sample-level
concurrency unlocks parallelism within cells).
`.gitlab-ci.yml` runs the unit test suite (`make test`, which uses
`-n auto`) on every push. PYTEST_XDIST_AUTO_NUM_WORKERS=4 caps the
worker count so we don't oversubscribe on shared runners — pytest-
xdist's `-n auto` reads os.cpu_count() and on cgroup-limited
containers that over-reports the host's cores. Local `make test`
stays uncapped (uses full machine).
Wall-clock on the 641-test suite:
serial: 38s
-n auto local: 11s (full machine, 8+ cores typically)
-n auto + cap 4: 22s (CI runners — 1.7× speedup, still safe)
Job structure mirrors sibling repos (unsandbox.com,
unfirehose-nextjs-logger):
- tags: build → in-house runner pool
- cache keyed on pyproject.toml, paths .venv/ + .pip-cache/
- artifacts on failure → .pytest_cache/ for triage
Excluded from CI (need live state or network access):
- make test-live → live Hermes endpoint + populated shards
- make bench-qa{,-quick,-smoke} → live LLM bench
- chain-check-shards / analyze-shards → real shard data
Optional `test-crawler` job: gated on commit-message tag
`[ci-crawler]` or manual trigger; installs `[crawler]` extras and
runs the network-bound crawler suite. allow_failure: true so it
doesn't block merges.
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
OpenAICompatibleClient.chat_completion previously constructed a
fresh httpx.Client per request inside a 'with' block:
with httpx.Client(timeout=self.timeout) as client:
resp = client.post(url, headers=headers, json=payload)
Each call paid a TLS handshake (~100-300ms) — wall-clock cost
that bench --concurrency surfaces sharply. With 426 calls per
bench sweep, the throwaway-client pattern was burning roughly
1-2 minutes of pure connection setup per sweep.
Move the httpx.Client to __init__ so it lives across calls.
HTTP/1.1 keep-alive holds the TCP+TLS connection open between
chat_completion invocations from the same client instance;
httpx's internal connection pool is thread-safe so the bench's
ThreadPoolExecutor can share one client across worker threads.
Add close() / __enter__ / __exit__ for clean shutdown — the
client now has connection pool state worth releasing
explicitly. Existing callers that don't use the context-manager
form work unchanged because httpx.Client cleanup runs on GC.
Smoke at --concurrency 6 against hermes.ai.unturf.com: 15 cells
in 2m24s, no errors. vLLM continuous-batching observed: first 6
cells completed together at ~76s, suggesting the batch filled
and processed as a unit. Subsequent waves at 14-50s as the queue
drained.
Predicted bench savings: 1-2 min off the ~51 min full sweep.
Combined with --concurrency 6, predicted total: ~30 min full
bench (vs 51 min at concurrency=4).
The four claim-lattice prompt strings —
CLAIM_LATTICE_SYSTEM_PROMPT
CLAIM_LATTICE_GROUNDING_REMINDER
CLAIM_LATTICE_JSON_SYSTEM_PROMPT
CLAIM_LATTICE_JSON_GROUNDING_REMINDER
— were duplicated byte-for-byte between aborist/qa/runner.py
DEFAULT_POLICY (single-document path) and aborist/qa/query.py
DEFAULT_QUERY_POLICY (multi-source path). Sprint 2 surfaced the
DRY violation when its agent had to update both files in lockstep
to add Rule 9. Two sources of truth = one source of drift.
Lift the four strings to aborist/qa/prompts.py as module-level
constants. Both DEFAULT_* dicts import them and reference by
name. The surrounding policy entries legitimately differ between
the two paths (different system_prompt/grounding_reminder for
single-doc vs multi-source framing, different max_tokens,
different chunk caps) — only the lattice prompts collapse.
Net shrinkage: -240 duplicated lines across runner.py + query.py,
+153 in prompts.py = ~87 lines removed. Sole authoritative copy
lives in prompts.py.
Verified via runtime equality check: both DEFAULT_POLICY and
DEFAULT_QUERY_POLICY resolve to the same string objects post-
import, matching the prompts.py constants byte-for-byte. Test
suite (751/34) passes clean in 11s with pytest -n auto.
No behavioural change. governance_policy_hash unchanged because
the strings themselves are unchanged. Cache namespace
unaffected.
Sprint 2 — add Rule 9 to the claim_lattice_system_prompt to
discourage lazy-anchoring. Targets the structural pattern surfaced
by the 2026-05-02 bench: 49/142 pointer-mode rows (35%) had
lazy_anchor_ratio >= 0.75, and 19/71 questions show pointer fully
HHHHH while JSON aces SSSSS — pointer cites a topic-overview chunk
where JSON's structured per-claim linkage forces specificity.
The new rule (positive form per docs/bench-maxing.md):
9. Pick the pointer whose own text contains the claim's
specific facts — its dates, names, places, numbers, and
verbs. A chunk whose title matches the question's topic is
the right anchor only when that chunk's text also states
the claim's facts; when a different chunk states the fact
more directly, cite that one instead.
Six lines, no negation, named failure mode. Aligns with Rule 8
(title-relevance) by definition: a chunk whose body states the
fact almost always shares stems with its title.
The pointer prompt is duplicated between aborist/qa/runner.py
DEFAULT_POLICY and aborist/qa/query.py DEFAULT_QUERY_POLICY.
Both updated in sync. (DRY violation worth cleaning up later —
the duplication can drift; one source of truth would be safer.)
Smoke (5-question fixture, n=1) — no regression, n=1 too noisy
to detect signal:
quote 5/5 → 4/5..5/5 (oscillation, single dinosaurs flip)
pointer 1/5 → 0/5..1/5 (sampling noise around the floor)
JSON 5/5 → 5/5 (held)
Real signal lives in the full 71-question n=3 bench (~50 min).
The change is principled, no calibration cost, and folds into
governance_policy_hash so prior cached records auto-partition.
Sprint 1b — replace the flat max_context_chars=60000 default with
per-mode budgets matching the peak bucket each mode achieved on
the 2026-05-02 bench:
quote → 24000 (peak 16-32KB)
claim_lattice_pointer → 24000 (peak 16-32KB)
claim_lattice (JSON) → 48000 (peak 32-64KB)
The flat 60K was past peak for quote/pointer (which degrade after
32 KB) and only marginally above peak for JSON. The bench's
recommended-context-budget table now flows back into the policy
defaults — operator-driven landing per the five-step algorithm
step 5: surface, don't auto-apply. The mapping lives in
DEFAULT_QUERY_POLICY so any tuning bumps governance_policy_hash
and partitions the cache namespace cleanly.
API boundary stays backward-compatible: an explicit
max_context_chars= caller value still wins. The per-mode default
fires only when the caller passes None (or omits the arg).
CLI: --max-context-chars default flips to None; help text spells
out the per-mode fallback.
Smoke (5-question fixture) confirmed no regressions vs the
pre-change baseline:
quote 2/5 → 5/5 (+3)
pointer 0/5 → 1/5 (+1)
JSON 5/5 → 5/5 (flat — already at ceiling on this fixture)
Quote's +3 is the unexpected win: the smaller 24K budget surfaces
tighter-relevance top-K instead of drowning the model in filler.
The full bench (~50 min) is the real scoreboard but smoke shape
matches the bench's predicted lift direction.
Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.
UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.
Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.
Touched:
aborist/qa/dag.py:_sha256_hex (the loud one)
aborist/qa/keys.py:_sha256
aborist/qa/evidence.py:_sha256_hex
aborist/store.py: chain_audit_events + append_audit
Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).
Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.
Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.
Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.
pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.
Bench-max scoreboard (predicted lift from this commit alone):
+1pp lattice modes (UTF fix)
+cycle-time enabler (smoke fixture, xdist)
no calibration cost — none of the verifier checks moved.
Captures the post-Rule-8 / post-warrant-generalization /
post-frame-detector bench taken with --concurrency 4 (~51 min
wall-clock on 426 runs).
Headlines (vs 2026-04-30 post-retry rerun):
quote 0.47 → 0.50 (+3pp)
claim_lattice_pointer 0.24 → 0.23 (−1pp)
claim_lattice (JSON) 0.50 → 0.44 (−6pp)
Pointer mode is essentially flat. The earlier 'pointer regressed
13pp' alarm was wrong — that compared to the pre-retry
2026-04-30 numbers (0.36) instead of the post-retry rerun (0.24)
which is the correct reference.
Real news: JSON's 6pp drop is the honesty cost of Rule 8
title-relevance promotion + warrant-class generalization. Lattice
modes now hit 99% directive coverage (D2/D3/D4/D6/D7); the 1% gap
is 2 UTF-16 surrogate errors per mode on one question.
Pointer-mode failure pattern shifted to lazy-anchor: 49/142 rows
have lazy_anchor_ratio >= 0.75. JSON's structured-per-claim
linkage absorbs Rule 8's pressure better than pointer's
prose-with-tags. 19/71 questions show JSON > pointer by 50pp+.
Recommended context budget surfaced for the first time:
- claim_lattice (JSON) peaks at 32-64KB (0.51)
- pointer + quote peak at 16-32KB
CLAUDE.md headline numbers refreshed; docs index points to the
new journal alongside the prior 2026-04-30 reference.
zstandard.ZstdCompressor and ZstdDecompressor instances each carry an
internal libzstd context that is NOT thread-safe. Calling .compress()
or .decompress() on a single shared instance from multiple threads
corrupts the context and raises:
ZstdError: decompression error: Data corruption detected
Surfaced when bench/qa_sweep.py learned a --concurrency flag and ran
4 (question, mode) cells in parallel. ~19% of retrievals failed on
zstd corruption before the fix. The prior comment claiming the
singletons were 'stateless across calls — safe to share across
threads' was wrong.
Replace the module-level singletons with threading.local() caches.
Each thread reuses its own ZstdCompressor / ZstdDecompressor; no
contention across threads. Init cost is negligible vs decompression.
bench/qa_sweep.py:
--concurrency N (default 1) parallelizes (question, mode) CELLS
using a ThreadPoolExecutor. Samples within a cell stay sequential
so burn-then-insert against a single cache_key never races itself.
Lock-protected JSONL writes & progress prints. Exception in any
worker surfaces via fut.result().
vLLM handles concurrent requests well; 4-8 is a reasonable starting
point. With --n 2 --concurrency 4, expect ~10 min wall-clock for the
full 426-run sweep against hermes.ai.unturf.com (vs ~2h sequential).
Three small fixes flagged by the whitepaper rewrite sweep:
1. Note the four-rung ladder render in the providence-cache layer
description (audit_mode is schema-level; lattice modes display
POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED, with
ENTAILMENT-VERIFIED reserved).
2. Add a Conventions entry for the ladder right after the three
answer modes — names the rung discrimination rules, what each
violation demotes to, and that quote/span/entity/paraphrase
modes keep their original tokens.
3. Replace the 'Open tickets' section with a one-liner pointing to
docs/TICKETS.md as authoritative; all five shipped tickets are
closed as of 2026-05-02. The previous listing claimed #000001-3
were open, which has been false since the prior session.
Phase 2 of the holistic-tuning pair. Replaces the un-indexable
LOWER(title) LIKE '%tok%' title-search with an FTS5 MATCH-based
lookup. The structural fix that was deferred when the >5-token
bypass landed (commit 1d70c4f).
(1) Schema: new `documents_fts` virtual table over the title column.
Contentless mode (same trick as chunks_fts) — stores only the
inverted index, not a copy of the title. Joins back to documents
via rowid for the post-filter caller.
(2) Extractor: backfill_documents_fts (registered as evidence_kind
"documents_fts"). Reset-and-rebuild the index from documents in
one INSERT...SELECT. Idempotent. Cost: ~2.5s per 870k-doc shard.
(3) `_search_titles` rewritten: try FTS5 MATCH first, fall back to
LOWER(title) LIKE only on shards lacking documents_fts data
(legacy ingest pre-this-commit). MATCH expression OR-joins
quoted query tokens; falls through to LIKE form on tokenizer
edge cases.
(4) Removed the `>5 accept_tokens` bypass in `_search_corpus`
that commit 1d70c4f added as a workaround for the LIKE
full-scan cost. With FTS5 the title search is sub-second
regardless of token count, so synonym-expanded title search
is affordable at any query length.
Live verified on the 19-token brain-tech query with 50 accept_tokens
(post-IDF expansion):
Pre-FTS5: ~50s (LIKE '%tok%' × 870k docs × 4 shards)
Post-FTS5: ~0.04s ← 1250× speedup
End-to-end query: 10.4s (was 11.6-13s; the saved title-search
time partially absorbed by Hermes inference variance).
Backfill cost: 10.5s wall-clock across 4 wiki shards + 1 crawl
shard. Storage: ~30 MB per wiki shard for documents_fts (well
inside the 90 MB/shard concept-layer budget). Re-running is
idempotent (DELETE FROM ... INSERT INTO ...).
Tests: 641 passed (no regression).
Phase 1 of the holistic-tuning pair fox requested 2026-05-02. Replaces
the dumb alphabetical-truncation heuristic in synonym_expand's cap
path with corpus-derived IDF ranking — rare topical synonyms win the
truncation, common-corpus noise drops.
(1) New per-shard table: concept_token_idf (token, doc_freq, total_docs,
derived_at). Indexed on doc_freq for ORDER BY ranking. Folds into
the cross-shard UNION view via _SHARDABLE_TABLES.
(2) Extractor: backfill_token_idf (registered as evidence_kind
"token_idf"). Reads chunk-frequency from FTS5's fts5vocab virtual
table, bounded by the union of token + target columns in
concept_relations (only synonym tokens get an IDF row, not the 6M
full corpus vocabulary). Idempotent INSERT OR REPLACE.
(3) Query layer: _load_token_idf sums per-token doc_freq across all
shards (cross-shard ranking). _get_indices return tuple grew from
(manual, derived, rivalry) → (manual, derived, rivalry, idf). Cache
LRU keyed on shards_dir mtime so the IDF lookup is per-process-once.
(4) synonym_expand cap-time truncation: when expanded set exceeds
MAX_TOTAL_TOKENS, sort neighbors by (doc_freq ASC, token ASC) so
rare tokens come first. Tokens missing from concept_token_idf get
a sentinel high-rarity score (treats unknown as hapax — fail-safe
when the IDF backfill hasn't run yet).
Live verified on the brain-tech 19-token query:
Pre-IDF expansion (alphabetical 50): bumper, buster, channel,
convention, dave, duck, esp, field, glen, mind, minimum, ...
Post-IDF expansion (rare-first 50): neuroimaging, neurons,
neuroscience, neurotechnology, psychokinesis, telepathic,
telepathy, thinking, tms, transcranial — all the brain-tech
terms that lost the alphabetical race in pre-IDF now win.
Backfill cost: ~50s wall-clock across 4 wiki shards (~22k tokens
indexed per shard). Fts5vocab over chunks_fts is the right
infrastructure — no full corpus scan needed since we filter by the
concept_relations token set. Storage: ~1 MB per shard for the IDF
table. Stays well inside the ~90 MB/shard concept-layer budget fox
set earlier.
Tests: 641 passed (no regression). Live fixtures (boss baby,
amazon river/rainforest) all pass — ranking doesn't break gating.
The link was still pointing at the old pre-Providence slug
(/downloads/merkle-reverse-rag-whitepaper.pdf). Canonical URL is
/merkle-providence-reverse-rag-whitepaper.pdf at the static root,
where `make publish-and-commit` keeps it fresh on every whitepaper
edit. Same fix as the unsandbox.com Phoenix templates (commit
0d1a69c) — last remaining stale reference.
Two compounding things in one commit because they're tightly coupled
(the new violation is what makes the new fixtures meaningful):
(1) DEFLECTION_DETECTED — new soft-demote violation kind.
`aborist.qa.inspect.diagnose_deflection` was a read-only sidecar:
when the question's subject anchor (last content token after
stopword strip) is missing from the answer, it returned
`kind="deflection"`. Useful telemetry but never affected the
verdict. 2026-05-02 fox surfaced two cases where the missing
signal mattered:
Q: "who burns the amazon river?"
A: "Deforestation in the Amazon Rainforest is primarily caused
by human activities..." [E10 | Deforestation of the Amazon
Rainforest]
Pre: EVIDENCE-WARRANTED 1/1 ← silent river→rainforest topic shift
Post: ANCHOR-WARRANTED-PARTIAL 1/2 with DEFLECTION_DETECTED
The verifier now runs diagnose_deflection() after all hard checks.
On `kind == "deflection"` it appends a DEFLECTION_DETECTED
violation (rationale + subject_anchor + overlap_ratio) and caps
audit_mode at HYBRID. Renderer's four-rung ladder picks it up via
`_SOFT_DEMOTE_VIOLATION_KINDS` (cli.py) and demotes EVIDENCE-
WARRANTED → ANCHOR-WARRANTED automatically.
Folds into verifier_policy_hash via new
`claim_lattice_deflection_check_enabled: True` policy field
(defaults on). Bumps the 9-dim cache_key when toggled.
Wired into both verify_claim_lattice (pointer) and
verify_claim_lattice_json (JSON). Deferred import keeps verify.py
module-load clean.
(2) Three live fixtures gating the river/rainforest/deflection family:
- test_amazon_river_burning_who_deflects_or_grounds_river
Gates the silent topic-shift pattern: if model deflects, the
verifier MUST catch it (DEFLECTION_DETECTED present OR
UNGROUNDED OR "river" actually in answer). Failure mode this
catches: EVIDENCE-WARRANTED with rainforest answer to a river
question and no soft-demote.
- test_amazon_river_burning_culture_refused_or_ungrounded
Adversarial-impossible mirror of Mars-BDFL: rivers don't burn,
so model must refuse / deflect / refute. Failure mode:
STRICT/HYBRID with affirmative culture-burns-river claim.
- test_amazon_rainforest_burning_culture_grounds_or_deflection_caught
Feasible-but-framing-specific: corpus has deforestation
content but no "culture" framing. Either grounds with rainforest
content OR a soft-demote/UNGROUNDED catches the framing gap.
Live verified (2026-05-02): all 3 pass against the real Hermes
endpoint. 641 unit tests still pass (deflection check skipped on
fixtures lacking question or rendered_text).
Captures the canonical "model knows it but corpus doesn't" case fox
surfaced 2026-05-02:
Q: what is boss baby?
Expected: UNGROUNDED (the 2010-11 wiki snapshot has no Boss Baby
coverage — the 2017 DreamWorks film didn't exist; the 2010 Marla
Frazee picture book had just been published & no article)
Model: emits training-prior content about the film/book
Verifier: 0/N verified — no evidence chunk anchors the claims
This is the failure-by-honest-admission case the verifier must
preserve. Two compounding false-positive risks the test guards:
1. Verifier accepting the claim against an unrelated chunk that
happens to share content tokens (Death of Baby P, Cake Boss,
Beanie Baby — all adjacent-token noise observed at top-K).
This is the warrant-failure class warrant-lite was built to
catch — if it ever passes here, anchor-class warrant has a hole.
2. Verifier accepting via OR-mode FTS5 noise — a token-level match
of "baby" or "boss" in unrelated content surfacing as positive
grounding.
Acceptable behaviors:
A. UNGROUNDED (verifier returns 0/N — corpus-truth over prior)
B. STRICT/HYBRID without "boss baby" anywhere in the answer
(deflection — model declined the false-corpus question and
answered an adjacent grounded fact, like Mars-BDFL → Guido
pattern).
Failure: STRICT/HYBRID + "boss baby" in answer = false-positive
grounding. Test fails by name pointing at the warrant gap.
Live verified (2026-05-02): UNGROUNDED 0/2, model emitted both the
2017-film claim AND the 2010-book claim, verifier rejected both
honestly. Test passes in 10s.
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.
Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.
OR pool example for the brain-tech query:
Pre-synonym: reconstruct, technology, understand, available, thoughts
Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
→ 3/5 brain-tech terms surface naturally in OR-mode.
AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.
Live verified on the 19-token brain-tech query:
- 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
- 2/2 verified pairs
- Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
brain-tech corpus sources, not Universal-pragmatics nonsense)
`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.
Tests: 641 passed (no regression).
Both retrieval phases were doing O(corpus × tokens) full-scan work
on long queries — synonym work surfaced the latent issues but they
predate it. Hermes is 6-8s; retrieval needs to be sub-second.
(1) `_search_titles` (LIKE-based title backup search) is fundamentally
O(corpus × |accept_tokens|) — `LOWER(title) LIKE '%tok%'` can't
use any index. 50 patterns × 870k docs × 4 shards = ~56s pure
full-scan. Skipped for queries with > 5 accept_tokens, where
FTS5 BM25 already returns better candidates. Title-LIKE remains
a backup for short focused queries (1-5 tokens).
(2) FTS5 OR-mode fallback was OR-joining ALL post-stopword query
tokens. A 19-token query OR-clause matched millions of docs;
BM25 then ranked them all. 13s/shard observed.
Capped to top-5 LONGEST tokens (proxy for rarity / topical
specificity). Long words like "neurotechnology" matter; short
words like "soon" don't. OR-mode now 0.25s/shard.
(3) Stopword set extended with 6 connector words (one, some,
another, without, soon, currently). Conservative addition —
these have zero topical signal. Reduces the AND-mode token
count, raising the chance AND succeeds before falling back to
OR.
Verified live: the 19-token brain-tech query that hung indefinitely
now returns in 18s (Hermes + verify + render dominate). Source
`_search_corpus` no longer expands synonyms for the title-LIKE call
either — synonym expansion stays in `_filter_by_title_relevance`
(post-retrieval, in-memory, cheap regardless of accept-set size)
where it actually does useful work.
Tests: 641 passed (no regression).
