Two paired changes addressing fox's "learn this hyperparameter
from model use, not hard coding" + the 1M-context-window
caveat:
(1) Log-scale prompt-size buckets extend from 8KB through 1M+:
<8KB / 8-16KB / 16-32KB / 32-64KB / 64-128KB / 128-256KB /
256-512KB / 512K-1M / >=1M
The same bench harness now covers 8B-class models (Hermes 82K
context, max useful prompt ~32-64KB) through 1M-context models
(Gemini 1.5 Pro, Claude with extended context, Llama 4) without
code change. A model whose context window stops at 82K simply
never populates the giant buckets; a 1M-context model fills
them and finds its own sweet spot.
(2) New "recommended context budget (learned from this bench)"
section — per-mode peak-strict-rate bucket. Operator-driven
landing per the five-step algorithm step 5: surfaced, not
auto-applied. Minimum sample size of 5 runs per bucket so
statistical noise doesn't masquerade as signal. Tie-break on
smaller-bucket-wins so equivalent strict-rates favor the
cheaper choice.
The substrate is now self-tuning at the OBSERVATION layer: bench
records what budget actually grades best per model. Per-model
profile JSON (storing the recommended budget back into
~/.aborist/model_profiles/<model>.json) is the next beat once
this surface is observable in real bench runs.
3 new bench tests cover the recommended-budget section, the
sample-size floor, and the giant-context bucket coverage.
Full suite: 746 passed (was 738, +8).
Connects to D8 (automate after test-pinning): the bench tells us;
we don't guess.
Wires the seven-point program (docs/seven-point-program.md) into
the bench harness as a per-mode coverage table.
Per-row computation in _directive_compliance(answer_mode, result, err):
D2_pointer_clauses — answer_mode is lattice variant
D3_cti_substrate_ready — lattice mode + run_dag_root populated
(full coverage pending ticket #000002)
D4_evidence_map_bound — run_dag_root populated
(retrieval-plan binding pending #000001)
D6_warrant_fired — warrant ran (lattice mode today;
per-shape gating pending #000003)
D7_honest_label — audit_mode in canonical enum
(renderer transformation pinned in
test_cli_render.py)
D1 (no LLM in verifier), D5 (verifier_method enum), D8 (test-pinning
discipline) are global properties of the substrate and don't appear
per-row — they get tracked once in CLAUDE.md / the program doc.
_summarize aggregates per-row booleans into per-mode pass counts.
_render_markdown emits a 'directive coverage (seven-point program)'
section with per-mode per-directive coverage as 'count/N (pct%)'.
Discipline rationale per CLAUDE.md's five-step algorithm step 5:
the bench harness IS the automation substrate, so it must enforce
the directives before any feature gets layered on top. Bench scores
that climb without directive coverage climbing are graveyard-digging
(step 4: don't dig the grave faster).
5 new bench tests cover: per-mode pass aggregation, the markdown
section presence and content, the helper's behavior on quote mode
(D2 fail by design), lattice mode (all directives pass), and error
rows (empty dict — no signal). Full suite 669 passed.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Per-call accounting of prompt capacity so we can answer 'does
context size affect strict-rate?' at aggregate scale.
Per-call result dict gains:
"prompt_chars": {
"system_prompt": <int>, # system message chars
"grounding_reminder": <int>, # restated rule message (if any)
"user_question": <int>, # the original question text
"evidence_or_context":<int>, # rendered evidence map / wikitext context
"messages_total": <int>, # sum of all messages[].content
}
"answer_chars": <int> # rendered answer length
Char-level for now: model-agnostic, fast, and a workable proxy
for prompt tokens (~4 chars/token English prose, ~2.5 for JSON-
evidence-block heavy contexts). Token-exact accounting can layer
on later if a tokenizer is cheap enough to wire in per-call.
Surfaces in three places:
(1) result["prompt_chars"] — for programmatic callers. Cache hits
return prompt_chars too (computed before lookup).
(2) `aborist query` human render — one-liner showing the breakdown
and the answer length, so an operator can tell at a glance
whether STRICT came from a 5KB tight prompt or a 60KB stuffed
context.
(3) bench/qa_sweep.py — per-row prompt_chars_{total,evidence,system,
question} + answer_chars columns in the JSONL, plus a new
"strict-rate by prompt size" section in the markdown summary
bucketing runs by capacity (<8KB, 8-16KB, 16-32KB, 32-64KB,
>=64KB) so the bench reveals whether input size correlates
with verdict quality across modes.
Never enters cache_key — runtime measurements, not policy.
Two paired changes:
(1) Retrieval template-phrase stopwords. Bench finding 2026-04-30:
'tell me all there is to know about york england?' missed the
en.wikipedia.org/wiki/York article entirely — corpus has it but the
FTS5 AND-mode MATCH expanded to "all" AND "there" AND "know" AND
"york" AND "england", which filters to only docs containing every
template word. Articles like "Charles I of England" happen to mention
"all" + "there" + "know" + "england" enough to win; the bare "York"
stub doesn't, so it falls out of the top-K.
Added `all there know everything anything something` to both
_FTS5_STOPWORDS (search/fts5.py) and _TITLE_STOPWORDS (qa/query.py)
so they stay in sync. After the change "York" article ranks #1 on
both 'tell me all there is to know about' and 'tell me everything
about' phrasings.
These are retrieval-time stopwords only — they don't enter the
cache_key or governance_policy_hash (cache_key hashes the original
question text via question_hash). Existing cached records remain
valid; new queries get better retrieval.
(2) bench/qa_sweep.py: per-row deflection_kind + subject_anchor +
subject_in_answer (from qa.inspect.diagnose_deflection); summary
table grows a 'deflections' column tracking STRICT/HYBRID rows
where the question's subject anchor is missing from the answer.
Surfaces the Mars-BDFL pattern at aggregate scale.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Question-set churn (bench/qa_questions.txt):
- narrow factoid: drop 'who founded apple computer?'; add
'a bridge between new london & groton?' (single-answer factoid
with named-entity grounding via Gold Star Memorial Bridge)
- broad descriptive: drop 'tell me about the apollo program'
- entity list: drop 'list the founders of microsoft'; add
'name simpsons family members including pets?',
'list of obelisk in connecticut'
- leading/adversarial: add 'whales are endangered from over hunting'
(declarative-shape, tests verifier under premise-seeking phrasing)
- out-of-corpus: rephrase Mars no-such-entity question to
'who is a benevolent dictator for life for mars?' — sharper
adversarial shape since BDFL is a real OSS-governance term
the model is tempted to project onto Mars
- minor edits: 'GNU linux', 'C programming language', 'method man',
'cold fusion breakthrough' phrasing
tests/test_qa_quality_live.py: rename test_mars_prime_minister_*
to test_mars_benevolent_dictator_*; refresh refutation markers
to match the new BDFL phrasing; expand to cover 'no one' /
'no person' negations the model uses on this shape.
Adds _repair_truncated_json() to recover from mid-output truncation
where Hermes-3-8B writes one long claim text and runs out of
max_tokens before closing the structure. Pre-fix the lenient parser
saw something like:
{"claims":[{"text":"The Apollo program was the United States
spaceflight effort which landed... <CUT>
with no closing `"`, no closing `}`, no closing `]`, no closing
outer `}`. Couldn't parse, lost the whole answer.
Post-fix the repair walks the input once tracking string state
and bracket stack:
- close any open string (append `"`)
- drop dangling structural artifacts:
`"key":` (partial key with colon)
`"key"` after `,` (partial key without colon)
trailing `,` before close
- close the bracket stack in reverse (`{` → `}`, `[` → `]`)
Conservative: never inserts content (no key names, no values, no
commas) — only closes / drops. Worst case the repair is a no-op
or makes parsing fail in a different way; never silently
fabricates data.
Apollo live test result: UNGROUNDED 0/0 (lost answer) →
UNGROUNDED 0/1 (partial first claim recovered with evidence_ids
intact, partial second claim dropped). The user sees the model's
attempted answer in the unverified bucket instead of empty space.
The audit chain captures the repair via fixups labels
(close_string, close_brace, close_bracket, drop_partial_key,
strip_trailing_comma) so an auditor can see what the parser had
to peel.
Pipeline ordering in _lenient_json_parse: strict parse first,
then peel (fence/prose-trim/curly-quotes/trailing-comma), then
strict parse again, then truncation repair as last resort. Each
step's fixups append to the same list so the verdict's
`json_fixups` field is comprehensive.
Bench question set picked up "name simpsons family members
including pets?" — a multi-part list with adjacent-relationship
shape, addresses a gap in the corpus probe.
460 unit tests pass; 12 skipped (live fixtures unchanged).
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Adds bench/qa_sweep.py + bench/qa_questions.txt + `make bench-qa` target.
Sweeps a fixed question set through quote and claim_lattice_pointer modes,
N samples per cell (default 3), each sample burns the cached record so
Hermes nondeterminism becomes the variance source.
Outputs:
bench/qa_results/<utc>.jsonl one row per (mode, question, sample)
bench/qa_results/<utc>.md summary + per-question vote counts
Question set spans the failure-mode shapes the verifier needs to handle:
narrow factoid, broad descriptive ("tell me about X"), entity list,
relationship, and out-of-corpus (should land UNGROUNDED honestly).
Methodology gap this closes: at n=1 the strict-rate swings ±20pp on a
fixed prompt purely from sampling noise. n=3 makes 5pp deltas legible.
aborist/progress.py — stdlib-only rate-limited stderr reporter.
Periodic lines (default every 2s) showing:
elapsed | seen | inserted | docs/s now | docs/s avg | percent | ETA
Wired into ingest_source via a `progress` parameter; CLI default-on
with --quiet to suppress and --total-estimate N to enable percent/ETA.
Demo on a 5k-doc ingest with --progress-interval 0.8:
[ 1s] | 200 seen | 200 new | 193 now | ( 193 avg) docs/s | 4.0% | ETA 24s
[ 7s] | 3,200 seen | 3,200 new | 458 now | ( 424 avg) docs/s | 64.0% | ETA 3s
[ 12s] | 5,001 seen | 5,000 new | 392 now | ( 413 avg) docs/s | 100.0% | ETA 0s
bench/run.sh + `make bench` — reproducible 5000-doc workload through
three configs:
serial single process, single SQLite
parallel-shared N shards, one shared SQLite (WAL serialized)
attached N shards, per-shard SQLite (true parallel writes)
Output is a one-shot table plus CSV at /tmp/aborist-bench/results.csv
so the ratchet is visible as we keep optimizing. Override via
BENCH_DOCS=N and SHARDS=N.
Latest baseline (this commit, on this machine):
config wall_s docs docs/s
serial 11.74 5000 425.7
parallel-shared 10.39 5000 481.1
attached 7.99 5000 625.5
53 tests still passing.
