Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Question-set churn (bench/qa_questions.txt):
- narrow factoid: drop 'who founded apple computer?'; add
'a bridge between new london & groton?' (single-answer factoid
with named-entity grounding via Gold Star Memorial Bridge)
- broad descriptive: drop 'tell me about the apollo program'
- entity list: drop 'list the founders of microsoft'; add
'name simpsons family members including pets?',
'list of obelisk in connecticut'
- leading/adversarial: add 'whales are endangered from over hunting'
(declarative-shape, tests verifier under premise-seeking phrasing)
- out-of-corpus: rephrase Mars no-such-entity question to
'who is a benevolent dictator for life for mars?' — sharper
adversarial shape since BDFL is a real OSS-governance term
the model is tempted to project onto Mars
- minor edits: 'GNU linux', 'C programming language', 'method man',
'cold fusion breakthrough' phrasing
tests/test_qa_quality_live.py: rename test_mars_prime_minister_*
to test_mars_benevolent_dictator_*; refresh refutation markers
to match the new BDFL phrasing; expand to cover 'no one' /
'no person' negations the model uses on this shape.
Adds _repair_truncated_json() to recover from mid-output truncation
where Hermes-3-8B writes one long claim text and runs out of
max_tokens before closing the structure. Pre-fix the lenient parser
saw something like:
{"claims":[{"text":"The Apollo program was the United States
spaceflight effort which landed... <CUT>
with no closing `"`, no closing `}`, no closing `]`, no closing
outer `}`. Couldn't parse, lost the whole answer.
Post-fix the repair walks the input once tracking string state
and bracket stack:
- close any open string (append `"`)
- drop dangling structural artifacts:
`"key":` (partial key with colon)
`"key"` after `,` (partial key without colon)
trailing `,` before close
- close the bracket stack in reverse (`{` → `}`, `[` → `]`)
Conservative: never inserts content (no key names, no values, no
commas) — only closes / drops. Worst case the repair is a no-op
or makes parsing fail in a different way; never silently
fabricates data.
Apollo live test result: UNGROUNDED 0/0 (lost answer) →
UNGROUNDED 0/1 (partial first claim recovered with evidence_ids
intact, partial second claim dropped). The user sees the model's
attempted answer in the unverified bucket instead of empty space.
The audit chain captures the repair via fixups labels
(close_string, close_brace, close_bracket, drop_partial_key,
strip_trailing_comma) so an auditor can see what the parser had
to peel.
Pipeline ordering in _lenient_json_parse: strict parse first,
then peel (fence/prose-trim/curly-quotes/trailing-comma), then
strict parse again, then truncation repair as last resort. Each
step's fixups append to the same list so the verdict's
`json_fixups` field is comprehensive.
Bench question set picked up "name simpsons family members
including pets?" — a multi-part list with adjacent-relationship
shape, addresses a gap in the corpus probe.
460 unit tests pass; 12 skipped (live fixtures unchanged).
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Adds bench/qa_sweep.py + bench/qa_questions.txt + `make bench-qa` target.
Sweeps a fixed question set through quote and claim_lattice_pointer modes,
N samples per cell (default 3), each sample burns the cached record so
Hermes nondeterminism becomes the variance source.
Outputs:
bench/qa_results/<utc>.jsonl one row per (mode, question, sample)
bench/qa_results/<utc>.md summary + per-question vote counts
Question set spans the failure-mode shapes the verifier needs to handle:
narrow factoid, broad descriptive ("tell me about X"), entity list,
relationship, and out-of-corpus (should land UNGROUNDED honestly).
Methodology gap this closes: at n=1 the strict-rate swings ±20pp on a
fixed prompt purely from sampling noise. n=3 makes 5pp deltas legible.
aborist/progress.py — stdlib-only rate-limited stderr reporter.
Periodic lines (default every 2s) showing:
elapsed | seen | inserted | docs/s now | docs/s avg | percent | ETA
Wired into ingest_source via a `progress` parameter; CLI default-on
with --quiet to suppress and --total-estimate N to enable percent/ETA.
Demo on a 5k-doc ingest with --progress-interval 0.8:
[ 1s] | 200 seen | 200 new | 193 now | ( 193 avg) docs/s | 4.0% | ETA 24s
[ 7s] | 3,200 seen | 3,200 new | 458 now | ( 424 avg) docs/s | 64.0% | ETA 3s
[ 12s] | 5,001 seen | 5,000 new | 392 now | ( 413 avg) docs/s | 100.0% | ETA 0s
bench/run.sh + `make bench` — reproducible 5000-doc workload through
three configs:
serial single process, single SQLite
parallel-shared N shards, one shared SQLite (WAL serialized)
attached N shards, per-shard SQLite (true parallel writes)
Output is a one-shot table plus CSV at /tmp/aborist-bench/results.csv
so the ratchet is visible as we keep optimizing. Override via
BENCH_DOCS=N and SHARDS=N.
Latest baseline (this commit, on this machine):
config wall_s docs docs/s
serial 11.74 5000 425.7
parallel-shared 10.39 5000 481.1
attached 7.99 5000 625.5
53 tests still passing.
