russell@unturf.com
933a4f4752
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
44 lines
1.5 KiB
Text
44 lines
1.5 KiB
Text
# aborist QA-quality benchmark question set.
|
|
#
|
|
# One question per line. `#`-prefixed lines and blank lines ignored.
|
|
# Designed for the 2003-05-16 Wikipedia cur snapshot — questions must
|
|
# resolve against ~2003 article content. Stress-tests across the
|
|
# failure-mode shapes the verifier needs to handle.
|
|
|
|
# narrow factoid — well-anchored single-fact questions
|
|
who founded apple computer?
|
|
what is the capital of france?
|
|
who wrote the linux kernel?
|
|
when was the python programming language created?
|
|
who painted the mona lisa?
|
|
|
|
# broad descriptive — encyclopedic shape, prone to mode-collapse
|
|
tell me about connecticut
|
|
tell me about the python programming language
|
|
tell me about charles babbage
|
|
tell me about the apollo program
|
|
|
|
# entity list — invites lazy-anchor on a magnet chunk
|
|
list the founders of microsoft
|
|
what dinosaurs were in the first jurassic park film?
|
|
who are the members of the beatles?
|
|
|
|
# relationship / multi-fact
|
|
who is supermans girlfriend?
|
|
who is bilbo baggins's nephew?
|
|
what is the relationship between linux and unix?
|
|
|
|
# comparison — multi-entity, prone to attribution drift
|
|
what's the difference between linux and bsd?
|
|
how does intel compare to amd?
|
|
|
|
# niche / partial — corpus may be thin
|
|
what is the boltzmann constant?
|
|
who invented the doppler effect?
|
|
|
|
# leading / adversarial — model should refuse over-claim
|
|
isn't it true that the great wall of china is visible from space?
|
|
|
|
# out-of-corpus — should land UNGROUNDED honestly
|
|
who is the prime minister of mars?
|
|
what year did the cold fusion breakthrough happen?
|