2026-05-19: huge-N #000057 control-arm sweep (f63b00d → 9dc02e4) burned
our Opus quota. Pin the policy into repo CLAUDE.md so future agents
don't reach for Opus on autopilot:
- Sweep + bench-qa default endpoints: Hermes-3-8B + Qwen via uncloseai
- Opus drops out of control-arm grids until fox explicitly re-adds it
- Existing bench/results/ data stays valid; just no new Opus expansion
- Translation still uses local pinned opus-mt (sandwich MT edges), not
Hermes/Qwen 8B
- Grok rejoin path noted as an HTML comment (paid spend, fox-gated)
Adds two subsections under '## Live endpoints':
- 'Budget discipline — Hermes / Qwen first, Opus deferred'
- 'Per-call model selection (general agent work, not sweep arms)'
fox: 'not 11 hours it shouldn't take that long'. Probed endpoints —
Qwen-27B absorbs 8 concurrent with 0 errors (0.5->2.1s); the
bottleneck is the serialized claude -p Opus judge, so more workers
≈ near-linear speedup. control_sweep.py gains --resume PATH: appends
to an existing JSONL, skips items already COMPLETE (full
models×variants for solo + variants for arborist if i<=arborist-n),
re-runs partial items; _aggregate now dedupes (i,arm,model,variant)
last-wins so a killed-mid-unit restart never double-counts, and
_load_recs tolerates a truncated trailing line from the kill.
Makefile control-sweep gains CONTROL_SWEEP_WORKERS / _RESUME / _ARB_N
so make stays the interface. 6-worker run killed cleanly (specific
pids, no pkill), relaunched resume @ 12 workers — 12 done items
preserved, 374 to run, ~5h -> ~2-2.5h.
fox: 'make the n huge huge, check in every ~7 turns'.
- stale fixture re-mined to the FULL pool: 386 questions (180 pres /
119 PM / 34 premier / 20 gov-gen / 17 chancellor / 15 CM / 1 FM),
deterministic.
- control_sweep.py rebuilt: ThreadPoolExecutor work-unit-per-item
(per-unit qa_db -> zero SQLite write contention; shards read-only
safe under concurrent readers), incremental flushed JSONL, and a
--report-only PATH mode that aggregates a PARTIAL file with ZERO
LLM/judge spend (the interim check-in path).
- huge N goes on the CONTROL (solo x3 models x3 framings) where the
open statistical question lives; Arborist A/B = fixed --arborist-n
(default 40), not re-measured 386x (power belongs on the control,
and query() over ~40GB shards is the heaviest call).
- bench-maxing doctrine applied: independent hermetic judge calls
fanned out, 'serial-by-caution is halting in disguise'.
N=3 (prior run) already shows the coherent, review-relevant story:
plain -> all 3 models confidently assert (0% abstain); source_relative
-> abstain jumps (hermes 33 / qwen-think 67 / qwen-nothink 100%);
as_of_corpus -> ~100% correct all models. The gap is largely
framing+snapshot, not a universal capability deficit.
fox ruling: 'we both do not know which framing is right, measure all
benchmarks and bring results forward for review'; 'we have qwen with
and without reasoning to use'.
bench/control_sweep.py: sweeps the CONTROL (solo) arm across
{Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning} ×
{plain, source-relative, as-of-corpus-era} questions, judged vs the
fixed corpus-vintage gold; Arborist-Hermes treatment reference run
alongside at the same N. Presents the SAME judge verdicts under both
the accuracy framing (the naive read fox flagged as unfair-as-truth,
shown for contrast) and the grounding-fidelity framing (the
defensible read), plus an explicit note on the faithfulness-ablation
framing + Arborist×Qwen — both deliberately NOT run (proof-path
surgery / different instrument, not a relabelling).
Answers fox's open question empirically: does a 27B *reasoning*
control honestly ABSTAIN where the 8B fabricates (gap = weak-small-
model artefact) or still confidently assert post-corpus (gap = real,
scale-independent)? as_of_corpus separates 'can't recall the era'
from 'won't constrain to a source'.
Qwen toggles probed live: reasoning answer in message.content (CoT in
separate reasoning_content, not surfaced — logged limitation);
no-reasoning via chat_template_kwargs{enable_thinking:false}. Gate =
in-script judge self_test (aborts on fail) — stronger than a make
edge, no double self-test spend.
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
control_ab.py called client.chat_completion(messages) but model is a
keyword-only REQUIRED arg → TypeError every call → solo answer was
always '[solo-error: TypeError]', which the hermetic judge correctly
classified ABSTAINED. Every prior 'Hermes-solo abstained' smoke
(incl. the n=2 the ticket cites as a case against the treatment) was
a broken-control-arm artefact, not a measurement. Fix: pass
model=a.model (symmetry with the Arborist arm). Error token now
carries the exception message so the next failure self-describes.
Also adds the #000057 'stale' fabrication-bait mining class
(time-varying executive offices; corpus = ~2010-2011 Wikipedia
VERIFIED from the artifact, NOT the 2003 dump CLAUDE.md names —
flagged) and restores the numeral fixture deterministically.
b573c59's accent-fold work removed 210 lines from
qa_questions_numeral_map.json (40 -> 5 entries) — collateral of an
earlier `mine_questions.py --cls numeral --limit 5` regression-check
that got committed. The control-ab run scored n=5 (not the --n 12
passed) because the fixture itself only had 5 rows. Re-mined at the
default --limit 40; deterministic mine yields a byte-identical
restoration of the a3ac653 original. Caught by reading the artifact,
not assuming.
bench/control_ab.py + `make control-ab` (gated on judge-self-test
as a make dependency — instrument gate cannot be skipped). Same
model both arms; gold = target-article text by target_root; Arborist
[E…] scaffolding stripped (blinding — format can't betray the arm);
Arborist UNGROUNDED credited as honest abstention; hermetic Opus
judge; deterministic aggregate; self-auditing JSONL; threats-to-
validity printed in the report.
N=2 smoke: clean end-to-end, 0 JUDGE_ERROR — and already surfaced a
case AGAINST the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG). The instrument can falsify the Arborist value claim;
that is the point. n=2 proves nothing (report says so) — verdict
needs a real N.
fox ruled judge = Opus via `claude -p`. bench/judge.py:
hermetic (`env -u CLAUDECODE claude -p`, fresh process, context =
only (Q, answer, gold) — no arm label, no Arborist context, no
session), blinded-by-caller, reference-grounded against the fixed
gold (ignore parametric knowledge), structured via FINAL_VERDICT=
sentinel parsed LAST-match.
Instrument-before-experiment gate worked: first cut parsed
first-match over the model's chain-of-thought → 0/3 self-test. The
judge REASONED correctly; the parser was the defect (+ two bad test
fixtures, my error). Hardened (sentinel contract + fixed fixtures),
re-verified: `make judge-self-test` = 4/4 on known-verdict triples
via real claude -p. The make target is the precondition gate; no
control run trusts the judge until it passes.
Threat to validity recorded, not hidden: same model family judging;
mitigated (blind + no-stake + reference-grounded) not eliminated —
different-family SOTA cross-check is the only full removal.
Next: bench/control_ab.py + `make control-ab` (Hermes-solo vs
Arborist, gold=target-article text, blinded, judged) — NOT yet
built; no broken make target shipped for it.
fox: should the judge get the same docs, or the 8 retrieved/cited?
Neither. Per-arm retrieved docs (a) are undefined for Hermes-solo
(no retrieval → breaks A/B symmetry), (b) re-import circularity
(judging the treatment vs its own retrieval), (c) measure citation
self-consistency (Arborist's verifier already does that), not
truth. Judge's reference = the mined question's fixed ground-truth
target/gold, identical for both arms, independent of retrieval —
the ground-truth-carrying fixture supplies exactly this. Recorded a
distinct, non-headline faithfulness ablation (judge Arborist vs its
own context) so it is never conflated with the control delta.
Index row unchanged (skim surface; ticket body is the design log).
I over-applied a proof-path rule. "No LLM-as-judge" guards Arborist's
*verifier* (an LLM in the proof path = the circularity Arborist
kills; #000049's poor reliability is a proof-path-model constraint).
The control experiment's judge is external science — outside BOTH
arms, scoring post-hoc, touching neither system's internals — so it
cannot break the experiment. fox: use a strong SOTA model
(stronger than both systems under test), blinded + reference-
grounded + transcript-logged. This lifts the wrongly-imposed
atomic-fact-only limit (judge handles essay grounding; #000049's
exclusion was right for the proof path, wrong for external science);
deterministic gold-key demoted to a cheap atomic cross-check.
Still fox calls: fixture population, which SOTA judge (API
cost/egress), spend. Scaffold; no build past gate-1.
The review's strongest point (§4.1) + fox: every measurement so far
is WITHIN Arborist — no control arm. Folded into #000057 (not a new
ticket): control = Hermes-solo; judge external/deterministic/
paraphrase-tolerant via a mined gold-fact answer key (resolves
circularity + no-LLM-judge + paraphrase at once); headline metric =
fabrication-vs-honest-abstention delta, not raw accuracy (common-
knowledge fixture would falsely show null — parametric memory).
Smallest-proof reorder: v1 = single-shot Hermes-solo vs Arborist on
a fabrication-bait atomic-fact fixture; recursive-drift loop is v2,
gated on v1. Atomic-fact only; essay-grounding = #000049's unsolved
NLI, OUT. Still fox calls: fixture population, gold-key shapes,
spend. Scaffold; no build past gate-1.
The 2026-05-19 GPT-5.5 Hamming-framed review's ONE arborist-scoped,
ticket-worthy nugget: prove the Merkle-Providence-Reverse-RAG
whitepaper's headline claim (untracked evidence loss -> unbounded
recursive drift; witness-preserving state bounds it). Scaffold only,
awaiting fox go/no-go on scope.
Discipline encoded from the 2026-05-18 precedent (a grand target is
unfalsifiable until the instrument can resolve it — four hypotheses
died, only the deterministic mined-recall instrument broke it):
instrument-before-experiment, ONE task not eight, minimal ON/OFF
A/B, non-claims pinned (necessary substrate, NOT AGI).
Everything else in the review (rename, corpus hierarchy, IQ/talent,
ToE/Riemann/identity/geopolitics) deliberately NOT ticketed —
narrative/positioning, not arborist engineering; don't-proliferate.
Exactly one ticket. Next ID 000057 -> 000058 (same commit).
The review's central, correct finding: _title_query_tokens is
hot-path (every query AND title) and its fold set (hyphen #000007 +
numeral/accent/honorific/brit) changes which documents retrieve, but
that normalization's version was bound nowhere → a replay cannot
identify which token-normalization produced an old providence
record's sources. Same provenance class as the #000001 keyword gap.
Severity is honest: replay-provenance gap, NOT cache corruption —
different folds → different sources → different context_root →
different cache_key, so no false answer-cache aliasing or false
STRICT. Verifier/proof path unchanged.
Fix follows the repo's OWN #000001 §5/§6 decision (bind retrieval
transforms into the run-DAG RetrievalPlan/retrieval_plan_hash, NOT
governance_policy_hash). The review suggested governance "Option A";
repo precedent is run-DAG binding (same status as retrieval_keywords
and #000056 MT-engine identity) — the discrepancy is surfaced for
fox as an explicit call, not silently overridden.
- RetrievalPlan.title_token_policy (empty default → omitted from
canonical() → every prior retrieval_plan_hash byte-identical; the
§5 zero-churn discipline, same as the #000056 MT fields).
- _TITLE_TOKEN_POLICY single source of truth in query.py, bound at
the plan construction site; bump on any fold change.
- Plus the review's edge cases: Roman-substring-in-word not folded,
out-of-range not folded, Unicode Roman explicitly unsupported,
hyphen∘numeral composition. Full suite 2498, 0 regressions.
Declined (not engineering, per don't-proliferate): the review's
SelfModel/MemoryRoot/5S-5T-5F/capital-ledger ceremony — the ticket
design log is the single source of truth; scope recorded there.
Two more MEASURED fold-search wins on mined ground-truth fixtures
(deterministic recall, no LLM), both lifting at @1/@3/@8 (not
coarse-k artifacts):
honorific (Mt/St/Dr <-> Mount/Saint/Doctor): recall@1 45% -> 75%
(+30pp), @8 62% -> 85%, misses 15 -> 6
brit (British <-> American spelling): recall@1 50% -> 70%
(+20pp), @8 70% -> 85%, misses 12 -> 6
Both _*_fold_variants are additive+symmetric, strict closed sets
(no English-word collision), no-op outside their class (verified
independent: brit no-ops on honorific titles & vice versa), unioned
into _title_query_tokens beside hyphen(#000007)/numeral/accent.
Full suite 2488 passed, 0 regressions (hot-path); real-path tests
(FakeSource->ingest->query()->real _Hit).
Fold-search FINAL across the survey backlog, ranked by MEASURED @1
headroom (not prevalence — the instrument's job):
SHIPPED: numeral (a3ac653) accent (b573c59) honorific brit (here)
NO-BUILD: hyphen — existing #000007 already delivers 90%@1
(the measure-the-unmeasured-thing check pays off)
NO-BUILD: amp — 82%@1 with no fold (prevalence-overranked;
instrument killed it cheaply, like digit-ordinal pre-build)
Net: 4 deterministic retrieval wins + a reusable mined-recall
instrument + the discipline codified in CLAUDE.md, from a goal that
4 prior hypotheses died on because the bench couldn't measure them.
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
Compressed headline: when a lever's failure class is ≤3-5/75 q the
curated n=3 audit_mode bench can't resolve it (four 2026-05-18
hypotheses died there). Fix the instrument: mine ground-truth-
carrying questions (bench/mine_questions.py) graded by deterministic
retrieval recall@k via `query --dry-run` (bench/recall_at_k.py) —
no LLM/verifier/n=3-noise/5pp-floor, ±1-q resolution. Caveats codified:
measures retrieval surfacing (necessary-not-sufficient for STRICT);
complements never replaces the curated adversarial trap gate.
The first MEASURED, above-noise retrieval win this thread. The 75-q
n=3 audit_mode bench couldn't resolve any single lever (every failure
class <=3-5 q, sub the 5pp floor — four hypotheses died there). Fix
the instrument, not just the lever:
- bench/mine_questions.py + bench/recall_at_k.py: mine questions from
corpus titles (ground-truth target known by construction), grade by
deterministic retrieval recall@k via `query --dry-run` — no LLM, no
verifier, no n=3 noise, scalable to the 22K-deep numeral pool. The
curated qa_questions.txt stays the separate verifier-honesty/trap
gate; mined fixtures measure the answerable long tail per class.
- _numeral_fold_variants in query.py: ordinal-word ("Alexander the
second") <-> multi-char Roman ("Alexander II"), additive+symmetric,
unioned into _title_query_tokens exactly like _hyphen_fold_variants
(#000007). Strict 2..40 Roman set → no English-word collision;
single-char Romans (I/V/X) intentionally out of scope (universal
len>1 token filter — stated before building, ~4 of 10 residual
misses).
Measured on the mined numeral fixture: recall@8 22/40 (55%) -> 30/40
(75%), +20pp; 20 hits now rank-1. Discipline applied end to end:
measured-first, mirrored precedent, full-suite regression run (2482
passed, 0 regressions — numeral-fold is hot-path in
_title_query_tokens), real-path test (FakeSource->ingest->query()->
real _Hit, not a hand-built object), measured-after on a noise-free
instrument. The ~6 multi-char residual misses are a different
downstream cause the instrument now exposes for future iteration.
The fr-no-mask sweep (2026-05-18) corrected an over-pessimistic read:
French was reported 47% but that was the failed-mask artifact; honest
fr no-mask is 61%. §9 + index now carry the firmed n=1 table (EN 85%
· es-nomask 71% · es+mask 65% · fr-nomask 61% · fr+mask 47%), the
comparator-corrected verdict (sandwich is a ~0%→61-71% net win vs the
real no-support baseline; the −14/−24pp is vs native English, not a
regression), and the entity-mask lever's confirmed net-negative
status (default-OFF). Doc-only; no code change.
Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
Previously linked to the bare domain, which serves a marketing page. The actual OpenAI-compatible endpoint is /v1; /v1/models is the clickable verification (returns the served model card on the live deployment).
Shard capacity convention is documented in arborist/search/fts5.py:113 ("~50ms cold per token on a 10GB shard"). The earlier "~2 GB for a Wikipedia-sized corpus" line in both pagers was a fabricated figure off by 20×. Replaced with the real numbers: per-shard ~10 GB design target, live deployment of four shards totalling ~38 GB and holding 3.5 M documents / 6.2 M chunks.
gitlab-runner rotates between build dirs (builds/RUNNER_ID/0, /1, /3, ...). The cached .venv/ embeds the absolute build-dir path into __editable__.arborist-0.0.1.pth + the dist-info RECORD via pip's editable install. When the cache is restored into a different build dir, pip's implicit uninstall-then-reinstall step fails with `OSError: No such file or directory` looking for files at the old build dir.
Fix: shared .warm-venv-setup hidden job referenced from every real job's before_script. It wipes arborist-*.dist-info, the __editable__*.pth marker, the finder, and the bin/arborist entry script before pip install — so pip does a fresh install at the current build dir. Cache stays warm (deps don't re-download), only the editable-install bookkeeping is rebuilt (~1s per job).
DRY'd via `!reference [.warm-venv-setup, before_script]` so test / bench-suite / substrate-score / test-crawler all share the same setup; each job adds only its own install line ('.[dev]' vs '.[dev,crawler]').
Old drafts read as internal substrate notes. Rewrites lead with what the system does for a consumer or evaluator and what it costs to run, with no references to internal tickets, table names, schema-version strings, governance hash dimensions, or per-record audit-mode tokens. Appendix diagrams updated in lockstep: friendly labels ("grounded / partly grounded / not grounded") replace the schema-column trichotomy, layer names paraphrased away from SURFACE/CORE/PROVIDENCE.
1-pager (docs/_source/arborist-one-pager.rst, 1 page) for AI-literate readers: the trichotomy, the 8-dim cache key, CTI synthetic-elision-impossible, soft-channel separation, real-traffic bench numbers (mis-cite 100% @ 0% FP, warrant 92/92, quote 0.54 STRICT-rate).
2-pager (docs/_source/arborist-two-pager.rst, 3 pages = 2 body + 1 appendix) for technical reviewers: letterhead, Permacomputer Preamble license box, six numbered sections, plus appendix figures (pager-arch-stack 3-layer architecture, pager-verifier-flow question→pointer→verifier→trichotomy).
Both pages live under docs/_source/ so the same RST renders into the Sphinx readthedocs site (toctree caption "Summary pages" added to docs/_source/index.rst) AND into standalone PDFs via rst2pdf (docs/pager.style, lazy install into .venv).
Makefile targets: docs-one-pager, docs-two-pager, docs-pagers, docs-pagers-clean. Diagrams render through the existing DOT pipeline.
Synthesizes the bench-maxing work across:
#000049 NLI recombination veto (bart-large-mnli/k=12/margin/θ=0.999
→ 48% real-haystack recall at 0/808 STRICT FP — partial closure,
Phase-3 semantic candidate selector for full closure)
#000052 §3.1 diagnose_coherence (lexical sidecar, advisory-only,
1.1% real-STRICT FP after round-2 patch)
#000052 §3.2 relevance reranker (bge-reranker-large + cleaned +
θ ≤ -2.42 → 100% mis-cite / 55% deflection / 0% STRICT FP —
motivating Zionist-shape failure fully covered)
The three are architecturally orthogonal (§3.2.2 step 3C verified:
combining lexical sidecars with the relevance reranker gives no
lift; each owns its own failure-shape slice). Three structurally
distinct demote-only signals layered on the binary verifier.
Three runtime-promotion decisions for fox+dav1d:
- §3.1: keep advisory or wire policy hook? (probably advisory)
- §3.2: promote at the fp=0 operating point? (sign-off folds
relevance_policy_hash into governance_policy_hash)
- #000049: promote at 48% partial closure, or wait for Phase 3
semantic candidate selector?
Bench-maxing methodology codified in CLAUDE.md is the transferable
artifact: 'clean candidate-bench can mis-predict in BOTH directions
— real-data fixtures on both precision AND recall axes are the
only load-bearing measurement'. Eight instances across the two
arcs; the discipline applies to any future model-based addition.
Indexed in docs/TICKETS.md 'Distinction from other docs' section as
a non-ticket reference doc. Production verifier unchanged; nothing
in audit_mode; all work SHADOW pending sign-off.
Ran diagnose_deflection and diagnose_coherence on the 20+20 NEG
fixtures from step 3 parts A+B, plus all 808 STRICT.
Result: 0/40 NEG fire on diagnose_deflection (because the fixtures
are token-overlap-correct by construction — the question's subject
appears in the answer; that's exactly the failure mode #000052 was
built to catch beyond lexical). 0/40 on diagnose_coherence (the
fixtures are well-formed sentences).
So:
UNION (relevance OR deflection) = relevance alone (no lift)
INTERSECT (relevance AND deflection) = 0/40
Multi-signal combination doesn't help on these failure shapes.
But the architectural finding is positive in a different way: the
sidecars and the relevance reranker cover NON-OVERLAPPING failure
shapes cleanly:
- diagnose_coherence: structural breakage (word-salad, vacuous,
phrase-component-reuse). Owns the 'incoherent answer' slice.
- diagnose_deflection: token-overlap mismatch (subject anchor not
in answer). Owns the 'wholesale topic drift' slice.
- bge-large relevance: semantic aboutness mismatch despite shared
tokens. Owns the 'topic-collision / mis-cite / on-topic-but-
not-answering' slice — what §3.2 was built for.
Each signal owns its own slice; combining is redundant on these
cases. That's the architectural validation of the §3.1 + §3.2 +
existing-lexical-sidecars split as orthogonal, not overlapping.
Manifest runtime_viability.as_multi_signal_factor updated from
'viable' to 'TESTED — does not lift; sidecars are complementary
not combinatorial on these shapes'. Cleaned up stale
nli-shadow-grid-n1-minilm.json.
Also cleaned up a stale nli-shadow-grid JSON.
The §3.2 arc is now complete:
step 1 candidate-bench (overclaim, contrived data)
step 2 real STRICT FP (over-pessimistic 'NOT VIABLE')
step 3A real-context deflection (positive reversal, 55% at fp=0)
step 3B real-context mis-cite (100% at fp=0, motivating failure covered)
step 3C multi-signal combination (no lift; clean architectural split)
The recommended operating point holds: bge-reranker-large + cleaned
+ θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.
Built bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl —
20 hand-crafted (claim, source) mis-cite pairs: claim about X, source
about Y, X≠Y but shared tokens. Each pair survives the lexical
title-relevance + verifier sidecars by construction. This is the
Zionist-entity failure mode (claim about a different entity than the
cited source, both lexically related).
Examples:
- 'Mercury is the smallest planet' / Roman-god Mercury source
- 'Java is a programming language' / Java-the-island source
- 'Apple Inc. was co-founded by Steve Jobs' / apple-the-fruit source
- 'The Eiffel Tower is in Paris' / Gustave-Eiffel-person source
- 'Mozart composed The Magic Flute' / Mozart-effect-theory source
Headline (bge-reranker-large + cleaned + θ ≤ -2.42):
Mis-cite catch: 20/20 = 100% ← FULL COVERAGE of motivating shape
Deflection catch: 11/20 = 55%
Combined NEG: 31/40 = 78%
Real-STRICT FP: 0/808 = 0% ← strictly safe
bge-large mis-cite scores: -9.37 to -3.81 (max). STRICT min: -2.42.
Mis-cite is STRICTLY SEPARABLE from real STRICT — there's a 1.4-pt
gap with no overlap. (Deflection harder; some overlap with weak STRICT.)
MiniLM-L-6 cost-pick (5× smaller, cleaned, θ=+3):
100% mis-cite + 65% deflect + 0.4% STRICT FP
Mis-cite is structurally MUCH easier than deflection — both models
hit 95-100% mis-cite catch at modest θ; deflection is harder because
'answer doesn't quite address question' can look like a weak STRICT.
That's appropriate: mis-cite is 'wrong topic entirely'; deflection
is 'right topic, not answering'.
Manifest:
- runtime_viability flipped (step 2 → step 3): NOT VIABLE → VIABLE
at the bge-large fp=0 operating point.
- demote_below_score still null pending fox+dav1d sign-off (setting
it folds relevance_policy_hash into governance_policy_hash per
#000049 §7 #2).
- PRIMARY RECOMMENDATION: bge-reranker-large + cleaned +
θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.
Production verifier unchanged; still SHADOW. The §3.2 arc went:
candidate-bench (overclaim) → real STRICT step 2 (over-pessimistic
NOT VIABLE) → real-context deflection step 3 (positive reversal,
55% at fp=0) → real-context mis-cite step 3 part B (full closure
at fp=0 for the motivating failure mode). Eight meta-lesson
instances over the §000049 + §000052 arc, with the sharpest one
yet: clean candidate-bench can mis-predict in BOTH directions —
real-data fixtures on BOTH precision and recall axes are the
only load-bearing measurement.
Built bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl —
20 hand-crafted deflection answers against real bench-qa questions
(each: coherent well-formed answer using context-tokens, but NOT
addressing the question — the kind of failure §3.2 was built for).
Real-context deflection scores (cleaned, MiniLM-L-6):
min -6.71, median +1.56, max +6.34
Real-context deflection scores (cleaned, bge-reranker-large):
min -5.57, median -3.02, max +5.01
vs STRICT (cleaned):
MiniLM-L-6: min -5.29, p10 +6.90, median +9.32
bge-large: min -2.42, p10 +3.56, median +6.48
Distributions are CLEAN-separable on real data — 0% of MiniLM
deflections score above p10(STRICT); bge-large median is -3.02 vs
STRICT median +6.48.
Pareto frontier (MiniLM-L-6, cleaned):
θ=-5.29: catch 10% / fp 0% (fp=0 floor, low signal but safe)
θ=+3: catch 65% / fp 0.4% ← strong runtime soft-veto
θ=+5: catch 90% / fp 1.9% ← aggressive soft-veto
θ=+7: catch 100% / fp 10.4% (too FP for runtime)
bge-reranker-large:
θ=-2.42: catch 55% / fp 0% ← VIABLE runtime soft-veto at strict fp=0
The step-2 'NOT VIABLE' verdict was an artifact of using the
candidate-bench NEG distribution (tight contrived band, overlapped
real STRICT) as the recall denominator. Real-context deflections sit
in a much lower band (median around -3 for bge-large) than real
STRICT, so absolute thresholds DO separate them cleanly. The
relevance reranker IS a viable runtime soft-veto on this design —
the candidate-bench-only step 2 measurement misled us.
Manifest's runtime_viability.as_runtime_demotion_veto flipped from
'NOT VIABLE' to 'VIABLE at low-to-moderate FP', with both per-model
Pareto frontiers recorded. demote_below_score stays null pending
fox+dav1d sign-off; setting it folds relevance_policy_hash into
governance_policy_hash per #000049 §7 #2 discipline.
Still SHADOW; production verifier unchanged; no audit_mode effect.
Eighth instance of the meta-lesson, with the lesson sharpening AGAIN:
clean candidate-bench can MIS-PREDICT in BOTH directions — over-
optimistic on threshold (step 2) AND over-pessimistic on viability
(step 3 reversal). Real-data fixtures are the only load-bearing
denominators on either axis.
Tested Q→A score vs A→context_lead score on 808 pooled STRICT pairs
with cleaned MiniLM-L-6. The hypothesis was: deflected answers
should have HIGH A→ctx_lead but LOW Q→A (still grounded but
off-topic). Reality:
Q→A quantiles: p10 +6.90 p50 +9.32 p90 +10.57
A→ctx quantiles: p10 -6.73 p50 +1.51 p90 +8.68
Δ=Q→A-Actx: p10 +0.90 p50 +6.95 p90 +14.62
Of the 73 cleaned STRICT-fires at cb-θ=6.754:
- 71 (97%) have BOTH axes low (co-varying — not deflection)
- 2 (3%) match the deflection signature; both are the same
'who wrote GNU linux?' replica
Why it doesn't separate: the A→ctx axis isn't measuring what the
hypothesis assumed. A focused-claim against a 30KB topic-broad
Wikipedia haystack scores LOW by default — that IS the normal
STRICT shape (the answer is one clause in a sprawling document).
The cross-encoder expects the document to be ABOUT the query
(MS-MARCO retrieval shape); a STRICT (answer, full-haystack) pair
violates that. So both axes co-vary and the delta is a poor
discriminator.
This closes the third precision-side rescue path I'd left open
after step 2's verdict:
✗ absolute threshold — universal walk-back across 6 models
✗ cleaning preprocessing — helps but doesn't separate
✗ contrastive Q→A vs A→ctx — doesn't separate either
The relevance reranker conclusively cannot be a runtime
demotion-only veto on this design. The soft-signal uses
(render-tail, multi-signal advisory, etc.) remain viable.
runtime_viability.as_contrastive_signal updated from 'worth
measuring' to 'tested, doesn't separate'. Production verifier
unchanged; no audit_mode effect.
Hand-inspection of the bottom-15 STRICT-fires from the raw §3.2.2 step 2
sweep showed claim-lattice overlay markup ([E\d+ | title | hash: '…'])
depressing scores on correct concise answers (the 6× Henry-VIII case),
while true-positive deflections (broad-question / narrow-answer like
'winners of all major sports?' → just-one-sport) remained correctly
low-scored. So the noise FP class is the bracket metadata; cleaning it
should reduce FP without losing true-positive signal.
Built clean_for_relevance() in arborist/qa/relevance/shadow.py — strips
[E\d+ | ... ] blocks + trailing '...']' tails. Baked into
ShadowRelevance.check_question_answer / check_claim_source by default
(opt out with clean_input=False). relevance_shadow_sweep.py applies it
to inputs before _score_batch (opt out with --no-clean).
Re-ran the full 6-model sweep on the 808-cell pooled STRICT with
cleaning:
bge-reranker-large 21.8% → 18.6% (-3.2)
MiniLM-L-4-v2 15.0% → 9.4% (-5.6 pts, -37% rel)
MiniLM-L-6-v2 11.6% → 9.0% (-2.6)
MiniLM-L-12-v2 11.0% → 8.0% (-3.0)
bge-reranker-base 9.5% → 9.0% (-0.5)
MiniLM-L-2-v2 4.5% → 1.5% (-3.0 pts, -67% rel)
Universal improvement, every model better. Big surprise: MiniLM-L-2-v2
— the model that FAILED the candidate-bench separability (margin
-1.97, declared 'capacity floor') — has the LOWEST real-traffic FP
rate at its own cb θ (1.5%). Because L-2's compressed score range
gives it a low cb θ which few real STRICT pairs score below.
SEVENTH instance of 'candidate-bench doesn't predict real-traffic'.
Runtime-veto verdict UNCHANGED — still not viable; smallest fp=0 θ on
real STRICT is below the cb NEG max for every model, so at any
runtime-safe θ the catch on cb NEG is 0/12. But cleaning is now FREE
improvement for any soft-signal / advisory / contrastive use of the
relevance score. Hand-inspected bottom-10 post-cleaning confirms true-
positive deflection signal preserved.
All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12)
false-fire on 4.5–21.8% of real bench-qa STRICT at their
candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real
STRICT is BELOW the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG
= 0/12 across the board.
Structural reason: real bench-qa STRICT answers have a much wider
score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50
+5.80, p90 +7.54) than the tight contrived candidate-bench POS
band. The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. Distributions overlap heavily; no
threshold separates them.
This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back —
clean candidate-bench → fails real-pipeline gate. Same diagnosis:
lexical-candidate selection + cross-encoder scoring + hard threshold
doesn't survive real-pipeline heterogeneity.
Verdict: relevance reranker CANNOT be promoted to a runtime
demotion-only veto on this design. demote_below_score stays null;
manifest gains runtime_viability block documenting the negative
result + the still-viable advisory soft-signal uses (render-tail,
multi-signal advisory, contrastive Q→A vs claim→source delta).
bench/scripts/relevance_shadow_sweep.py + bench/results/
relevance-shadow-sweep-pooled808.json committed. Production
verifier unchanged; advisory only; no audit_mode effect.
Five rule tightenings, each targeting a specific bench-qa STRICT
false-positive shape (the xfail regressions from the previous commit):
1. claim-lattice bracket-artifact skip — sentences matching
`\[E\d+\s*\|` (pointer markup) or `..."\]` (truncation tail)
are no longer parsed as natural-language assertions; `Such a
thesis was..."]` no longer fires vacuous.
2. Circular-rule differentia cap — circular now requires the
predicate to be (a) entirely vacuous OR (b) leads with a subject
token AND has ≤ 2 non-subject non-filler differentia tokens.
"Michael Jordan's Restaurant was a restaurant in Chicago,
Illinois, named after the basketball player Michael Jordan"
has 6 differentia → no longer fires. Pre-existing positive test
"The entity is the entity referring to the State of Israel"
has 2 differentia (state, israel) → still fires (under threshold).
3. phrase_component_reuse translation-chain exception — when the
predicate ALSO contains a quoted phrase (translation /
definition / etymology context), token reuse with the subject's
quoted phrase is legitimate, not circular. "The name 'Rosebud
River' is a translation … 'the river of the roses'" no longer
fires.
4. Vacuous-rule short-acronym escape — `_coherence_predicate_has_short_acronym_content`
recognizes title-cased element-symbols (Au, Fe, Pb…) and all-caps
2-5-char acronyms (DNA, FBI, USB, NASA…) as content even though
they're below the ≥3-char content-token filter. "The chemical
symbol for gold is Au." no longer fires; "Iron has the chemical
symbol Fe." also clean; tautology "DNA stands for DNA." still
correctly flagged circular.
5. (the 'term <X>' idiom xfail stays xfail — borderline, no clean
lexical fix.)
5 xfail → passing (the regression coverage is now executable proof
of fix); 1 xfail remains. Pooled bench-qa STRICT FP rate test ceiling
tightened from 7% to 2%. All 7 pre-existing positive coherence tests
still fire correctly. 110 total tests pass; 1 xfailed; no regressions
on doc-counts / nli / relevance.
§3.1 diagnose_coherence (now 19 tests, +10 from the parallel session's 9):
- 3 more positive shapes (multi-sentence vacuous, named-entity circular,
grammar-term phrase_component_reuse).
- 5 xfail regression tests for SHAPES THAT FALSE-FIRE on real bench-qa
STRICT data (44/808 = 5.4% FP rate measured on the pooled n=1+3+5
STRICT answers). Each xfail names the exact shape + why it should
ideally be 'ok' + which rule needs tightening:
* 'The chemical symbol for gold is Au.' → vacuous (short predicate)
* 'Michael Jordan's Restaurant was a restaurant ... named after
Michael Jordan.' → circular (named-after re-use)
* 'The Western X was the western half of the X' → circular
* 'The name <Phrase> is a translation ... of the <derivative>' →
phrase_component_reuse (translation/etymology)
* claim-lattice [E1 | … …"] tails → vacuous (truncated bracket
fragment)
* 'The term <X>' → phrase_component_reuse (idiomatic English)
- 1 load-bearing real-traffic test: FP rate on 808-cell pooled STRICT
must stay ≤ 7% (current 5.4%) — fires loud if a future change
regresses it. Skips on fresh-checkout (bench/qa_results/ gitignored).
§3.2 ShadowRelevance (now 20 tests, +7 from the round-1 scaffold):
- Manifest tests for round-2 primary (bge-reranker-large), the size
spectrum coverage (50-560MB), the candidate-bench findings block
(biggest-within-family / not-across-families / deeper-not-better /
capacity-floor).
- Pair-kind distinction (question_answer vs claim_source recorded
separately for downstream telemetry / governance hashing).
- Batch-order preservation (_score_batch must return scores in input
order — load-bearing for downstream zip-back).
- Empty-input handling (Q empty, D empty, whitespace-only).
- Zionist-entity discriminator sanity (on-topic > off-topic logit).
- demote_below_score-stays-null invariant (the §7 #18→#27 discipline:
no hardcoded threshold; must come from a real-traffic shadow sweep).
Total: 101 passed + 6 xfailed (5 §3.1 regressions documented + 1 from
parallel session). The 5 xfails are the bench-maxing receipts — they
document EXACTLY which shapes §3.1 false-fires on, with the rule that
needs tightening named in each reason.
`make bench-qa BENCH_QA_N=3 BENCH_QA_LIMIT=5` (2026-05-13T14:24Z) on
the bench/qa_questions.txt set: 30/45 STRICT (67%), zero regressions
on mona lisa / capital of france / new london bridge (each 9/9 STRICT
across quote / pointer / lattice modes). The bench-qa-smoke n=1
flickers I saw ("dinosaurs" S→U, "soviet union" S→H) were LLM
stochasticity at n=1, not retrieval bugs. GNU-linux + python failures
look like genuine hard questions, not Phase-2 side effects.
End-to-end gap-close from the Phase 1 extractor. Five interlocking
fixes; live-verified that `what is a CPU?` → "Central processing unit"
at #1, `what is a GPU?` → "Graphics processing unit" at #1
EVIDENCE-WARRANTED 1/1; Mount Kilimanjaro / Soviet Union queries
unchanged (no regression).
(a) `synonym_expand` over-cap path is now rank-and-truncate by
source-frequency (descending) instead of hard-skip. CPU has 13
legitimate homonym expansions across the corpus; the prior
MAX_NEIGHBORS_PER_TOKEN=8 cap contributed *zero* expansion → no
canonical-article surfacing. Now: keep the 8 dominant by per-(token,
target) source-root count via _load_neighbor_source_freq.
(b) `_search_titles` orders by FTS5 bm25 ASC instead of LENGTH(title)
ASC on the FTS5-MATCH path. The length-asc tie-break was correct for
the 2026-05-02 "Back to the Future" LIKE-substring case but
counter-productive on FTS5 (tokenized; no substring junk; length-asc
preferred "Unit" / "Unite" / "B unit" over "Graphics processing
unit"). LIKE fallback keeps length-asc since the substring issue
persists there.
(c) `accept_tokens` (feeds title-search, core-keyword, title-rerank)
uses the synonym-expanded set instead of qtokens-only. The Phase 1
expansion existed but was only used in the FTS5 OR-fallback; satellite
articles saturated the budget before the canonical article entered.
(d) ARCHITECTURAL: `synonym_expand_strict()` (new — high-trust
evidence-kind subset: manual + manual_legacy + acronym_parens,
**excludes** link_reciprocity) for use in the multiplicative
`_rerank_by_title_purity` and as the source for `accept_tokens`.
Reciprocal-wikilink edges express *topical adjacency*, not synonymy
(a `Dinosaurs` page reciprocally links to `Curious George Brigade` →
edge that should not amplify retrieval); a multiplicative ranker over
them blows up. Strict view preserves the acronym-parens surfacing
(those edges ARE the phrase=expansion identity) while keeping
link-reciprocity to additive retrieval-route boosts via the broad
synonym_expand (still wired to `or_synonym_pool` for the FTS5 OR
fallback).
(e) Extractor regex tightened `[A-Z]{2,6}` → `[A-Z]{3,6}` and purged
~21K 2-letter acronym edges from shards. 2-letter acronyms (AI/ML/OS/
US/UK/IT/PC/TV) homonym-collide too often with common 2-letter QUERY
tokens like `go`/`is`/`am` — without this, "why did the dinosaurs go
extinct?" pulled Curious George Brigade via GO-acronym edges. The
high-value acronyms (CPU/GPU/RAM/DNA/FBI/WHO/…) all clear 3 chars.
Also: CLAUDE.md gains a "prefer existing ticket; only split for
Dav1d-review audience" discipline note (saved as feedback memory) —
this work is itself an example: would have been #000055 + #000056 +
#000057 under the prior pattern; instead extends #000054.
Suite: 2531 passed (no regression). bench-qa in flight separately.
All 4 manifest candidates clean-separate (12/12 NEG catch at 0/14 POS FP),
so 'reranker discriminates Zionist-entity-style mis-cites' is a property
of MS-MARCO-trained rerankers as a class — not the specific L-6 I picked
first. Ranked by separation margin (per §7 #18: separation beats raw):
ms-marco-electra-base +5.349 ← new primary
BAAI/bge-reranker-base +3.076
ms-marco-MiniLM-L-6-v2 +3.053 (previous primary, demoted)
ms-marco-MiniLM-L-12-v2 +2.663 (worst — deeper ≠ better)
Manifest primary moved to electra-base for the cushion. demote_below_score
STAYS null — clean candidate-bench thresholds don't predict real-pipeline
behavior (the §7 #18→#27 history is 6 verdict flips on the NLI side);
§3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) is
what sets it. Expect a walk-back. Reranker still doesn't catch the
Kilimanjaro/Mount-Kenya recombination (aboutness ≠ truth-of-attribution;
that's #000049 territory). fox's bench-maxing correction applied.
arborist/qa/relevance/ — manifest pins cross-encoder/ms-marco-MiniLM-L-6-v2
(~80MB, Apache) as primary; alternates: L-12, BAAI/bge-reranker-base,
ms-marco-electra-base. demote_below_score=null on purpose — the
#000049 §7 #18→#27 discipline (proved 6× that clean-eval thresholds
don't transfer to bench-qa data) requires the threshold to be set by a
shadow sweep against pooled real STRICT, not by a literature number.
ShadowRelevance class mirrors ShadowNLI (lazy [nli]-extra import, cuda
auto-detect via ARBORIST_RELEVANCE_DEVICE or ARBORIST_NLI_DEVICE,
batched _score_batch, graceful degrade-to-available=False). Two surface
methods: check_question_answer (deflection / Q-A drift) and
check_claim_source (topic-collision mis-cite). 13 tests.
Sanity on the motivating field case (Zionist entity): ON-topic +9.96
vs OFF-topic -9.04 → 18-pt margin. Mona Lisa Q→A deflection: on +10.45
vs deflect +3.56 → ~7-pt margin. The model CLEANLY discriminates the
failure modes #000052 §1 named. It does NOT catch the
recombination-where-the-different-entity-clause-also-mentions-the-target
case (Kilimanjaro/Mount Kenya) — and that's the right architectural
split: aboutness (#000052 §3.2) and entailment (#000049 NLI) are
orthogonal axes; the Kilimanjaro recombination case needs the semantic
candidate selector (#000050/#000051 vec hybrid).
Remaining: build candidate-bench eval (~20-30 deflection + mis-cite
fixtures), shadow-sweep θ over pooled bench-qa STRICT (expect another
walk-back per the #000049 lesson), recall-side realism check, then
fox+dav1d sign-off. Still SHADOW; production verifier unchanged.
test_concepts_extract.py grew 20 -> 28 with the acronym_parens
extractor's 8 tests; the AUTOCOUNT discipline (CLAUDE.md) fires the
doc-counts regression on stale numeric claims.
`arborist/concepts/extract.py:acronym_parens_synonym` — new
corpus-agnostic extractor. Scans each doc's lead chunk (first 4000
chars) for `<Multi-Word Phrase> (ACRO)` where the all-caps acronym's
letters strictly match the content-word initials of the phrase, in
order, after function-word filtering. Emits bidirectional synonym
edges between the lowercased acronym and each ≥3-char content token
of the phrase, evidence_kind="acronym_parens", anchored to that doc's
document_root. Idempotent like link_reciprocity_synonym.
Why this complements link_reciprocity: Wikipedia represents
abbreviation→expansion as a one-way *redirect* (CPU →
Central processing unit), which the ingest does not record as an
edge — so the existing reciprocal-link extractor never learned the
relation. The relation IS in body text by near-universal convention
("Central processing unit (CPU) is..."), which this extractor reads.
Corpus-agnostic: HTML, blogs, textbooks benefit equally.
Conservative: strict 1:1 acronym-to-atom match (rejects HTTP-shape,
where letters land mid-word), function words filtered, repeated
definitions deduped per doc, ≥3-char target floor. 8 new tests
covering CPU bidirectional emit, RAM idempotency, FBI function-word
filter, HTTP length-mismatch reject, XYZ initial-mismatch reject,
ROM hyphenated-word handling, per-doc dedupe, registry presence.
Retrieval-side only — synonym edges reshape FTS5 candidate selection
via synonym_expand at query time, never enter audit_mode / cache_key
/ audit_event_hash. No governance hash bump, no cache invalidation.
Closes#000050 §2a's CPU/GPU abbreviation rows *upstream* of vec;
the Orwell-shape conceptual-allusion row remains the genuine #000050
justification. Operational follow-up (not code): run on each shard
via `arborist concepts derive --extractor acronym_parens` (CLI
surface itself is aspirational in docstrings; extractors are called
programmatically today). Next ID 000054 -> 000055.
Built bench/fixtures/5f/recombination-realcontext-v1.jsonl: 25 hand-built
recombination claims (deliberate re-attributions within real bench-qa
30 KB Wikipedia haystacks the pipeline produced — Mount Kenya into a
Kilimanjaro answer, Galileo onto all 63 Jupiter moons, Joshua Reynolds
onto the Mona Lisa, etc.). Ran the full 7-model × full-grid sweep vs
the 808 pooled STRICT. Result:
- bart-large-mnli / k=12 / max / θc=0.999 → 12/25 = 0.48 catch · 0/808 FP
(the real-haystack fp=0 leader)
- deberta-large-mnli / k=3 / margin / θc=0.995 → 6/25 = 0.24 (§7 #26's
'settled' config — 28/28 synthetic, 0.24 real-haystack: 4× over-estimate)
- roberta-large 0.12, MiniLM 0.08, deberta-base 0.04
So §7 #26's 'boundary closed' walks back to 'boundary PARTIALLY closed'
on real haystacks. The bottleneck is architectural: top-k by token
overlap misses the contradicting clause when it shares few subject-area
tokens with the answer (e.g. the Mount Kenya clause only shares 'Kenya'
with a Kilimanjaro claim — ranked low, NLI never sees it). Threshold
tuning doesn't lift the ceiling; a SEMANTIC candidate selector
(vec-driven, sibling of #000050/#000051's hybrid retrieval) does.
bart's pareto above fp=0: fp=0.011 catch=0.52, fp=0.057 catch=0.84,
fp=0.068 catch=0.92 — permissive operating points are on the menu if
fox+dav1d sign off. recommended_operating_point updated to
bart-large-mnli/k=12/max/θc=0.999; deberta-large/margin kept as the
synthetic-eval reference. Sixth meta-lesson instance: clean synthetic
eval doesn't predict bench-qa precision OR recall — neither contrived
dataset axis is load-bearing, only the real pipeline shape is.
Production verifier unchanged; falsification-hard stays 10/12. Still
SHADOW; runtime promotion fox+dav1d-decides.
(a) Mined the pooled n=1+3+5 bench-qa runs (808 distinct STRICT answers)
for natural recombinations at lowered θc≥0.7 → 37 would-fires, ALL
token-collision FPs on inspection (Mount Kenya pulled into a Kilimanjaro
answer, Dalí into da Vinci, Donovan into Superman). ZERO genuine
recombination errors — the boundary is theoretical-in-practice; the
failure mode is the candidate selector (top-k by token overlap) pulling
different-entity same-subject-area clauses.
(b) Re-ran the grid against the 808-cell pooled STRICT set: the §7 #25
'max@0.96' was itself a small-sample artifact — the n=5 444-cell set
lacked the high-confidence spurious hits the pooled set has. On 808
cells θc goes back to ~0.995, and at θc=0.995 only agg=margin still
catches 28/28 (max gets 27/28). microsoft/deberta-large-mnli / k=3 /
agg=margin / θc=0.995 → 28/28 synthetic recombinations · 0/808 pooled
real STRICT FP · 0/26 synthetic legit — the ONLY config in the
7-model×full-grid sweep that hits 1.0/0.0 on 808 cells, held at n=3
too. recommended_operating_point reverted to margin@0.995.
Realistic next check: ~20-30 hand-built synthetic-recombination-vs-
real-bench-qa-context fixtures (real haystack, deliberate re-attribution).
Still SHADOW; runtime promotion fox+dav1d-decides. Production verifier
unchanged; falsification-hard stays 10/12.
Meta-lesson instance five: a bigger sample can vindicate a config a
smaller one made look unnecessary — re-confirm the config choice (not
just the threshold) each time the denominator grows.
