All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12)
false-fire on 4.5–21.8% of real bench-qa STRICT at their
candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real
STRICT is BELOW the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG
= 0/12 across the board.
Structural reason: real bench-qa STRICT answers have a much wider
score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50
+5.80, p90 +7.54) than the tight contrived candidate-bench POS
band. The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. Distributions overlap heavily; no
threshold separates them.
This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back —
clean candidate-bench → fails real-pipeline gate. Same diagnosis:
lexical-candidate selection + cross-encoder scoring + hard threshold
doesn't survive real-pipeline heterogeneity.
Verdict: relevance reranker CANNOT be promoted to a runtime
demotion-only veto on this design. demote_below_score stays null;
manifest gains runtime_viability block documenting the negative
result + the still-viable advisory soft-signal uses (render-tail,
multi-signal advisory, contrastive Q→A vs claim→source delta).
bench/scripts/relevance_shadow_sweep.py + bench/results/
relevance-shadow-sweep-pooled808.json committed. Production
verifier unchanged; advisory only; no audit_mode effect.
All 4 manifest candidates clean-separate (12/12 NEG catch at 0/14 POS FP),
so 'reranker discriminates Zionist-entity-style mis-cites' is a property
of MS-MARCO-trained rerankers as a class — not the specific L-6 I picked
first. Ranked by separation margin (per §7 #18: separation beats raw):
ms-marco-electra-base +5.349 ← new primary
BAAI/bge-reranker-base +3.076
ms-marco-MiniLM-L-6-v2 +3.053 (previous primary, demoted)
ms-marco-MiniLM-L-12-v2 +2.663 (worst — deeper ≠ better)
Manifest primary moved to electra-base for the cushion. demote_below_score
STAYS null — clean candidate-bench thresholds don't predict real-pipeline
behavior (the §7 #18→#27 history is 6 verdict flips on the NLI side);
§3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) is
what sets it. Expect a walk-back. Reranker still doesn't catch the
Kilimanjaro/Mount-Kenya recombination (aboutness ≠ truth-of-attribution;
that's #000049 territory). fox's bench-maxing correction applied.
Built bench/fixtures/5f/recombination-realcontext-v1.jsonl: 25 hand-built
recombination claims (deliberate re-attributions within real bench-qa
30 KB Wikipedia haystacks the pipeline produced — Mount Kenya into a
Kilimanjaro answer, Galileo onto all 63 Jupiter moons, Joshua Reynolds
onto the Mona Lisa, etc.). Ran the full 7-model × full-grid sweep vs
the 808 pooled STRICT. Result:
- bart-large-mnli / k=12 / max / θc=0.999 → 12/25 = 0.48 catch · 0/808 FP
(the real-haystack fp=0 leader)
- deberta-large-mnli / k=3 / margin / θc=0.995 → 6/25 = 0.24 (§7 #26's
'settled' config — 28/28 synthetic, 0.24 real-haystack: 4× over-estimate)
- roberta-large 0.12, MiniLM 0.08, deberta-base 0.04
So §7 #26's 'boundary closed' walks back to 'boundary PARTIALLY closed'
on real haystacks. The bottleneck is architectural: top-k by token
overlap misses the contradicting clause when it shares few subject-area
tokens with the answer (e.g. the Mount Kenya clause only shares 'Kenya'
with a Kilimanjaro claim — ranked low, NLI never sees it). Threshold
tuning doesn't lift the ceiling; a SEMANTIC candidate selector
(vec-driven, sibling of #000050/#000051's hybrid retrieval) does.
bart's pareto above fp=0: fp=0.011 catch=0.52, fp=0.057 catch=0.84,
fp=0.068 catch=0.92 — permissive operating points are on the menu if
fox+dav1d sign off. recommended_operating_point updated to
bart-large-mnli/k=12/max/θc=0.999; deberta-large/margin kept as the
synthetic-eval reference. Sixth meta-lesson instance: clean synthetic
eval doesn't predict bench-qa precision OR recall — neither contrived
dataset axis is load-bearing, only the real pipeline shape is.
Production verifier unchanged; falsification-hard stays 10/12. Still
SHADOW; runtime promotion fox+dav1d-decides.
(a) Mined the pooled n=1+3+5 bench-qa runs (808 distinct STRICT answers)
for natural recombinations at lowered θc≥0.7 → 37 would-fires, ALL
token-collision FPs on inspection (Mount Kenya pulled into a Kilimanjaro
answer, Dalí into da Vinci, Donovan into Superman). ZERO genuine
recombination errors — the boundary is theoretical-in-practice; the
failure mode is the candidate selector (top-k by token overlap) pulling
different-entity same-subject-area clauses.
(b) Re-ran the grid against the 808-cell pooled STRICT set: the §7 #25
'max@0.96' was itself a small-sample artifact — the n=5 444-cell set
lacked the high-confidence spurious hits the pooled set has. On 808
cells θc goes back to ~0.995, and at θc=0.995 only agg=margin still
catches 28/28 (max gets 27/28). microsoft/deberta-large-mnli / k=3 /
agg=margin / θc=0.995 → 28/28 synthetic recombinations · 0/808 pooled
real STRICT FP · 0/26 synthetic legit — the ONLY config in the
7-model×full-grid sweep that hits 1.0/0.0 on 808 cells, held at n=3
too. recommended_operating_point reverted to margin@0.995.
Realistic next check: ~20-30 hand-built synthetic-recombination-vs-
real-bench-qa-context fixtures (real haystack, deliberate re-attribution).
Still SHADOW; runtime promotion fox+dav1d-decides. Production verifier
unchanged; falsification-hard stays 10/12.
Meta-lesson instance five: a bigger sample can vindicate a config a
smaller one made look unnecessary — re-confirm the config choice (not
just the threshold) each time the denominator grows.
ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=5 → 1125 cells, 444 real
STRICT (5x n=1, 1.6x n=3). Re-ran the 7-model mega-grid: the lexical-
candidate NLI veto robustly clears the §7 #12 gate with
microsoft/deberta-large-mnli / k=2 / agg=max / guard=max_entail /
θc≈0.96 / θe=0.9 → 28/28 synthetic recombinations (incl. both fixtures)
· 0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts θc headroom;
roberta-large-mnli equally good (k=2/max/θc=0.95). Resolved: agg=max +
max_entail guard is the robust score-shape (§7 #24's margin win was a
sample tie); the large checkpoints (~350-400M deberta-large-mnli /
roberta-large-mnli) hit 1.0/0.0, bart-large (similar size) only ~0.71,
deberta-base at the cliff (27/28 here, 11/28 at n=3), small models
(MiniLM-82M, deberta-v3-small) cap at ~0.82 — so §7 #18's 'MiniLM is
the cost-pick' is OVERTURNED by the proper-n evidence; k=2 consistent
winner; int8-ONNX costs ~1 catch. recommended_operating_point updated.
Remaining: a bench-qa-derived recombination set (the one check not
done); n=9 if dav1d wants more; still SHADOW; runtime promotion is
fox+dav1d-decides. Production verifier unchanged; falsification-hard
stays 10/12. This run is the worked example behind CLAUDE.md's new
bench-maxing line.
ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3 → 275 real STRICT cells
(3x the n=1 sample). Re-ran the expanded grid (7 aggregations incl.
margin = max_clause(p_contra - p_entail), paired-entail guard variant,
θc to 0.999, --extra-models) over all manifest models + 4 extra
xsmall→large (microsoft/deberta-large-mnli, roberta-large-mnli,
nli-deberta-v3-small, deberta-v3-xsmall), synth-28 recombination vs the
275 STRICT cells. Result: the §7 #23 deberta-base/k=2/θc=0.99 config
does NOT survive — it catches only 11/28 at θc=0.995 (which the larger
STRICT sample forces). BUT the broader sweep found the config that does:
microsoft/deberta-large-mnli / k=3 / agg=margin / θc=0.995 → 28/28
synthetic recombinations (incl. both fixtures) + 8/12 falsification-hard,
0/275 real STRICT FP, 0/26 synthetic legit FP — a passing config at
proper n. Findings: margin is the right score-shape (single threshold,
folds the guard in); the specific checkpoint matters more than param
count (deberta-large-mnli wins clean, deberta-base collapses,
roberta/bart ~0.71-0.75 — no 'bigger is better' law). recommended_operating_point
updated. Still SHADOW; runtime promotion needs a bigger STRICT sample +
a bigger recombination set + fox/dav1d sign-off. Production verifier
unchanged; falsification-hard stays 10/12.
Meta-lesson sharpened twice: clean eval ≠ bench-qa precision (§7 #18→#20),
default config ≠ best config (§7 #22→#23), small FP sample ≠ large FP
rate (§7 #23→#24).
The §7 #22 'fails the gate' was the verdict for the DEFAULT config
(k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set), not the
approach. A {model × candidate-cap k × aggregation × θc × θe} grid
sweep (bench/scripts/nli_shadow_grid.py — NLI runs once per
(model,record) over the top-12 candidate clauses, the k/agg/θ grid is
then arithmetic on cached scores; ~10s on the 4090 for 4 models) finds
clean passing configs: on the 89 real STRICT cells (n=1 bench-qa),
deberta-base-184M / k=2 / agg=max / θc=0.99 / θe=0.9 → 27/28 synthetic
recombinations caught (incl. both 5f-fal-hard fixtures), 0/89 STRICT
FP. MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M >
407M for fp=0 recombination recall; int8-ONNX costs ~1 catch vs fp32.
Caveats: FP side is n=1 (BENCH_QA_N=3 run in flight); recall is on the
synthetic set; flipping to a runtime demotion-only veto is fox-decides
(then nli_policy_hash folds into governance_policy_hash per §7 #2).
Manifest active defaults stay k=6/θc=0.5; recommended_operating_point
(deberta-base, k=2, agg=max, θc=0.99, θe=0.9) documented in the
manifest. 3 grid result JSONs committed. Standing lesson: neither the
clean synthetic eval NOR the default config predicts bench-qa precision
— you have to sweep. Production verifier unchanged; falsification-hard
stays 10/12.
First grid (MiniLM, n=1 bench-qa 89 STRICT cells): the §7 #22 'fails the
gate' verdict was config-specific (defaults k=6/θc=0.5/θe=0.9) — the grid
finds k=1/θc=0.93/θe=0.5 → catch 8/12 falsification-hard (incl. BOTH
recombination fixtures hard-003 Mercury + hard-005 Einstein), 0/89 STRICT
FP, 0/26 synthetic-legit FP; 18/28 synthetic recombination. So the
lexical-candidate approach is NOT a dead end. (k=1 = no haystack;
θe=0.5 tighter than the clean-set 0.9.) Full multi-model + n=3
characterization next.
Speedup (§3 plan): ShadowNLI._nli_batch batches forwards
(ARBORIST_NLI_BATCH=64); device auto-detect (ARBORIST_NLI_DEVICE, else
cuda-if-available); auto-prefer an ONNX export — bench/scripts/export_nli_onnx.py
/ make export-nli-onnx exports + int8-dynamic-quantizes the pinned
checkpoint into ~/.arborist/models/nli/<ver>/onnx/ (operator state, NOT
committed), _ensure_loaded loads model_quantized.onnx via
optimum.onnxruntime (backend onnx-int8), falls back to torch silently.
torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair (~4x);
seconds on a 4090. optimum[onnxruntime] added to the [nli] extra; 24
tests.
Gate-item-4 verdict at proper n: ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=1 → 223 cells (89 STRICT / 90 HYBRID / 44 UNGROUNDED; also
surfaced + fixed a lone-surrogate bug). Shadow sweep over those: NLI-as-
runtime-veto on STRICT has ~26% FP at θc 0.5, ~8% at θc 0.90, ~0% only
at θc 0.99 — and θc 0.99 gives up most recombination recall (hard
synthetic recombinations bottom out ~0.76). FAILS the §7 #12 gate on
this design. Only untried path that might pass: a Phase-3 runtime hook
running NLI on the verifier's actual matched clauses (1-3), not
top-6-by-overlap. Until then: runtime NLI demotion stays off; the 2
fixtures stay permanent boundary markers; θc stays 0.5. Production
verifier unchanged; falsification-hard stays 10/12.
ARBORIST_NLI_SHADOW=1 carries the raw verifier-input text into bench
rows; real Wikipedia context occasionally has U+D800–U+DFFF code points
(mangled source encoding) that json.dumps(..., ensure_ascii=False) then
refuses to UTF-8-encode → the run died at row 224/225. Scrub via
encode('utf-8','replace').decode() — U+FFFD is fine for a measurement
field. Only the two new shadow fields are touched.
candidate_clauses() — NLI now runs only on the top-N source clauses by
content-token overlap with the answer claim (max_candidate_clauses=6),
not the whole context; records n_candidate_clauses / best_clause_overlap
/ recombination_risk. Synthetic sweep unchanged (28/28 recombination,
0/26 legit FP, mean 1.45 candidate clauses/record). Real-traffic smoke
re-run: STRICT would-demote 30% → 20%, overall 47% → 33% — better, not
fixed; recombination-risk split doesn't separate either. Residual STRICT
false-contras at ~0.83-0.92 → θc would need ≈ 0.90 (vs the clean-set
0.5); at θc=0.90 the data in hand gives 27/28 synthetic recall, 0/26
legit FP, 0/10 smoke STRICT FP — but n=10 is too small to set on.
Next: a fuller ARBORIST_NLI_SHADOW=1 bench-qa run → sweep θc on hundreds
of STRICT cells → confirm → set it. θc stays 0.5; runtime NLI demotion
stays off. Production verifier unchanged; falsification-hard stays 10/12.
Live hook: ARBORIST_NLI_SHADOW=1 makes query() surface the verifier-input
text (gated off-by-default, never a cache_key/governance/audit_mode
input); qa_sweep.py carries it + the answer into bench rows; the shadow
sweep reads them and buckets by audit_mode. ARBORIST_NLI_SHADOW=1 make
bench-qa-smoke (15 cells) → the naive 'NLI on every context clause'
scaffold has a ~30% would-demote rate on STRICT answers — a haystack /
multiple-comparisons artifact (real contexts segment into 100-336
clauses; max-over-all almost always finds a tangential clause the model
reads as contradicting; a paraphrased STRICT answer often isn't verbatim-
entailed by any single clause so the entailment guard doesn't rescue it).
Lesson: the §7 #5 'candidate source clauses' + recombination-risk gating
is load-bearing, not optional. Do NOT enable runtime NLI demotion on the
current scaffold; next step is the candidate-clause restriction, then
re-run, then gate item 4 is meaningful. Production verifier unchanged;
falsification-hard stays 10/12.
arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.
First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
make bench-emergent EMERGENT_N=30 — 30 fresh 3-word-triangulation
cycles appended to bench/emergent_log.jsonl.
Aggregate: 29/30 UNGROUNDED, 1/30 HYBRID, 0 STRICT — the expected
healthy null result. The triangulator generates absurd questions
("How does the depth of one's music-making abilities on a kazoo …")
with no real corpus support; the verifier ladder honestly returns
UNGROUNDED rather than hallucinating-then-STRICTing. 0 false-positive
STRICT on garbage input.
The one HYBRID (perversely/succeeds/disagreeably) is a re-run of
failure shape E ("total deflection to unrelated trivia"): the common
words pulled Hirabah / Original-righteousness chunks via FTS5; the
model quoted them verbatim (3/4 claims verify) but the answer is
total topic-drift → DEFLECTION_DETECTED + CITATION_MISMATCH → capped
at HYBRID, not STRICT. The deflection sidecar + verbatim verifier
holding on adversarial-by-construction input — nothing new, logged as
a confirmation.
No new failure shape or correctness concern. #000006 amend records
it; the 30 entries join the --print-pending queue if a future
teacher-review pass wants a closer look. Doc + log only.
Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl.
The parser was line/bullet-only — one line ⇒ one claim — so a line
that crammed several pointered claims onto one row ("Water is wet
[E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became
one monolithic claim with all the pointers, and a wrapped bullet
became two.
arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';',
sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes
(' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since
'/' while ', inline '(N)' enumeration markers, and commas — with
'(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it.
_segment_line keeps the split ONLY IF every resulting non-empty
segment is a well-pointered claim — a legit single claim ("The cat
is black and white [E1].", "The cast: A, B, C [E1].") is never
broken because splitting it would manufacture pointer-less prose
fragments → guard rejects; a leading colon-terminated header with no
pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a
wrapped-bullet join: a continuation line (leading whitespace then a
lowercase letter, no bullet glyph) folds its text + pointers into the
previous claim.
Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling
— a harder Formulate tier would re-open below-ceiling headroom; a
#000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE
recombinations in falsification-hard (Mercury, Einstein — step 2.2).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims
feeds the 450 claim_lattice_pointer + claim_lattice cells) after
(bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline
(...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55,
pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise
floor. Per-row diff: the segmenter changed the parsed-claim count on
the SAME answer text for 7 of the 450 lattice cells (0 in
claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an
audit_mode change — both correct: a wrap-join recovered an answer's
intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID)
into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1
monolithic claim, all pointers → STRICT) split into 8 claims, some
not individually verifying → HYBRID (the honest verdict — false-
positive STRICT was the corruption). Every other lattice/quote delta
is LLM re-answer variance. No regression — the segmenter's only
visible effects on real traffic are honest improvements. Summarized
in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4.
Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction
splits; pointerless-fragment + cast-list guards; leading-colon-header
drop; wrapped-bullet join; pointer-order/multi-pointer); existing
parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack
re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped.
#000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes updated.
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in
falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a
multi-word proper noun matches the source — but "Insulin was
discovered by Alexander Fleming" against "Penicillin was discovered by
Alexander Fleming" matches on the shared "Alexander Fleming" while the
swapped subject "Insulin" (the falsehood) is ignored.
arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx)
flags a >4-char Capitalized content token (stopword-filtered) or a
digit-number in the answer absent from the source.
_is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in
verify_quotes' entity branch (proximity policy) in the weakest-grounding
slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence
AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is
what keeps a structured multi-claim summary untouched — the Matrix cast
list (many entities, a tight cluster) and the TMNT answer (a numbered
list with parenthetical nicknames the source omits): model-added
accurate detail in a real summary isn't a contradiction, only the
single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid
entity-path regression tests still pass, pinned untouched.
Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin
/ 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it
newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard
pack around) and 5f-fal-live-028 — had expected_reason updated
HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes
actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE
recombinations (Mercury, Einstein — step 2.2) + 8 Formulate
mis-segments (step 2.4).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after
(bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline
(...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54,
pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise
floor. Per-row diff (675 common cells, 30 quote-mode rows changed
audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity
path — the gate fired on 0 legitimate QA answers in the whole bench.
Every transition was LLM re-answer variance (verifier quote→quote with
the verdict flipping); pointer/lattice deltas are noise too (the gate
is in verify_quotes / quote mode, not the claim-lattice verifier). No
regression — the gate is provably narrow on real traffic. Summarized in
qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1.
Tests: 4 new in test_verify.py (_is_single_sentence helper,
_entity_salient_disagrees helper, swapped-subject → UNGROUNDED,
gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling
re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated
(positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5
dilution; ACCEPT via a degraded-parent sub-scenario).
make test 2343 passed, 28 skipped.
#000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes / baseline JSON updated.
Closes the harder-fixture-tier ticket: a real verify_quotes
tightening lifts the falsification-hard rate 4/12 → 6/12, bench-gated,
ForkScore's bench-Δ goes positive on it — the loop is closed
end-to-end.
arborist/qa/verify.py: _numeric_signature(text) extracts comma-
stripped digit-runs ('8,849' and '8849' collapse; '300' stays
distinct from '300000' ← '300,000'). _check_each_with_paraphrase
gains a gate: a span that token-covers the source ≥ paraphrase_coverage
but asserts a digit-number the source lacks (modulo thousands-comma)
is no longer paraphrase-grounded — it goes to unverified. Catches the
near-miss the lexical coverage check is blind to ("Water boils at 50
degrees" against a source saying 100 token-covers 100% because
'50'/'100' aren't >4-char content tokens). Narrow by construction:
fires only on the paraphrase fallback (verbatim/span/entity/claim-
lattice paths untouched), only on a digit-number. A rounding-
paraphrase demoting here is the honest verdict — it isn't a verbatim
grounding.
Hard pack: lifts 5f-fal-hard-004 (50 vs 100) and -007 (300 vs
300,000) to UNGROUNDED → falsification-hard rate 4/12 → 6/12 = 0.5.
The other 6 over-grounds (4 HYBRID_ENTITY + 2 recombined-no-number
STRICT_PARAPHRASE) and the Formulate hard pack are unaffected —
headroom for a bigger order/dependency-aware verifier upgrade, an
optional follow-up, not a #000046 blocker.
Bench gate: make bench-qa (n=3 × 75 questions × 3 modes = 675 cells)
before / after. STRICT-rate quote 0.53→0.50, pointer 0.23→0.25,
lattice 0.44→0.45 — all within the 5-pp noise floor. Per-row diff
(675 common cells, 74 changed audit_mode): the only clearly
gate-attributable QA shift was the fictional "our cold fusion
breakthrough" year-claim demoting STRICT→HYBRID (×3 samples) — a
correct demotion; every other transition was quote→quote /
claim_lattice→claim_lattice LLM re-answer variance. No regression on
legit answers. Artifacts: bench/qa_results/2026-05-11T13-42-38Z (before)
and ...T14-19-51Z (after) — gitignored; summarized in qa-modes-bench.md
Addendum 5 + ticket-000046 §5 Phase 3.
Worked example: fork_score on the real change — parent {5f/
falsification: 4/12} → child {5f/falsification: 6/12} → γ·Δ5f =
(1/6)/5 ≈ +0.033 > 0 (positive; a single improvement of this size is
MARGINAL by the ÷5 dilution, the test pins the full-lift-to-1.0 case
at ACCEPT).
Tests: 5 new in tests/test_verify.py (numeric_signature normalization
+ subset-matches-comma-variant + disagreement-rejected +
gate-is-narrow + number-present-still-verifies);
test_5f_falsification_hard_pack_below_ceiling re-pinned 4/12 → 6/12.
make test 2339 passed, 28 skipped.
#000046 → closed; #000012 §8 §4 + TICKETS.md row + falsification-hard
_meta / hard-004,007 notes + Makefile 8/12→6/12 comments updated.
(Makefile also carries an uncommitted chain-check-SQL improvement from
the concurrent session — NOT in this commit; staged only the
#000046-comment hunks.)
Extends the #000046 hard tier to the Formulate sub-battery.
bench/fixtures/5f/formulate-hard-v1.jsonl — 12 prose inputs that
arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a
particular claim lattice (recorded in expected_lattice). The parser
is line/bullet-based — one line ⇒ one claim, [E#] tokens attach to
it — so 8 of 12 it mis-segments: merges and-/semicolon-/dash-joined
or (1)(2)-enumerated multi-claim lines into one claim with all the
pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD
on claim-count mismatch; the other 4 are well-formed bullet/numbered
lists / single claims the parser handles right. Rate at HEAD = 4/12
= 0.333, stable (parse_pointer_claims is deterministic). A
claim-lattice parser that does sentence/clause segmentation (split on
'. ', ';', subordinating conjunctions, inline enumerations) + joins
wrapped bullets lifts the rate toward 1.0 → positive γ·Δ5f for that
child fork.
make bench-5f-formulate-hard runs it (|| true past the runner's
nonzero-on-failures exit). Not in `make bench-5f` / `runner --all`.
tests/test_bench_batteries.py — test_5f_formulate_hard_pack_below_ceiling
(pins rate 4/12, source=live, the 8 fails are claim-count
mis-segments).
#000046 → "Phase 1 + Phase 2 landed"; two below-ceiling 5F subs now
exist (falsification, formulate). Closure still pending an actual
surface improvement (verify_quotes tightening — bench-gated — or
parse_pointer_claims segmentation) that lifts a rate. §5 + §6 +
TICKETS.md row updated.
(Makefile also carries an uncommitted chain-check-SQL improvement
from the concurrent #000039 session — NOT included in this commit;
staged only the bench-5f-formulate-hard hunk + the .PHONY line.)
The #000025 §10.14 calibration showed _delta_5{s,t,f} mean over a
battery's 5 subs, so a single-sub gain weighs 1/5 of face value (the
5× dilution). #000047 ships the knob to pick the aggregation, default
unchanged.
WeightSet.delta_aggregator ∈ {"mean","max","sum"} (default "mean") —
a categorical field, validated in __post_init__ against
DELTA_AGGREGATORS; from_dict takes it as a string. Default unchanged →
ScoredFork output byte-identical → no fork_score.ESTIMATOR_VERSION
bump.
fork_score._aggregate(deltas, how): mean = arithmetic mean, max =
max(0.0, max_i Δ_i), sum = Σ Δ_i; empty → 0.0. _delta_5s/_delta_5t/
_delta_5f take an aggregator arg (default "mean"); the 5F efficiency
bonus is added after the aggregated base (aggregator-independent).
fork_score passes weights.delta_aggregator. The per-sub
HARD_REGRESSION_FLOOR flags are computed before aggregation, so a
single-sub regression still forces REJECT under max/sum. The chosen
aggregator is recorded in ScoredFork.weights["delta_aggregator"] (via
WeightSet.as_dict()); fork_score_branches traceability stays via the
opaque weights_id — no schema migration.
bench/scripts/fivef_threshold_calibration.py gained §5 — runs the
#000046 below-ceiling pack (5f/falsification at 0.333) and shows the
verdict / γ·Δ5f under each aggregator; bench/results/5f-threshold-
calibration-2026-05-11.md §5 is the captured record. Default stays
"mean" — the conservative, noise-robust, regression-symmetric choice
matching docs/bench-maxing.md's per-rate floor framing; v8 picks
max/sum per-deployment.
Tests: 8 new in tests/test_fork_score.py + 1 anchor in
tests/test_fivef_threshold_calibration.py; tests/test_weights.py
as_dict field-set test updated to include delta_aggregator;
test_fork_score.py AUTOCOUNT tags (#000012 §286, warrant-substrate-
cookbook.md ×2) bumped 23 → 31.
#000047 closed; #000012 §8 §3 + TICKETS.md row updated.
Full suite: 2330 passed, 28 skipped.
Closes the "everything is at rate 1.0 so fork_score's bench-Δ terms
are inert" gap the #000025 §10.14 calibration surfaced — at least on
the 5F/falsification axis.
bench/fixtures/5f/falsification-hard-v1.jsonl — 12 near-misses, each a
FALSE/unsupported claim whose correct verdict is UNGROUNDED (recorded
in expected_reason). 8 of 12 are over-grounded by
arborist.qa.verify.verify_quotes at HEAD — its paraphrase
token-coverage strategy returns STRICT_PARAPHRASE, its entity-proximity
strategy returns HYBRID_ENTITY, both matching on incidental overlap
(shared entities/numbers, the same key terms stated in the opposite
direction) — so those tasks fail by design; the other 4 the verifier
handles correctly. Rate at HEAD = 4/12 = 0.333, stable (verify_quotes
is pure-lexical / deterministic). Built around the pre-documented gap
5f-fal-live-003.
make bench-5f-falsification-hard runs the pack; make
bench-fork-baseline-hard pins it to
bench/results/baseline-falsification-hard.json. Both targets `|| true`
past the runner's nonzero-on-failures exit (8 fixtures fail by design;
the JSON is still written).
tests/test_bench_batteries.py — test_5f_falsification_hard_pack_below_ceiling
(pins rate 4/12, source=live, every fixture asserts UNGROUNDED, the 8
fails are over-grounds not abstentions) +
test_fork_score_positive_gamma_5f_on_hard_falsification_improvement
(the worked example: fork_score(parent={5f/falsification: 1/3},
child={5f/falsification: 1.0}) → gamma*Delta5f ≈ +0.133 > 0, verdict
ACCEPT, no regression flags — the bench Δ-rate carrying signal it
can't carry while every canonical pack is at ceiling).
NOT in `make bench-5f` / `make bench-5s5t5f` / `make
bench-fork-baseline` / `runner --all` — the hard pack is a separate,
deliberately-failing artifact pinned on its own.
#000046 flipped to "in progress · Phase 1 landed"; closure pending an
actual verify_quotes tightening that lifts the rate (a separate,
larger task). ticket-000012 §8 §4 + TICKETS.md row updated.
Full suite: 2314 passed, 28 skipped.
"One more iteration then close" (fox): added committed KAT-regeneration
scripts for both the T3 calculator and φ_PRG — the regen step was a
throwaway temp script before; now it's reproducible and the phi_prg
test's skipif reason ("run scripts/generate_phi_prg_kat.py") points at
a file that exists. Then closed#000036.
New scripts:
- scripts/generate_t3_bound_kat.py — regenerates
bench/fixtures/t3-bound/known-answer-tests.jsonl from a fixed 12-config
list (the §7 worked examples under max_envelope + non-default-C_B*
+ g=0 edge + explicit-b1_model pins for the other three models).
- scripts/generate_phi_prg_kat.py — regenerates
bench/fixtures/phi-prg/known-answer-tests.jsonl from a fixed 10-entry
list (placeholder/random seeds, one-bit-flip variants, block-boundary
dim_h=16/17, 4096 counter-rollover stress).
- Both verified to reproduce the committed fixture data lines byte-
for-byte (only the header comments changed, to reference the script).
Each docstring states: run after any algorithm change, then bump the
module version (CALCULATOR_VERSION / PHI_PRG_VERSION) so the fixture's
version field changes too.
Doc/test:
- test_t3_bound_calculator.py skipif reason now references the regen
script (matches the phi_prg test pattern).
- #000035 §3.3 + t3-bound.md §10.1 reference the regen scripts.
Closure (#000036):
- Status → closed · 2026-05-11 in the ticket file + TICKETS.md row.
Phase 1 + dav1d Tier-1/Tier-2 (Option B in v1) + KAT-regen tooling
all landed; all §5 acceptance criteria met; both dav1d closure
blockers cleared. Continuation: empirical C_B1/C_B2/C_B3 tightening
under #000043 (parks on v7 deployment data); landing the bound's
framing into a v7 plastic-training spec parks on that spec gaining
a deployment target; R2's architectural integrations (Merkle audit-
event commitment, SQD canonicalization, CTI clause-lattice, 5F
trigger, ForkScore security-risk) are separate tickets if wanted.
- t3-bound.md header flipped to "closed 2026-05-11".
Full suite: 2312 passed, 28 skipped.
v7's canonical integer byte-order was confirmed little-endian by
inspecting merkle-agi-dag_v7.txt §A1 — every to_bytes / astype in the
TLV encoding is little-endian (TLV length prefixes to_bytes(4,'little'),
enc_int to_bytes(8,'little'), quantized tensors '<i8'); no big-endian
anywhere. Per dav1d's 2026-05-11 review rule ("if v7 TLV canonical
integer encoding is little-endian, flip §3.4 to little-endian before
KAT freeze"), flip done — this is the -le variant.
Implementation (arborist/substrate/anchor_prg.py):
- PHI_PRG_VERSION → "phi-prg-v1-hmac-sha512-le" (still "v1";
the -le suffix records the endianness; future re-flip MUST bump).
- _expand: counter.to_bytes(4, 'big') → 'little'.
- _bytes_to_floats: int.from_bytes(..., 'big') → 'little' (the
uint32-word interpretation, for full consistency with v7).
- Module + function docstrings updated: little-endian throughout,
with the merkle-agi-dag_v7.txt §A1 verification note.
- Note: at counter=0 the bytes are identical regardless of
endianness, so 5 of the 10 KAT entries (dim_h ≤ 16, single block)
keep the same output_sha256; the 5 multi-block entries (dim_h 17/
32/64×3/4096) change.
KAT fixture (bench/fixtures/phi-prg/known-answer-tests.jsonl):
- Regenerated under the little-endian counter. Each entry now also
carries a "version" field (phi-prg-v1-hmac-sha512-le). Header
comment updated.
Tests (tests/test_anchor_prg.py, 30 → 31):
- test_module_exports_version_string: assert the -le suffix.
- test_bytes_to_floats_midpoint_maps_to_zero: 2^31 is b'\x00\x00\x00\x80'
in little-endian, not b'\x80\x00\x00\x00'.
- New test_bytes_to_floats_reads_little_endian: pins the byte-order
so an accidental re-flip is caught.
- test_phi_prg_first_block_matches_direct_hmac: uint32-word reads
little-endian (counter=0 bytes unchanged either way).
- test_phi_prg_known_answer_tests: assert kat['version'] == module
version when present.
Spec text (#000035 §3.4): folded the little-endian variant of
dav1d's §9.10 wording — counter_le32, uint32_le word reads, an
"all integers little-endian, matching v7 TLV §A1" preamble, and an
"Endianness — RESOLVED 2026-05-11" note replacing the open
big-vs-little question. soft-hash-channel-analysis.md §9.2/§11 +
#000035 status + TICKETS.md row updated. AUTOCOUNT for
test_anchor_prg.py bumped 30 → 31; PHI_PRG_VERSION refs in docs
bumped to -le.
Full suite: 2312 passed, 28 skipped.
Closes the three open Phase-1b items of #000025; every §10 closure
criterion is now met, so the ticket flips to closed.
§10.14 — ForkScore threshold-calibration handoff to #000012.
bench/scripts/fivef_threshold_calibration.py (make bench-5f-threshold-
calibration) runs the canonical 5S/5T/5F packs + the 5F live packs and
reports baseline rates, observability granularity (1/n), and fork_score
verdicts on the parent vs synthetic child perturbations →
bench/results/5f-threshold-calibration-2026-05-11.md. Findings written
into ticket-000012 §8: keep SIGNAL_FLOOR / HARD_REGRESSION_FLOOR at
0.05; the small 5S packs (syntax n=10, semantics n=8) are coarser than
the floors so any regression there trips hard-reject (intended zero-
tolerance); the 5x averaging dilution in _delta_*; ceiling saturation
(every pack at 1.0 -> delta-rate terms <= 0). No constant change
shipped. 6 tests in tests/test_fivef_threshold_calibration.py.
§10.13 — feedback latency / efficiency on real workload.
run_feedback_loop now computes feedback_latency (listed in §5.5 since
Phase 1a, never implemented) — wall-clock seconds to apply a live
chain against its temp shard, surfaced per-task
(feedback_latency_seconds) + battery (feedback_latency_mean_seconds,
feedback_live_task_count). For live chains feedback_efficiency's cost
denominator switched from len(chain) (count of requested ops) to the
persisted footprint _persisted_cost = audit-event rows the chain
actually wrote + their body bytes / 1e6. Embedded chains keep
len(chain) and report feedback_latency_seconds = None. Latency is a
wall-clock field (run-to-run variable, like BatteryResult.timestamp)
and is not a fork_score input. 3 tests in tests/test_bench_batteries.py.
§10.11 — real selfmodel finetuning chains.
bench/scripts/selfmodel_chain_snapshot.py (make bench-5f-selfmodel-
snapshot) appends one chained SelfModel snapshot per run to a
persistent shard (~/.arborist/shards/selfmodel-chain.db, override via
ARBORIST_SELFMODEL_CHAIN_DB) with one CapabilityClaim per sub-battery
(metric = "5S-syntax" etc., measured_value = that pack's rate,
eval_digest = the pack's fixture digest, threshold = SIGNAL_FLOOR).
snapshot() auto-parents, so each snapshot is a distinct root and the
lineage grows by one per run. run_finetuning gains a third dispatch
mode — shard-chain (gated on a task's selfmodel_shard key) — via
_chain_finetuning_measure: reads the two most-recent snapshots
(latest() = child, its parent_selfmodel_root = parent) and measures
improvement on target_capability between them. This is the real
lineage replacing Phase-1a's synthetic parent->child pairs; the
chained delta reflects genuine cross-run drift (0.0 today — the
embedded packs are at ceiling). Operator pack
bench/fixtures/5f/finetuning-shardchain-v1.jsonl (6 tasks) + make
bench-5f-finetuning-shardchain; not in `make bench-5f`, `make test`,
or a fresh checkout (a missing/too-short chain fails honestly). The
real chain shard was bootstrapped 2-deep on 2026-05-11; make
chain-check-shards reports 0 breaks on it (and all other shards).
10 tests in tests/test_selfmodel_chain.py.
Full suite: 2311 passed, 28 skipped.
Per fox: apply the conservative max_envelope B1 model by changing the
v1 calculator's default — NOT by forking a v2. CALCULATOR_VERSION stays
"t3-bound-v1-bottou-refinement" (the descriptor names the unchanged B3
term); b1_model is echoed in the output AND the inputs dict so KAT
replays are unambiguous about which model produced a row.
Calculator (bench/scripts/t3_bound_calculator.py):
- New b1_model kwarg + --b1-model CLI flag, choices:
max_envelope (default) max(fraction_channels, aggregate_bias)
fraction_channels g · W · log₂(1 + G/σ)
aggregate_bias W · log₂(1 + g·G/σ)
effective_control_v1 g · W · log₂(1 + g·G/σ) (old non-worst-case)
- Default is now max_envelope — genuinely upper-bounding across both
interpretations of g (dav1d review §3 closure blocker, RESOLVED).
- Every output reports all three concrete B1 variants
(B1_fraction_channels / B1_aggregate_bias / B1_effective_control_v1),
b1_selected, and both SNR readings (snr_grad = g·G/σ,
snr_per_channel = G/σ) regardless of which b1_model was requested.
- model_assumptions[] now carries f"B1_model_{b1_model}".
- inputs echo now includes c_b1/c_b2/c_b3/b1_model (replay-complete).
- Invalid b1_model rejected with a ValueError naming the field.
- Baseline I_window: 625.8716 (effective_control_v1) → 6183.0154
(max_envelope: B1=aggregate_bias 5849.63 dominates fraction_channels
1729.72), certification_status NOT_CERTIFIED_BY_BOUND at W=10000.
KAT fixture (bench/fixtures/t3-bound/known-answer-tests.jsonl):
- Regenerated 2026-05-11 — 12 entries: the 8 §7-derived configs under
the new max_envelope default, a g=0 edge case, plus explicit-mode
pins for effective_control_v1 / fraction_channels / aggregate_bias.
- Each entry carries b1_model, expected_b1_selected,
expected_b1_{fraction_channels,aggregate_bias,effective_control_v1},
expected_snr_per_channel, expected_certification_status.
Tests (tests/test_t3_bound_calculator.py, 75 → 83):
- test_t3_bound_known_answer_tests no longer skips (fixture active);
pins b1_model, b1_selected, certification_status + numbers, tolerates
optional new fields on older fixtures.
- New: test_b1_max_envelope_exact_formula, test_invalid_b1_model_rejected,
test_cli_b1_model_flag (effective_control_v1 / fraction_channels /
aggregate_bias). test_b1_exact_formula renamed
test_b1_effective_control_v1_exact_formula and now passes the explicit
model. Updated baseline / below-256 / CLI tests for the new numbers.
Doc (docs/soft-hash-channel-t3-bound.md):
- Header + §0 + §3.1 + §6 + §7 (worked examples) + §8 (operator
guidance W-solving) + §10 (closure blockers RESOLVED) + §10.1 +
§11 (calculator schema) + §12 all updated for the max_envelope
default. §8: target-256 W drops from ~4196 to ~415 steps under the
conservative model — the ~10× cost of not assuming which g-reading
holds; operators who can measure effective-control applies can use
--b1-model effective_control_v1 for the looser W (a calibration
claim they must justify, not a default).
Status (#000036 ticket + TICKETS.md): both prior dav1d closure
blockers cleared (B1 worst-case model + active KAT fixture); remaining
= fox's final close-or-iterate call.
AUTOCOUNT markers bumped 75 → 83. Full suite: 2288 passed, 28 skipped.
trigger_1_branch_density in bench/prometheus_sigma_trigger_probe.py read
the fork_score_branches table via branch_set_density() instead of the
"density check not yet implemented" stub. It groups rows by branch_set_id,
fires when the most-recently-recorded checkpoint carries >= 4 branches
(BRANCH_DENSITY_FLOOR), and surfaces n_checkpoints / latest_density /
max_density / n_checkpoints_clearing_floor in the markdown report so
section 12's "regularly" qualifier stays visible. Density sums across
shards per branch_set_id.
With no branch sets persisted yet the probe reports "table present but
empty across shards" (data_available True, fires False) rather than a
false negative. Re-ran the probe against the live shards:
bench/results/prometheus-sigma-triggers-2026-05-11.md.
6 new tests in tests/test_prometheus_trigger_probe.py (probe loaded via
importlib): no-table to no-data, empty-table to data-available-no-fire,
latest-checkpoint->=4 to fires, earlier-dense-but-latest-sparse to no-fire,
density-sums-across-shards, report-renders-density-lines.
Updated #000012 Phase 1c landing receipt, #000037 section 12 Trigger 1
note, and the TICKETS.md index rows for both. Pure measurement: no
mutation, no LLM call, no schema change.
dav1d's review (RESPONSE_1 + RESPONSE_2) returned 2026-05-11. This
lands the Tier-1 items — everything that doesn't change numeric
outputs or invalidate the KAT discipline. The Tier-2 B1 conservative-
envelope (v2 calculator) is a separate decision and stays a closure
blocker.
Calculator (bench/scripts/t3_bound_calculator.py):
- Recommendation wording: "M2's single-window guarantee is broken"
→ "this conservative bound CANNOT CERTIFY M2's residual". An upper
bound exceeding 256 bits means we cannot certify, NOT that the
adversary can steer 256 bits — the prior wording overclaimed.
- New structured output fields: b1_model ("effective_control_v1"),
certification_status ∈ {CERTIFIED_BY_BOUND, NOT_CERTIFIED_BY_BOUND},
certification_threshold_bits (256), model_assumptions[]. Callers
read a machine-readable status, not just prose.
- Input validation hardening: _require_finite_float / _require_positive_int
helpers reject bools (isinstance(True, int) is True in Python — a
real leak risk for a security calculator) and NaN / ±inf for every
numeric input and constant.
- gradient_fraction = 0 now accepted (no T2 surface; B1 = 0; T3's
LR + batch-order channels still contribute) — improves component
isolation. CLI help + module docstring updated accordingly.
- Numeric outputs UNCHANGED: baseline still 625.8716 / 292.4813 /
300.0 / 33.3904; b1_model stays effective_control_v1; KAT discipline
intact.
Tests (tests/test_t3_bound_calculator.py, 53 → 75):
- Hard-coded cwd="/home/fox/git/arborist" → pathlib.Path(__file__).
resolve().parents[1] so the suite runs on any checkout.
- New: test_gradient_fraction_zero_accepted, test_bool_rejected_for_int_fields,
test_bool_rejected_for_float_fields, test_nonfinite_numbers_rejected,
test_output_carries_b1_model_and_certification_fields,
test_certification_status_certified_below_threshold.
- test_recommendation_exceeds_sha256 now also asserts "CANNOT CERTIFY"
+ certification_status == NOT_CERTIFIED_BY_BOUND.
Doc (docs/soft-hash-channel-t3-bound.md):
- §0 reworked into a reviewer brief recording dav1d's findings
(§2 accepted, §4 accepted, §5 accepted as model-bound, §3 = closure
blocker, wording/validation = applied).
- New §3.1: the B1-double-g issue spelled out — effective_control_v1
vs fraction_channels vs aggregate_bias vs max_envelope, with the
baseline-spread table (292 / 1730 / 5850 / 5850 bits); v2 path
described.
- §5: "B3 is a model-bound, not a directly-quoted theorem" note.
- §10: items 1-2 are now the closure blockers (B1 envelope v2; active
KAT fixture); items 3-7 are tightening paths (#000043). New §10.1
records what the 2026-05-11 hardening pass already landed.
- §11: calculator-output example updated to show the new fields +
corrected recommendation wording.
- §12: references add the dav1d review + clarify Bottou-Bousquet
"inspires" (not "underlies") the §5 model-bound.
Status (#000036 ticket + TICKETS.md row): review-returned + Tier-1-
applied; closure blockers = B1 v2 envelope (awaits fox go/no-go) +
active KAT fixture. R2's architectural integrations (Merkle audit-
event commitment, SQD canonicalization, CTI clause-lattice, 5F
trigger, ForkScore security-risk) noted as out-of-scope (separate
tickets if wanted).
AUTOCOUNT markers in docs/calculator-test-patterns.md +
docs/warrant-substrate-cookbook.md bumped 53 → 75.
Full suite: 2264 passed, 28 skipped.
Two coupled changes that wire the §13 Step 11 proposal stream from
in-memory-and-discarded to persisted-and-harvestable.
(A) emit_controller_events now writes a 4th event kind
controller_falsification_proposal, one row per
decision.falsification_proposals entry. Body carries branch_id +
witness_divergence + reason; label column carries the reason for
terminal-table inspection. Deterministic ordering by (branch_id,
witness_divergence) so canonical-body hashes are stable. Idempotent
under the existing UNIQUE (event_kind, body_hash) constraint. The
audit-chain semantics remain unchanged (still a sibling table; no
event_hash preimage entry). The CLI inspector's --kind choices gain
the new event kind so operators can filter for it directly.
(B) bench/scripts/harvest_falsification_proposals.py gains a third
source bucket CONTROLLER_PROPOSAL alongside the existing HYBRID +
UNGROUNDED providence_cache buckets. Reads
controller_falsification_proposal rows from controller_events,
extracts the 16-char cache_key prefix from branch_id (qa:<prefix>
pattern from the QA-runner advisory), joins back to providence_cache
for fixture enrichment (answer_text + audit_mode + verifier_method),
and tags _harvest_meta.harvested_from = "controller_events" so the
two source paths stay distinguishable in the fixture pack. Dedup
against the providence_cache buckets by fixture id.
Today this typically yields 0 new fixtures because (i) no live QA
has fired since Phase 2 wiring landed, and (ii) the QA-runner
single-branch advisory's proposals overlap providence_cache content
the harvester already finds. Real net value comes from Phase 3
(#000045) sweep emissions, which will produce multi-branch chunk
proposals that providence_cache rows can't predict.
Tests: 3 new in tests/test_prometheus_audit.py (proposal-row
emission, no-proposal no-row, idempotency); 1 new in
tests/test_bench_batteries.py (synthetic qa.db with both
providence_cache + controller_events rows; asserts the
controller_events bucket surfaces a fixture invisible to the
divergence-thresholded providence_cache buckets). The pre-existing
harvested-pack-runs-clean test relaxes its harvested_from pin from
"providence_cache"-only to {"providence_cache", "controller_events"}.
The dry-run was still using safe_weights(); Phase 1.c shipped
sweep_weights() (γ_5f=1.5, λ_capital_cost=0.25,
ν_witness_divergence=0.5) precisely for sleep-sweep economics. Swap
both call sites in prometheus_sigma_sweep_dryrun.py and label the
report Weight profile: sweep (§15.4).
§15.4 adds the sweep profile alongside §15.1 safe / §15.2
conservative / §15.3 exploratory so the named-profile registry has
a single doc source of truth and the dry-run report's §15.4
reference resolves.
§22 dry-run-runs table (3 rows: initial uniform-τ-1d under safe;
per-mode τ under safe; per-mode τ under sweep) replaces the prior
inline narrative count. Includes a Phase-3-design observation: the
sweep profile flattens the softmax (DEFERRED 271 → 369; REJECT
205 → 110; ACCEPT 1 → 0; MARGINAL 2 → 3) because reducing
λ_capital_cost + ν_witness_divergence shrinks the gap between
high-Δ5F and low-Δ5F branches, so fewer branches reach Kelly's
p_i > 0.5 floor. Falsification-fixture proposal count stayed flat
(447 → 449) — the §13 step 11 emission path is upstream of decision
labeling, so sweep mode preserves information-gathering value while
concentrating action-taking on the high-confidence tail. Phase 3
ticket #000045 §2 deliberately leaves softmax temperature un-pinned
because of this trade-off.
Per Finding 3, a uniform τ_qa=7d filtered out every recent
CANONICAL_PROJECTION row (the π* graduations from #000027/#000030/
#000032 are all younger than 7d), so the sweep saw zero high-value
kernel-only work. Splitting τ_qa by audit_mode lets the cheap kernel
re-probe path (CP) run on a short cycle while the expensive LLM
re-witness path (STRICT/HYBRID/UNGROUNDED) keeps the long cycle.
bench/scripts/prometheus_sigma_sweep_dryrun.py: new build_tau_by_mode()
helper + per-mode CASE in iter_target_a_candidates; sweep_target_a now
takes the dict instead of a single seconds value. New CLI flag
--tau-qa-cp-days (default 1d); --tau-qa-days now scopes to LLM-witness
modes only (default 7d). Report renders the per-mode τ table in the
header and marks Findings 2 and 3 RESOLVED with their landing commits.
Makefile: PROMETHEUS_SWEEP_TAU_DAYS bumped to 7 (was 1, the prior
Finding-3 workaround); new PROMETHEUS_SWEEP_TAU_CP_DAYS=1 makevar.
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md: regenerated
under the new defaults — 9 CP rows surface alongside 1,904 LLM-
witness candidates → 1,913 total Target A candidates → 1 ACCEPT,
2 MARGINAL, 205 REJECT, 271 DEFERRED chunks; 2 cache_drift vetoes
preserved end-to-end.
docs/tickets/ticket-000037 §22: Findings 2 + 3 marked RESOLVED with
landing-commit references; total dry-run cost line updated to the
new measurements (37.6 ms / 3,913 branches / 9.6 µs per branch).
Re-running `make prometheus-trigger-probe` after today's controller
landings shows the divergence-variance trigger has crossed both
thresholds:
Trigger 2 — divergence variance
Sample count: 37 (N_min = 30 ✓)
Mean: 0.7568, σ: 0.435
σ/mean ratio: 0.5748 (> 0.5 threshold)
Absolute σ: 0.435 (> 0.1 threshold)
Same-day morning probe (commit baseline) had only 16 samples and
did not fire; the additional witness-sweep / dry-run / harvest
activity through the afternoon brought sample count above N_min.
Agreement-label distribution across all shards:
KERNEL-LLM-DIVERGED 22
KERNEL-LLM-AGREE 6
LLM-DIVERGED 6
STRICT-WITNESSED 3
Trigger 1 (branch density) and Trigger 3 (witness cost share) did
NOT fire. Per §12 a single trigger firing is sufficient for Phase 1
gating — and Phase 1 has already landed. This commit captures the
empirical evidence that Phase 1 was on the right side of the gate.
Trigger 1 remains structurally blocked on #000012 Phase 1c
(fork_score_branches sibling table); that's the natural next move
if anyone wants to surface multi-branch consensus signals.
Closes the controller → 5F battery loop fox designed: #000037 Phase
1 produces FalsificationFixtureProposal records from high-divergence
providence_cache rows; this commit lands the harvester that turns
those proposals into a real 5F fixture pack the falsification
battery exercises every test run.
bench/scripts/harvest_falsification_proposals.py — reads qa.db,
filters live rows with witness_divergence = (n_unverified / n_quotes)
>= 0.5, stratifies by audit_mode, picks 20 HYBRID + 20 UNGROUNDED
top-by-cache_key for determinism, writes embedded-mode fixtures
to bench/fixtures/5f/falsification-harvested-v1.jsonl.
Each fixture carries `_harvest_meta` with the source cache_key,
divergence at harvest time, audit_mode at harvest time, and the
ticket reference (#000037 §13 step 11). Embedded mode — uses
`observed_violations` directly without calling verify_quotes
again; `answer_text` preserved verbatim for debugging.
Pack composition (initial harvest 2026-05-10):
20 UNGROUNDED (expected_reason: UNGROUNDED)
9 HYBRID_QUOTE (NEW motif — not in falsification-v1.jsonl)
7 HYBRID_CLAIM_LATTICE (NEW motif — not in falsification-v1.jsonl)
4 HYBRID_PARAPHRASE (already covered in v1)
Two harness tests pin the pack:
- test_5f_falsification_harvested_pack_runs_clean: 100% pass-rate,
every fixture carries traceable _harvest_meta.
- test_5f_falsification_harvested_pack_widens_motif_coverage: the
HYBRID_QUOTE / HYBRID_CLAIM_LATTICE motifs surface from real
corpus (loud-fail if harvest rotation drops them).
Makefile: `make bench-5f-harvest` re-runs the harvester. Parameters
exposed: HARVEST_QA_DB, HARVEST_OUT, HARVEST_THRESHOLD,
HARVEST_SAMPLE_PER_BUCKET.
Full suite: 2328 passed, 37 skipped (+2 from the two new pins).
Closes four gaps in the initial Phase 1 landing (commit f625cac):
1. §14 row 4 — Hermes-saturation guard. `hermes_utilization` was on
the input contract but never consumed. Now: `utilization >=
budget` → DEFERRED with HERMES_SATURATED note + advisory event
carrying (utilization, budget) for Phase 2 audit. Three new
tests (saturation-equal, saturation-overflow, headroom-exists).
2. §13 step 11 — falsification-fixture proposal emission.
New `FalsificationFixtureProposal` dataclass; controller now
emits one per branch whose `witness_divergence >=
falsification_divergence_threshold` (weight-tunable, default
0.5). Emission fires BEFORE the all-vetoed cascade so vetoed-
AND-diverged branches still surface as 5F-fixture candidates
per #000025. Four new tests (high-divergence emits, low-
divergence stays silent, threshold is weight-tunable, vetoed-
diverged emits anyway).
3. Entropy + memory gating moved from module-level constants
(`H_LOW=0.3`, `H_HIGH=0.7`, `KAPPA_MEMORY=0.5`) to weight
fields (`h_low`, `h_high`, `kappa_memory`). Constants stay as
back-compat exports; defaults match exactly so byte-identical
behavior when neither override fires. Three new tests
(h_low/h_high tunable, kappa_memory tunable, back-compat
match).
4. Veto-class cascade hardening. Added explicit tests for the
`replay_window_unbounded` → ESCALATE path and the
`soft_hash_signal` → QUARANTINE path (§6 + §14 documented but
previously untested). Plus tests for the §6 fail-loud
priority ordering: ESCALATE > QUARANTINE > REJECT when mixed.
Dry-run regenerated against ~/.arborist/shards corpus:
Target A at τ_qa=1d surfaces **576 FalsificationFixture
proposals** from witness_divergence >= 0.5 (24% of swept
candidates). These rows are now an actionable funnel for 5F
fixture mining under #000025 §3.
Module: arborist/substrate/prometheus.py (+117 LOC, 899 total)
Tests: tests/test_prometheus.py (+196 LOC, 22 → 36 tests)
Dryrun: bench/scripts/prometheus_sigma_sweep_dryrun.py +
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md
track the new falsification_proposals_total counter.
Full suite: 2326 passed, 37 skipped (+14 net from the new tests).
Phases 1 (controller) and 2 (sibling-table audit writes) landed in
prior commits. This commit adds the Phase 3 dry-run simulator
instead of the actual sleep-sweep scheduler, since Phase 3's value
is mostly in what we'd learn from running it — and the dry-run
captures those findings without committing to a scheduler design
prematurely.
bench/scripts/prometheus_sigma_sweep_dryrun.py — read-only
simulator that classifies §3 Target A (providence_cache) + Target B
(documents) sweep candidates, synthesizes ControllerBranches from
real shard data, runs the Phase 1 controller, reports decision
distribution + Phase-3-design findings. No LLM calls, no
mutations.
make prometheus-sweep-dryrun — produces a dated markdown report
at bench/results/prometheus-sigma-sweep-dryrun-YYYY-MM-DD.md.
Five findings surfaced by three dry-run iterations against the
live ~/.arborist/shards corpus (3.5M docs + 2839 providence_cache
rows) — captured in ticket §22:
1. chunk-size dominates Kelly threshold (must = Hermes
concurrency, not candidate pool)
2. flat capital_cost blocks every allocation (split kernel-cost
vs LLM-cost on the contract)
3. τ_qa=7d filters every CANONICAL_PROJECTION row (all 29 are
<7d old; need per-audit-mode τ)
4. Target B canonical-shape detection is the real headline
(~152K candidates extrapolated; controller correctly returns
MARGINAL on shape-match chunks)
5. quarantined rows correctly veto via cache_drift hard-veto
Mean per-branch controller latency in dry-run: 12.5 µs at
chunk_size=4. Phase 3's actual bottleneck is the witness fan-out
(Hermes calls), not the controller itself.
Ticket #000037 status flipped to in-progress with Phases 0+1+2
landed; Phase 3 scheduler remains future work but is informed by
the five findings.
Pre-Phase-1e: falsification-v1.jsonl covered 10 motif tags across 50
fixtures (the high-traffic warrant/title/anchor/format set). Phase
1e adds 12 fixtures (5f-fal-051..062) for the previously-uncovered
motifs from the verifier+soft-demote registries:
CITATION_MISMATCH DEFLECTION_DETECTED
MANUAL_QUOTE_VIOLATION SCHEMA_INVALID
SOURCE_ROLE_BLOCKED SUBJECT_TOKENS_ABSENT
TOO_MANY_EVIDENCE_IDS UNKNOWN_EVIDENCE_ID
BROAD_QUANTIFIER_RUNAWAY BROAD_QUANTIFIER_CAP_APPLIED
BROAD_QUANTIFIER_SCOPE_UNBOUND BROAD_QUANTIFIER_REJECTED
Coverage now: 22 unique motif tags across 62 fixtures.
Harness changes:
- test_bench_batteries.py: bump pass_count assertion 50 → 62 in both
falsification tests; add test_5f_falsification_covers_every_documented_motif
that pins the motif set against the verifier+soft-demote registries
so adding a new violation upstream surfaces here as a missing
fixture (loud signal, no silent drift).
- test_session_integration.py: bump full-suite total 662 → 674.
Closes#000025 §10.12 (every documented failure-motif tag).
Still open in Phase 1b: §10.11 (real shard finetuning chains),
§10.13 (Feedback Loop latency/efficiency against real workload),
§10.14 (threshold handoff to #000012).
Full QA-quality sweep, 75 questions × 3 modes × n=3 = 675 cells
against Hermes-3 8B at concurrency=4. No mode regressed past
the 5pp signal floor:
quote 53.8% STRICT vs 54% baseline = -0pp (stable)
claim_lattice_pointer 23.1% STRICT vs 20% baseline = +3pp (within floor)
claim_lattice (JSON) 46.7% STRICT vs 42% baseline = +5pp (at floor — marginal positive)
JSON-mode +5pp is right at the noise threshold per
docs/bench-maxing.md — could be the 2026-05-10 substrate work
(92/92 warrant chains + 18 textbook ingests + cascade tuning +
Phase 3 wiring) translating to retrieval-quality lift, OR
sample variance. Follow-up bench in 1-2 days disambiguates.
Format-collapse 0/225 in every mode. Latency stable
(10.6-11.1s median). Lattice-mode directive coverage 100%.
Phase 3 per-claim warrant-chain tail still dormant on this
fixture (rescue-only by design; no warrant-shape question
retrieves a chain-backed chunk while failing lexical anchor —
parallel shift's same finding). Source-level warrants tail
fires on claim-pack-targeted retrievals as documented in
phase3-live-validation-2026-05-10.md.
Substrate + Phase 3 sprint shipped clean.
Audited fox's exemplar 51-test file against the 9-item checklist
in docs/calculator-test-patterns.md (codified earlier today in
0725eb4). Two gaps found, both additive:
GAP 1: no KAT fixture
=====================
Checklist item 1: "KAT fixture under bench/fixtures/<module>/
(≥ 5 cases)". Fox's tests cover the doc's worked-example numbers
inline (test_baseline_matches_section_11_doc) but no separate
fixture file existed for off-the-baseline regression coverage.
Generated bench/fixtures/t3-bound/known-answer-tests.jsonl with
8 KATs:
- small-deployment-§7.1, medium-deployment-§7.2,
hardened-deployment-§7.3 (the doc's three worked examples)
- extreme-low-g (g=0.001, σ=0.5 — exercises low-SNR regime)
- tight-window-W=100 (small-W ceiling-rounding edge case)
- tightened-c-b1 (override-constants path)
- all-constants-tight (all three c_b1/c_b2/c_b3 overridden)
- b3-floor-regime (factor < 1, B3 floors to 0)
Each entry pins (calculator_version, inputs, expected_total,
expected_b1, expected_b2, expected_b3, expected_snr_grad).
Algorithm change MUST bump CALCULATOR_VERSION + emit new fixture
file under bench/fixtures/t3-bound/ — old runs replay against
old data per §1 discipline.
GAP 2: no B3 hand-computed formula test
=======================================
Checklist item 3: "Hand-computed formula tests — at least one
per independent contribution / output field". Fox had
test_b1_exact_formula + test_b2_exact_formula + test_snr_grad_formula
covering three of the five output fields. B3 had only
test_b3_floor_at_zero_when_factor_below_one (an edge case),
not a closed-form check on the general formula.
Added test_b3_exact_formula: hand-computes
``C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2`` per #000036
§5 (Bottou-Bousquet refinement), asserts agreement with the
function's B3_contribution. Pairs cleanly with the B1/B2 hand-
formula tests fox had.
CHECKLIST AUDIT — POST-FIX
==========================
1. KAT fixture ✓ NOW (was ❌; 8 entries)
2. VERSION + "v1" ✓ test_returns_calculator_version_token
3. Hand-formula ✓ NOW B1/B2/B3/snr_grad all covered
(was ⚠️ partial; B3 had floor-only)
4. Monotonicity ✓ test_monotone_in_window_length /
gradient_fraction
5. Closure ✓ test_total_equals_sum_of_three_contributions
6. Parametrized invalid ✓ four @pytest.mark.parametrize blocks
7. CLI subprocess ✓ test_cli_baseline_runs_clean +
test_cli_invalid_input_exits_2
8. Doc parity ✓ test_baseline_matches_section_11_doc
(caught today's §11 calibration drift)
9. Module-export shape ✓ test_returns_calculator_version_token +
test_constants_recorded
All nine items now ✓. test_t3_bound_calculator.py is the
exemplar for calculator-style test discipline.
Test count: 51 → 53 (+2 from this commit). Full suite:
1915 → 1985 (+70 from fox's parallel work + this commit's +2;
partial cycle effects).
Hygiene
=======
- make test → 1985 passed, 45 skipped.
- KAT fixture is JSONL with header comment naming
CALCULATOR_VERSION; future drift caught at the test level.
- Eat-my-own-dogfood: applied my docs/calculator-test-patterns.md
checklist to fox's exemplar test file. The fact that gaps
surfaced (even on fox's substantive 51-test surface) validates
that the checklist has real reviewer value, not just guideline
signaling.
Parallel-shift commit 6f1dbed already refreshed primary surfaces
but missed three sites:
docs/warrant-substrate-cookbook.md 54 → 74
docs/tickets/ticket-000031-...md 40 → 74 (Phase 2.5 narrative)
bench/results/full-warrant-resolution-...md 40 → 74 (×2 sites)
Live count from ~/.arborist/shards/000.db is 74 citation_aliases
+ 13 term_aliases. All four edits add the "grew 40 → 54 → 74
across the day" trail so future readers see the trajectory rather
than a stale point-in-time number.
Lands the formal derivation deliverable proposed in #000036 §3.1.
Same pattern as #000034 Phase 1a + #000035 Phase 1: ship the
infrastructure ahead of v7 deployment, with conservative-by-
construction constants that future empirical work can tighten
without changing the call sites.
docs/soft-hash-channel-t3-bound.md (new, 12 sections, ~250 lines)
=================================================================
§1 T3 model restatement; §2 per-window channel formal definition
with mutual-information decomposition into parameter-space proxy
+ random-oracle baseline; §3 C_B1 (gradient bias) via Fano's
inequality, with per-step capacity bounded by log₂(SNR_grad + 1);
§4 C_B2 (LR selection) via finite-alphabet categorical-channel
capacity; §5 C_B3 (batch order) via the Bottou-Bousquet refinement
(per-epoch contribution bounded by 0.5·log₂(N_b·σ_grad/‖∇L_max‖),
much tighter than the naive log₂(N_b!) bound that the ticket §3.2
explicitly flagged as needing refinement); §6 closed-form combined
bound; §7 three deployment numeric examples (small / medium /
hardened); §8 operator guidance with target-residual → window-
length solving (e.g. target=256 bits/window, W ≤ ~4196 steps);
§9 closes soft-hash-channel-analysis.md §9.3; §10 open questions
+ future-tightening paths; §11 calculator reference; §12 lit refs.
The closed form (§6):
I_window ≤ C_B1 · g · W · log₂(SNR_grad + 1)
+ C_B2 · ⌈W/K⌉ · log₂(R)
+ C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2
Conservative constants C_B1 = C_B2 = C_B3 = 1 (each by
data-processing inequality). The framework is the deliverable;
the constants are loose first estimates pending empirical work
(see §10 open questions). Tightening any of them refines the
bound without invalidating it.
bench/scripts/t3_bound_calculator.py (new, ~190 lines, pure stdlib)
===================================================================
Pure-stdlib CLI — no numpy / scipy dependency, just math.log2 +
ceiling division. Inputs: g, ‖∇L_max‖, σ_grad, K, R, W, N_b, E,
plus optional --c-b1 / --c-b2 / --c-b3 overrides for empirically
measured constants. Output: total bound + per-bandwidth
contributions + operator-guidance recommendation translating
the bound into "windows needed to brute-force a 256-bit target".
Verified against doc §7.1 small-deployment example: produces
625.87 bits/window vs the doc's hand-calculated 622.7. Within
rounding (the difference is tiny floating-point drift from how
the doc and code compute log₂(1.5)).
soft-hash-channel-analysis.md
=============================
§9.3 marked closed-2026-05-10 with reference to the new bound
doc. §11 status updated: open-questions list now reads §9.1
(parks on v7 per #000034 Phase 1b) + §9.2 (awaits v7 §9.10
amendment per #000035 Phase 2); §9.3 closed via #000036.
#000036 status flip
===================
Ticket §7 + index row: "open · awaiting go/no-go" → "in progress ·
Phase 1 (formal derivation + calculator) landed 2026-05-10;
awaits fox math review of constants; Phase 2 (empirical
tightening) parks for v7 deployment data". Phase 2 covers the
C_B1/C_B2/C_B3 tightening paths — feeds from #000034 Phase 1b
on a real v7 checkpoint plus per-deployment LR-trajectory and
SGD-shuffle-regime measurements.
Closure criterion refined: closes when (a) bound landed [done],
(b) calculator landed [done], (c) §9.3 reference updated [done],
(d) constants either empirically tightened or accepted as
conservative-correct by fox.
Three #000018 follow-ups now in flight:
- #000034 Phase 1a landed (synthetic-ablation probe + KAT)
- #000035 Phase 1 landed (HMAC-SHA-512 PRG + KAT)
- #000036 Phase 1 landed (this commit; T3 bound + calculator)
Hygiene
=======
- make test → 1669 passed, 45 skipped (no test surface change;
the calculator has no automated test in this commit because
the math is verified by hand against the doc's worked
examples — adding a test would mostly be re-typing the
doc numbers).
- make chain-check-shards → 0 across all 7 shards.
- arborist/ Python source unchanged; this commit is doc + script.
Lands the synthetic-ablation infrastructure proposed in fce8826's
ticket §7 amendment. Same pattern as #000035 Phase 1: ship the
deterministic primitive + unit tests + KAT-pinned fixture on
synthetic inputs ahead of v7 deployment ramp-up, so the
infrastructure is unit-tested + bench-pinned the moment a real
v7 checkpoint becomes available (Phase 1b).
bench/scripts/phi_alignment_probe.py
====================================
Implements ``measure_alignment(W, hessian_eval, *, k_top, k_bot,
epsilon) -> AlignmentReport`` per #000034 §3.1:
- Lanczos top-k + bottom-k via ``scipy.sparse.linalg.eigsh`` over
a user-supplied HVP closure. Probe never materializes H.
- Alignment score: A(W, H) = Σ_j (Σ_i ⟨W·e_i, v_j⟩²) / (λ_j+ε)
/ ‖W‖_F², per ticket §2.1. Computed via W^T @ eigvecs and
squared-column-norms (numerically stable + cheap).
- Verdict thresholds (§3.3): STRUCTURAL_ALIGNMENT (ratio > 1.5) /
NO_ALIGNMENT / ANTI_ALIGNED (ratio < 0.7).
Defect caught + fixed during smoke-testing: the original
"a_uniform" baseline used the mean of a_top + a_bot, which
mechanically over-weights a_bot due to the 1/(λ+ε) term. Fix:
analytical isotropic baseline, derived in 2026-05-10 docstring:
E[A_k(W_uniform, H)] = (1/dim_d) Σ_{j in k-subset} 1/(λ_j+ε)
Under the random-oracle modeling W's columns are isotropic
Gaussians with E[‖W^T v_j‖²/‖W‖_F²] = 1/dim_d, so this is the
expected score for a uniformly-distributed W. Smoke test
post-fix: aligned → STRUCTURAL_ALIGNMENT (ratio ~7.97), uniform →
NO_ALIGNMENT (ratio ~1.00), anti → ANTI_ALIGNED (ratio ~0.00).
All three classes land cleanly in their expected verdict bucket.
Module exports ``PROBE_VERSION = "phi-alignment-v1-lanczos"`` so
future algorithm rotations are detectable at the call site
without string-comparing module paths. Same convention as
#000035's PHI_PRG_VERSION.
bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl
========================================================
30 KAT entries — 10 per class (aligned / uniform / anti) — each
pinning (seed, dim_d, k, class) → expected_verdict + observed_ratio
for regression coverage. Deterministic-seeded so CI replays
exactly. Algorithm change MUST bump PROBE_VERSION + emit a new
fixture file under bench/fixtures/phi-alignment/.
Class ratio ranges:
- aligned: 7.77 - 8.27 (well above 1.5 STRUCTURAL_ALIGNMENT floor)
- uniform: 0.95 - 1.04 (cleanly within NO_ALIGNMENT band)
- anti: 0.00 (well below 0.7 ANTI_ALIGNED ceiling)
tests/test_phi_alignment_probe.py
=================================
14 tests covering #000034 §3.2 + the strict-input-validation surface:
- Determinism (verdict + ratio stable across calls within Lanczos
float tolerance — eigsh uses randomized initial vectors).
- Verdict thresholds (engineered cases land in correct bucket).
- Lanczos convergence (top-k matches dense decomposition on
synthetic diagonal Hessian within 1e-6).
- Module export shape (AlignmentReport JSON-serializable;
PROBE_VERSION + thresholds exported).
- Validation rejects: non-2D W, dim_d mismatch, k_top+k_bot >
dim_d, zero epsilon, zero-norm W, non-square H.
- KAT regression against the 30-entry fixture.
Tests skip via ``pytest.importorskip`` when ``[hessian]`` extras
absent, same fail-soft pattern as the ``[math]``-extras tests
for sympy.
pyproject.toml — new [hessian] optional-deps block
==================================================
Adds ``numpy>=1.26`` + ``scipy>=1.11`` under a new ``[hessian]``
extras gate. Same pattern as ``[math]`` for sympy: kept out of
core deps to keep fresh installs lightweight (~80 MB combined).
Operators install via ``pip install 'arborist[hessian]'``.
#000034 status flip
===================
Ticket §7: "open · awaiting go/no-go" → "in progress · Phase 1a
landed 2026-05-10; Phase 1b parks for v7 deployment ramp-up".
Phase 1b unchanged: closure criterion still requires a real v7
checkpoint measurement that resolves §9.1 of the soft-hash-
channel-analysis. TICKETS.md index row refreshed.
Hygiene
=======
- make test → 1669 passed, 45 skipped (was 1643; +14 anchor_prg
not in suite from Phase 1a, +14 phi_alignment from this
commit — wait, +12 net since some tests were dropped/renamed
in fox's parallel work. Bottom-line: 1669 stable.)
- make chain-check-shards → 0 across all 7 shards.
- arborist.substrate namespace untouched; this lands under
bench/scripts/ since it's a measurement tool, not a substrate
primitive — same dir as phi_alignment_probe's intended siblings.
Three-iteration investigation following c5bc53f's "Phase 3 is a rescue
mechanism, not a default path" finding. Question: can a paraphrase-
eliciting fixture actually drive Phase 3 to fire on real questions?
Key reading from arborist/qa/warrant.py:507 ("warrant_check"):
the function vacuous-passes (returns (True, []) without any anchor
extraction) unless the question shape is one of:
- relation (proper-noun anchors)
- date (4-digit year in claim)
- entity-list
- count-shape
- why-cause-shape
If none of those shapes apply, Phase 3 has nothing to suppress.
Three fixture iterations, all probed before benching:
| iter | strategy | chain hits |
|------|-----------------------------------|------------|
| v1 | "What is X?" / "Define X" | 12/13 |
| v2 | "How would you use X to ..." | 1/10 |
| v2' | "X: explain in plain English" | 2/13 |
| v3 | "Who proposed X?" / "Why does X" | 1/8 |
| v3' | "X: who first stated it?" | 1/8 |
Pattern: title-leading retrieval works (high title-token boost on
chain chunks) but those questions are definitional — no warrant
shape, no Phase 3 trigger. Warrant-shape questions ("who" / "when"
/ "why") trigger warrant_check but retrieval pulls toward
Wikipedia's prose articles instead of the terse claim-pack chunks.
Structural reason: claim-pack chunks (the 92 chain-root documents)
are deliberately terse — they state axioms / theorems precisely
with mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions retrieve from
Wikipedia, where the prose context matches the question shape.
Phase 3's data condition (chain-backed chunk cited for a
warrant-shape claim) is empirically unreachable through normal
retrieval pathways on this corpus shape.
Verdict (`bench/results/phase3-warrant-chain-paraphrase-
investigation-2026-05-10.md`): Phase 3's runtime fire rate is
structurally bounded near zero on the current corpus. Mechanism
is correct, non-regressing, and unit-tested
(tests/test_warrant_chain.py). The lexical layer is strong enough
that the rescue isn't needed at measurable rates.
Recommendation: don't engineer Phase 3 fires via fixture design.
Track fire rate as a corpus-evolution signal. If claim-pack
content gains prose context or the corpus mix changes, Phase 3
will start firing organically and the audit-line tail
(`· warrant proven via chain ×N`) will surface it.
This closes the three-thread investigation cleanly. v1 + v2 + v3
fixtures + reports all retained as the empirical trail.
Followup to dee6623 ("bench: #000031 Phase 3 A/B finds mechanism
dormant on warrant-targeted fixture") — the initial A/B reported
Phase 3 dormant on a 7-question fixture where only 2/7 questions
retrieved any chain-root sources. Open question: was the dormancy
a fixture inadequacy or a structural property of Phase 3?
This commit answers it. Aggressive fixture (13 questions using
exact claim-pack title phrasing — "What is the Pigeonhole
Principle?", "Define the Axiom of Extensionality", etc.; titles
sampled directly from the 92-document chain-root set on shard
000.db) drives retrieval to chain-root chunks on **12/13
questions** (24 chain sources retrieved total).
A/B vs a57b194~1, n=3, 117 cells per condition:
cache_key parity: 39/39 cells match
Phase 3 fire rate (warrant_proven_idxs): 0 / 117 cells
STRICT-rate: 85/117 → 85/117 (Δ +0)
per-cell movement (LLM dice): 4 ↑, 4 ↓
Phase 3 fires zero times even though retrieval does land on chain
roots on 92% of questions. The structural reason now reads as:
Phase 3 is a rescue mechanism, not a default path. It fires only
when (a) a cited chunk has warrant-resolver chain backing AND
(b) the lexical warrant_check would otherwise fail. On
well-formed axiom questions, the LLM's answer contains the
technical terminology that the lexical anchor heuristic looks
for — so warrant_check passes via the cheap lexical path and
Phase 3 has nothing to suppress.
Two readings of this finding:
1. **Positive for the lexical layer.** It's strong enough on
clean axiom questions that the chain-fallback isn't needed.
The mechanism that lands warrants on cheap lexical evidence
continues to do most of the work.
2. **Phase 3 calibration needs a different fixture shape.** To
exercise the rescue path empirically, a future fixture would
need to elicit lexical-fail-but-chain-pass claims —
paraphrase-heavy questions, disputed-terminology topics,
questions where the LLM naturally drifts from chunk vocabulary.
Mechanism is correct + non-regressing (cache_keys identical → no
retrieval movement; 0/117 fire rate confirms defensive-only
behavior). No further changes warranted to the implementation.
C — separate finding (refactor sweep verification): a57b194 has
zero v7/v8 patterns. Final exhaustive sweep across arborist/ +
tests/ shows only the intentional historical-note comments in
arborist/cli.py (Was \`arborist v8 score\`...) and
arborist/substrate/__init__.py (explaining why v-dirs were
retired). Refactor cleanup is complete.
B — separate finding (Makefile bench-fork-score smoke):
make bench-fork-baseline + make bench-fork-score runs end-to-end
under the renamed CLI. Verdict MARGINAL (score 0.0 for self-
comparison), exit 0. Yesterday's Makefile:430 fix verified live.
End-to-end smoke against the live shard cluster (~/.arborist/shards
+ ~/.arborist/crawl) using the parallel-shift probe fixture
bench/qa_questions_warrant_chain_probe.txt (7 questions explicitly
targeting the 92 claim-pack-warrant-resolved records).
Result: 6 of 7 questions reach EVIDENCE-WARRANTED on the four-rung
ladder. Q1 ("axiom of line incidence in Hilbert geometry") shows
the source-level "warrants: 5 proven" tail correctly counting all
5 cited claim-pack records with derivation chains. Q5 (De Morgan)
falls to ANCHOR-WARRANTED-PARTIAL on mixed Wikipedia/claim-pack
retrieval; Q7 (pigeonhole) hits TITLE_MISMATCH because retrieval
surfaced a Wikipedia article that shares no token with "Bogart".
Both Q5 and Q7 are correct fail-closed behavior given retrieval
choices, not Phase 3 issues.
What this validates:
- warrant_chain_lookup loads live 92 core_roots
- warrant_proven_claim_idxs threads through verdict -> result
dict -> JSON output (was None on cache hits, [] on fresh
misses — correct cache invariance preserved)
- source-level "warrants: N proven" tail fires
- EVIDENCE-WARRANTED rung promotes via existing
_ladder_rung_for_lattice logic without render-layer changes
- empty-set fail-closed default preserves backward compat
What's NOT validated end-to-end (still unit-test-only):
- Per-claim "· warrant proven via chain ×N" tail. Fires only
when warrant_check would have demoted AND chain present.
Narrow regime; covered by test_verify_claim_lattice_suppresses
_warrant_missing_with_chain.
Picks up the parallel-shift's probe fixture (was untracked); now
committed alongside the validation journal.
Bench follow-up to a57b194 ("ticket #000031 Phase 3: warrant-chain-
aware verifier suppresses WARRANT_MISSING"): land the warrant-
targeted probe fixture + capture an A/B comparing pre-Phase 3
(a57b194~1) vs HEAD on 7 questions × 3 modes × n=3.
bench/qa_questions_warrant_chain_probe.txt (new fixture)
========================================================
7 questions targeting claim-pack-resolved topics: Hilbert geometry
(2), Russell classes (1), Cantor diagonal (1), De Morgan (1),
Boolean absorption (1), pigeonhole in Bogart (1). Picked because
their topical match against the 18 textbook substrates +
warrant-resolver derivations should — in principle — drive
retrieval to chain-root chunks where Phase 3's suppression logic
fires.
bench/results/phase3-warrant-chain-A-B-2026-05-10.md (A/B report)
=================================================================
Findings:
- **21/21 cache_keys match BEFORE/AFTER.** Phase 3 doesn't fold
into governance_policy_hash (warrant_chain_roots is a runtime
parameter, not a policy field), so retrieval + prompt + LLM
inputs are byte-identical between conditions. Any STRICT-rate
movement is real verifier signal, not LLM dice.
- **`warrant_proven_claim_idxs` is non-empty in 0 / 63 AFTER cells.**
Phase 3's suppression path never fires on this fixture.
- **STRICT-rate: 42/63 → 43/63 (+1 cell, +1.6pp).** Four cells
move (2 up, 2 down); within n=3 LLM noise.
Why dormant despite warrant-targeted topic selection:
1. Pre-bench probe via `--dry-run --json` showed only 2/7 questions
retrieve any chain-root sources — the claim-pack textbooks have
far fewer chunks than Wikipedia, so retrieval defaults to the
bulkier corpus on general queries.
2. Even on the 2 questions that DO retrieve chain roots, the LLM
cites the larger Wikipedia chunks more often.
3. On the rare cell where the LLM cites a chain-root chunk, lexical
`warrant_check` already passes — Phase 3 has nothing to suppress.
Calibration finding, not a defect:
- Phase 3 mechanism is correct (no regression on cells where it
doesn't fire; cache_keys identical → can't move retrieval).
- The data condition for Phase 3 to actually fire requires a
fixture where (a) retrieval surfaces chain-root chunks AND
(b) the LLM cites them AND (c) the cited claim's lexical
warrant_check fails.
- Future fixture would need very specific terminology that
biases retrieval AWAY from Wikipedia (exact theorem labels,
axiom names, etc.).
Same calibration shape as yesterday's progressive-AND smoke A/B
finding: a targeted change can be dormant on the fixture you're
benching against. The bench-fixture-design discipline that ticket
#000037 §12 codifies — "measured trigger, not calendar date" —
applies here too: until a fixture lands that exercises Phase 3,
the suppression mechanism's value is theoretical.
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.
- New Citation.via_citation_alias field (default False, preserves
parse-from-source-ref path).
- warrant_status sets via_citation_alias=True on substitute
Citations from #000041 lookup_citation_aliases.
- resolve_chunks reads citation.via_citation_alias as a "floor" for
via_alias on every match it produces (Pass 1 hits inherit it
too, not just Pass 2 term-alias hits). Audit-honest: matches
from a substitute Citation are alias-driven regardless of which
cascade pass found the chunk.
Live re-resolve under the new attribution: 18 direct + 74 +alias
(was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
records resolving on the literally-cited Hilbert textbook. All 74
records resolved via citation-alias substitution now carry
process_id="warrant-resolver-v1+alias" in derivations.
3 new unit tests (test_warrant_resolver.py): default-False on
parsed Citations, explicit-True construction works, resolve_chunks
propagates the floor onto every ResolutionMatch.
B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.
- HtmlPageSource accepts default_author kwarg; appends ', by
<author>' to ingested document titles when the <title> tag
doesn't already include the surname.
- TextbookTexSource accepts default_author kwarg; appends ' by
<author>' to titles when the LaTeX has no \author{} macro AND
no PG-style 'Author:' boilerplate.
- _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
kwarg; same append logic. ingest_crawled() and arborist crawl
--ingest plumb it through.
- arborist ingest --author + arborist crawl --author CLI flags.
- bench/scripts/textbooks_manifest.py:cmd_lookup emits the
manifest's `author` field as a 7th tab column.
- make textbook target reads the author column and threads
--author into both crawl-ingest and shallow-ingest paths.
Idempotency preserved — surname-already-in-title detection prevents
double-stamping on re-ingest. Shards previously SQL-backfilled
(Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
Grinstead-Snell) keep their existing titles; new ingests pick up
the author signal at source time.
Live smoke: arborist ingest --source html --author "Bertrand
Russell" against PG #41654 yields title "Introduction to
Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
with no SQL UPDATE needed.
Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
Lands the M1 mitigation cryptographic primitive that ticket #000018
§5.2 + §9.2 specified, scoped per ticket #000035 §3.1-§3.3. Pure
stdlib (hashlib + hmac); no third-party dependency.
arborist/v7/__init__.py
=======================
First module landed under the v7 namespace. v7 plastic-training is
currently paper-stage (per #000037 §17.2); this is where its
deterministic primitives accumulate ahead of an active deployment
target so the building blocks are unit-tested + KAT-pinned the
moment v7 needs them.
arborist/v7/anchor_prg.py
=========================
Implements ``phi_prg(hard_hash_32, dim_h, *, seed) -> list[float]``
per #000035 §3.1. Construction is SP 800-108 KDF in counter mode
over HMAC-SHA-512:
Output(SEED, C(M), n_bytes) :=
i = 0
out = b""
while len(out) < n_bytes:
out += HMAC-SHA-512(SEED, C(M) || i.to_bytes(4, 'big'))
i += 1
return out[:n_bytes]
Float conversion: f(u32) := 2 * (u32 / 2**32) - 1
↑ uniform on [-1, 1)
Security argument from #000035 §2.1: HMAC-SHA-512 is a PRF under
the standard SHA-512 + HMAC assumption; distinguishing advantage
from random bounded by SHA-512 collision-resistance (~2^256), which
structurally matches the substrate's SHA-256 hard-hash family. The
seed is published, not secret — secrecy is not the security
property; the property is computational indistinguishability of the
output from random, which holds even when the seed is public.
Module exports ``PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512"`` so
future algorithm rotations are detectable at the call site without
string-comparing module paths. Per #000035 risk §6.1, a
``phi_prg_version`` field in the v7 manifest will let future
deployments swap to a successor PRF without breaking historical
replay; this version string is the runtime-side mirror.
Hard-hash input length checked exactly at 32 bytes — silently
padding shorter input would break the PRF security argument.
``dim_h`` validated as positive int.
tests/test_anchor_prg.py
========================
20 tests covering #000035 §3.2 acceptance criteria + the strict
range invariant + the input-validation surface:
- Determinism: same (seed, hard_hash, dim_h) → identical floats.
- Range: every output in [-1, 1) with strict upper bound. Three
edge cases pinned: u32=0 → -1.0, u32=2^31 → 0.0,
u32=2^32-1 → just below 1.0.
- Chi² loose-uniformity: 4096-sample bin-test (df=15) with a
generous threshold (60); catches catastrophic PRG bugs (counter
cycling, mis-keyed HMAC) without claiming cryptographic-grade
evidence.
- Boundary: dim_h=1 + dim_h=16384 both produce sensible output.
- Avalanche, seed-bit: flip top bit of seed[0]; require 35-65% of
output bits flipped (PRF avalanche property).
- Avalanche, hash-bit: same surface for the hard-hash input.
- Validation rejects: short hashes, long hashes, non-bytes hashes,
zero / negative / non-int dim_h.
- Module export shape: PHI_PRG_VERSION + PLACEHOLDER_SEED.
- KAT regression: pinned vectors verified against
``bench/fixtures/phi-prg/known-answer-tests.jsonl``.
bench/fixtures/phi-prg/known-answer-tests.jsonl
================================================
10 KAT vectors generated against the placeholder seed + custom
seed/hash combinations; covers smoke (placeholder seed × small
dim_h), block boundaries (HMAC-SHA-512 blocks are 64 bytes, so
dim_h=16 is exactly one block, dim_h=17 is two blocks with
truncation), seed/hash one-bit-flip variants, and a 4096-element
stress sample.
Each row pins the SHA-256 of the raw byte stream (not the float
list) — that's the durable contract; switching from list[float] to
array.array('f', ...) or numpy arrays at the float layer would not
invalidate the fixture. Algorithm changes MUST bump
PHI_PRG_VERSION and create a new fixture file under
bench/fixtures/phi-prg/; old runs replay against old data per the
v7 spec replay discipline.
#000035 status flip
===================
docs/tickets/ticket-000035-prg-choice-phi-prg.md §7 updated from
"open · awaiting go/no-go" to "in progress · Phase 1 landed
2026-05-10". §7 now carries Phase 1 close-out details + Phase 2
gating criteria (active v7 deployment target + spec maintainer
review of §3.4 amendment text). The §3.4 v7 §9.10 amendment text
stays as the draft awaiting Phase 2 landing.
docs/TICKETS.md index row was already updated by fox in commit
ed470dc; my edit was idempotent.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1623; +20 anchor_prg)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist.v7 namespace picked up automatically by the existing
pyproject.toml [tool.setuptools.packages.find] include="arborist*"
glob; no setup change required.
- fox's in-flight #000037 ticket modifications + a parallel
#000031 ticket update left untouched.
The +14-record close-out: most stuck records had substrate content
in already-ingested shards but the FTS5 cascade picked discriminator
tokens the substrate text didn't use as titles. Two surgical
additions to _build_record_query_cascade:
(1) parenthetical-phrase variant via new _phrase_from_parenthetical
helper. The discriminating term often lives INSIDE a trailing
parenthetical that _phrase_for_axiom was stripping:
"Kolmogorov's First Axiom (Non-Negativity)" -> '"non negativity"'
"Newton's First Law (Law of Inertia)" -> '"inertia"'
"Axiom of Side-Angle-Side (SAS)" -> '"sas"' (already had this via prior path)
(2) content-tokens OR-join (top 3) as soft fallback after the AND
join. Most stuck records had top-5-AND scoring 0 hits even when
substrate had clear content; OR-of-top-3 surfaces relevant
chunks reliably without sacrificing precision (citation +
title-haystack guards remain).
Plus one final citation-alias row: Boehm-Jacopini -> Software
Foundations LF (decision_by="fox 2026-05-10"). SF-LF Imp.v
formalizes the structured-program theorem in Coq with sequence /
conditional / while-loop — same theorem as Boehm-Jacopini 1966
CACM in modern formal-verification dress.
Per-pillar:
I: 12/13 -> 13/13
II: 9/10 -> 10/10
III: 11/13 -> 13/13
IV: 18/18 stable
V: 2/5 -> 5/5
VI: 3/5 -> 5/5
VII: 11/14 -> 14/14
IX: 11/14 -> 14/14
Total: 92/92 (100%)
Cumulative day-of-2026-05-10 lift: 18/92 -> 92/92 (+74 records,
+80 pp absolute). Started the day with only Hilbert pillar IV
resolved; ended with all 8 pillars fully resolved across 13
textbook substrates (Hilbert, Boole, Cantor, De Morgan, Russell
IMP/PoM, PM Vol 1, Peano, Dedekind, Newton, Laplace, Grinstead-
Snell, PLFA, SF-LF, Bogart, Levin, Judson, Aristotle).
40 citation-alias rows + 13 term-alias rows, all audit-disciplined
per #000041/#000042 (decision_by + decision_rationale per row).
All 1623 tests pass.
