Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
Hand-inspection of the bottom-15 STRICT-fires from the raw §3.2.2 step 2
sweep showed claim-lattice overlay markup ([E\d+ | title | hash: '…'])
depressing scores on correct concise answers (the 6× Henry-VIII case),
while true-positive deflections (broad-question / narrow-answer like
'winners of all major sports?' → just-one-sport) remained correctly
low-scored. So the noise FP class is the bracket metadata; cleaning it
should reduce FP without losing true-positive signal.
Built clean_for_relevance() in arborist/qa/relevance/shadow.py — strips
[E\d+ | ... ] blocks + trailing '...']' tails. Baked into
ShadowRelevance.check_question_answer / check_claim_source by default
(opt out with clean_input=False). relevance_shadow_sweep.py applies it
to inputs before _score_batch (opt out with --no-clean).
Re-ran the full 6-model sweep on the 808-cell pooled STRICT with
cleaning:
bge-reranker-large 21.8% → 18.6% (-3.2)
MiniLM-L-4-v2 15.0% → 9.4% (-5.6 pts, -37% rel)
MiniLM-L-6-v2 11.6% → 9.0% (-2.6)
MiniLM-L-12-v2 11.0% → 8.0% (-3.0)
bge-reranker-base 9.5% → 9.0% (-0.5)
MiniLM-L-2-v2 4.5% → 1.5% (-3.0 pts, -67% rel)
Universal improvement, every model better. Big surprise: MiniLM-L-2-v2
— the model that FAILED the candidate-bench separability (margin
-1.97, declared 'capacity floor') — has the LOWEST real-traffic FP
rate at its own cb θ (1.5%). Because L-2's compressed score range
gives it a low cb θ which few real STRICT pairs score below.
SEVENTH instance of 'candidate-bench doesn't predict real-traffic'.
Runtime-veto verdict UNCHANGED — still not viable; smallest fp=0 θ on
real STRICT is below the cb NEG max for every model, so at any
runtime-safe θ the catch on cb NEG is 0/12. But cleaning is now FREE
improvement for any soft-signal / advisory / contrastive use of the
relevance score. Hand-inspected bottom-10 post-cleaning confirms true-
positive deflection signal preserved.
First grid (MiniLM, n=1 bench-qa 89 STRICT cells): the §7 #22 'fails the
gate' verdict was config-specific (defaults k=6/θc=0.5/θe=0.9) — the grid
finds k=1/θc=0.93/θe=0.5 → catch 8/12 falsification-hard (incl. BOTH
recombination fixtures hard-003 Mercury + hard-005 Einstein), 0/89 STRICT
FP, 0/26 synthetic-legit FP; 18/28 synthetic recombination. So the
lexical-candidate approach is NOT a dead end. (k=1 = no haystack;
θe=0.5 tighter than the clean-set 0.9.) Full multi-model + n=3
characterization next.
Speedup (§3 plan): ShadowNLI._nli_batch batches forwards
(ARBORIST_NLI_BATCH=64); device auto-detect (ARBORIST_NLI_DEVICE, else
cuda-if-available); auto-prefer an ONNX export — bench/scripts/export_nli_onnx.py
/ make export-nli-onnx exports + int8-dynamic-quantizes the pinned
checkpoint into ~/.arborist/models/nli/<ver>/onnx/ (operator state, NOT
committed), _ensure_loaded loads model_quantized.onnx via
optimum.onnxruntime (backend onnx-int8), falls back to torch silently.
torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair (~4x);
seconds on a 4090. optimum[onnxruntime] added to the [nli] extra; 24
tests.
Gate-item-4 verdict at proper n: ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=1 → 223 cells (89 STRICT / 90 HYBRID / 44 UNGROUNDED; also
surfaced + fixed a lone-surrogate bug). Shadow sweep over those: NLI-as-
runtime-veto on STRICT has ~26% FP at θc 0.5, ~8% at θc 0.90, ~0% only
at θc 0.99 — and θc 0.99 gives up most recombination recall (hard
synthetic recombinations bottom out ~0.76). FAILS the §7 #12 gate on
this design. Only untried path that might pass: a Phase-3 runtime hook
running NLI on the verifier's actual matched clauses (1-3), not
top-6-by-overlap. Until then: runtime NLI demotion stays off; the 2
fixtures stay permanent boundary markers; θc stays 0.5. Production
verifier unchanged; falsification-hard stays 10/12.
candidate_clauses() — NLI now runs only on the top-N source clauses by
content-token overlap with the answer claim (max_candidate_clauses=6),
not the whole context; records n_candidate_clauses / best_clause_overlap
/ recombination_risk. Synthetic sweep unchanged (28/28 recombination,
0/26 legit FP, mean 1.45 candidate clauses/record). Real-traffic smoke
re-run: STRICT would-demote 30% → 20%, overall 47% → 33% — better, not
fixed; recombination-risk split doesn't separate either. Residual STRICT
false-contras at ~0.83-0.92 → θc would need ≈ 0.90 (vs the clean-set
0.5); at θc=0.90 the data in hand gives 27/28 synthetic recall, 0/26
legit FP, 0/10 smoke STRICT FP — but n=10 is too small to set on.
Next: a fuller ARBORIST_NLI_SHADOW=1 bench-qa run → sweep θc on hundreds
of STRICT cells → confirm → set it. θc stays 0.5; runtime NLI demotion
stays off. Production verifier unchanged; falsification-hard stays 10/12.
Live hook: ARBORIST_NLI_SHADOW=1 makes query() surface the verifier-input
text (gated off-by-default, never a cache_key/governance/audit_mode
input); qa_sweep.py carries it + the answer into bench rows; the shadow
sweep reads them and buckets by audit_mode. ARBORIST_NLI_SHADOW=1 make
bench-qa-smoke (15 cells) → the naive 'NLI on every context clause'
scaffold has a ~30% would-demote rate on STRICT answers — a haystack /
multiple-comparisons artifact (real contexts segment into 100-336
clauses; max-over-all almost always finds a tangential clause the model
reads as contradicting; a paraphrased STRICT answer often isn't verbatim-
entailed by any single clause so the entailment guard doesn't rescue it).
Lesson: the §7 #5 'candidate source clauses' + recombination-risk gating
is load-bearing, not optional. Do NOT enable runtime NLI demotion on the
current scaffold; next step is the candidate-clause restriction, then
re-run, then gate item 4 is meaningful. Production verifier unchanged;
falsification-hard stays 10/12.
arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.
First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
The #000025 §10.14 calibration showed _delta_5{s,t,f} mean over a
battery's 5 subs, so a single-sub gain weighs 1/5 of face value (the
5× dilution). #000047 ships the knob to pick the aggregation, default
unchanged.
WeightSet.delta_aggregator ∈ {"mean","max","sum"} (default "mean") —
a categorical field, validated in __post_init__ against
DELTA_AGGREGATORS; from_dict takes it as a string. Default unchanged →
ScoredFork output byte-identical → no fork_score.ESTIMATOR_VERSION
bump.
fork_score._aggregate(deltas, how): mean = arithmetic mean, max =
max(0.0, max_i Δ_i), sum = Σ Δ_i; empty → 0.0. _delta_5s/_delta_5t/
_delta_5f take an aggregator arg (default "mean"); the 5F efficiency
bonus is added after the aggregated base (aggregator-independent).
fork_score passes weights.delta_aggregator. The per-sub
HARD_REGRESSION_FLOOR flags are computed before aggregation, so a
single-sub regression still forces REJECT under max/sum. The chosen
aggregator is recorded in ScoredFork.weights["delta_aggregator"] (via
WeightSet.as_dict()); fork_score_branches traceability stays via the
opaque weights_id — no schema migration.
bench/scripts/fivef_threshold_calibration.py gained §5 — runs the
#000046 below-ceiling pack (5f/falsification at 0.333) and shows the
verdict / γ·Δ5f under each aggregator; bench/results/5f-threshold-
calibration-2026-05-11.md §5 is the captured record. Default stays
"mean" — the conservative, noise-robust, regression-symmetric choice
matching docs/bench-maxing.md's per-rate floor framing; v8 picks
max/sum per-deployment.
Tests: 8 new in tests/test_fork_score.py + 1 anchor in
tests/test_fivef_threshold_calibration.py; tests/test_weights.py
as_dict field-set test updated to include delta_aggregator;
test_fork_score.py AUTOCOUNT tags (#000012 §286, warrant-substrate-
cookbook.md ×2) bumped 23 → 31.
#000047 closed; #000012 §8 §3 + TICKETS.md row updated.
Full suite: 2330 passed, 28 skipped.
Closes the three open Phase-1b items of #000025; every §10 closure
criterion is now met, so the ticket flips to closed.
§10.14 — ForkScore threshold-calibration handoff to #000012.
bench/scripts/fivef_threshold_calibration.py (make bench-5f-threshold-
calibration) runs the canonical 5S/5T/5F packs + the 5F live packs and
reports baseline rates, observability granularity (1/n), and fork_score
verdicts on the parent vs synthetic child perturbations →
bench/results/5f-threshold-calibration-2026-05-11.md. Findings written
into ticket-000012 §8: keep SIGNAL_FLOOR / HARD_REGRESSION_FLOOR at
0.05; the small 5S packs (syntax n=10, semantics n=8) are coarser than
the floors so any regression there trips hard-reject (intended zero-
tolerance); the 5x averaging dilution in _delta_*; ceiling saturation
(every pack at 1.0 -> delta-rate terms <= 0). No constant change
shipped. 6 tests in tests/test_fivef_threshold_calibration.py.
§10.13 — feedback latency / efficiency on real workload.
run_feedback_loop now computes feedback_latency (listed in §5.5 since
Phase 1a, never implemented) — wall-clock seconds to apply a live
chain against its temp shard, surfaced per-task
(feedback_latency_seconds) + battery (feedback_latency_mean_seconds,
feedback_live_task_count). For live chains feedback_efficiency's cost
denominator switched from len(chain) (count of requested ops) to the
persisted footprint _persisted_cost = audit-event rows the chain
actually wrote + their body bytes / 1e6. Embedded chains keep
len(chain) and report feedback_latency_seconds = None. Latency is a
wall-clock field (run-to-run variable, like BatteryResult.timestamp)
and is not a fork_score input. 3 tests in tests/test_bench_batteries.py.
§10.11 — real selfmodel finetuning chains.
bench/scripts/selfmodel_chain_snapshot.py (make bench-5f-selfmodel-
snapshot) appends one chained SelfModel snapshot per run to a
persistent shard (~/.arborist/shards/selfmodel-chain.db, override via
ARBORIST_SELFMODEL_CHAIN_DB) with one CapabilityClaim per sub-battery
(metric = "5S-syntax" etc., measured_value = that pack's rate,
eval_digest = the pack's fixture digest, threshold = SIGNAL_FLOOR).
snapshot() auto-parents, so each snapshot is a distinct root and the
lineage grows by one per run. run_finetuning gains a third dispatch
mode — shard-chain (gated on a task's selfmodel_shard key) — via
_chain_finetuning_measure: reads the two most-recent snapshots
(latest() = child, its parent_selfmodel_root = parent) and measures
improvement on target_capability between them. This is the real
lineage replacing Phase-1a's synthetic parent->child pairs; the
chained delta reflects genuine cross-run drift (0.0 today — the
embedded packs are at ceiling). Operator pack
bench/fixtures/5f/finetuning-shardchain-v1.jsonl (6 tasks) + make
bench-5f-finetuning-shardchain; not in `make bench-5f`, `make test`,
or a fresh checkout (a missing/too-short chain fails honestly). The
real chain shard was bootstrapped 2-deep on 2026-05-11; make
chain-check-shards reports 0 breaks on it (and all other shards).
10 tests in tests/test_selfmodel_chain.py.
Full suite: 2311 passed, 28 skipped.
Per fox: apply the conservative max_envelope B1 model by changing the
v1 calculator's default — NOT by forking a v2. CALCULATOR_VERSION stays
"t3-bound-v1-bottou-refinement" (the descriptor names the unchanged B3
term); b1_model is echoed in the output AND the inputs dict so KAT
replays are unambiguous about which model produced a row.
Calculator (bench/scripts/t3_bound_calculator.py):
- New b1_model kwarg + --b1-model CLI flag, choices:
max_envelope (default) max(fraction_channels, aggregate_bias)
fraction_channels g · W · log₂(1 + G/σ)
aggregate_bias W · log₂(1 + g·G/σ)
effective_control_v1 g · W · log₂(1 + g·G/σ) (old non-worst-case)
- Default is now max_envelope — genuinely upper-bounding across both
interpretations of g (dav1d review §3 closure blocker, RESOLVED).
- Every output reports all three concrete B1 variants
(B1_fraction_channels / B1_aggregate_bias / B1_effective_control_v1),
b1_selected, and both SNR readings (snr_grad = g·G/σ,
snr_per_channel = G/σ) regardless of which b1_model was requested.
- model_assumptions[] now carries f"B1_model_{b1_model}".
- inputs echo now includes c_b1/c_b2/c_b3/b1_model (replay-complete).
- Invalid b1_model rejected with a ValueError naming the field.
- Baseline I_window: 625.8716 (effective_control_v1) → 6183.0154
(max_envelope: B1=aggregate_bias 5849.63 dominates fraction_channels
1729.72), certification_status NOT_CERTIFIED_BY_BOUND at W=10000.
KAT fixture (bench/fixtures/t3-bound/known-answer-tests.jsonl):
- Regenerated 2026-05-11 — 12 entries: the 8 §7-derived configs under
the new max_envelope default, a g=0 edge case, plus explicit-mode
pins for effective_control_v1 / fraction_channels / aggregate_bias.
- Each entry carries b1_model, expected_b1_selected,
expected_b1_{fraction_channels,aggregate_bias,effective_control_v1},
expected_snr_per_channel, expected_certification_status.
Tests (tests/test_t3_bound_calculator.py, 75 → 83):
- test_t3_bound_known_answer_tests no longer skips (fixture active);
pins b1_model, b1_selected, certification_status + numbers, tolerates
optional new fields on older fixtures.
- New: test_b1_max_envelope_exact_formula, test_invalid_b1_model_rejected,
test_cli_b1_model_flag (effective_control_v1 / fraction_channels /
aggregate_bias). test_b1_exact_formula renamed
test_b1_effective_control_v1_exact_formula and now passes the explicit
model. Updated baseline / below-256 / CLI tests for the new numbers.
Doc (docs/soft-hash-channel-t3-bound.md):
- Header + §0 + §3.1 + §6 + §7 (worked examples) + §8 (operator
guidance W-solving) + §10 (closure blockers RESOLVED) + §10.1 +
§11 (calculator schema) + §12 all updated for the max_envelope
default. §8: target-256 W drops from ~4196 to ~415 steps under the
conservative model — the ~10× cost of not assuming which g-reading
holds; operators who can measure effective-control applies can use
--b1-model effective_control_v1 for the looser W (a calibration
claim they must justify, not a default).
Status (#000036 ticket + TICKETS.md): both prior dav1d closure
blockers cleared (B1 worst-case model + active KAT fixture); remaining
= fox's final close-or-iterate call.
AUTOCOUNT markers bumped 75 → 83. Full suite: 2288 passed, 28 skipped.
dav1d's review (RESPONSE_1 + RESPONSE_2) returned 2026-05-11. This
lands the Tier-1 items — everything that doesn't change numeric
outputs or invalidate the KAT discipline. The Tier-2 B1 conservative-
envelope (v2 calculator) is a separate decision and stays a closure
blocker.
Calculator (bench/scripts/t3_bound_calculator.py):
- Recommendation wording: "M2's single-window guarantee is broken"
→ "this conservative bound CANNOT CERTIFY M2's residual". An upper
bound exceeding 256 bits means we cannot certify, NOT that the
adversary can steer 256 bits — the prior wording overclaimed.
- New structured output fields: b1_model ("effective_control_v1"),
certification_status ∈ {CERTIFIED_BY_BOUND, NOT_CERTIFIED_BY_BOUND},
certification_threshold_bits (256), model_assumptions[]. Callers
read a machine-readable status, not just prose.
- Input validation hardening: _require_finite_float / _require_positive_int
helpers reject bools (isinstance(True, int) is True in Python — a
real leak risk for a security calculator) and NaN / ±inf for every
numeric input and constant.
- gradient_fraction = 0 now accepted (no T2 surface; B1 = 0; T3's
LR + batch-order channels still contribute) — improves component
isolation. CLI help + module docstring updated accordingly.
- Numeric outputs UNCHANGED: baseline still 625.8716 / 292.4813 /
300.0 / 33.3904; b1_model stays effective_control_v1; KAT discipline
intact.
Tests (tests/test_t3_bound_calculator.py, 53 → 75):
- Hard-coded cwd="/home/fox/git/arborist" → pathlib.Path(__file__).
resolve().parents[1] so the suite runs on any checkout.
- New: test_gradient_fraction_zero_accepted, test_bool_rejected_for_int_fields,
test_bool_rejected_for_float_fields, test_nonfinite_numbers_rejected,
test_output_carries_b1_model_and_certification_fields,
test_certification_status_certified_below_threshold.
- test_recommendation_exceeds_sha256 now also asserts "CANNOT CERTIFY"
+ certification_status == NOT_CERTIFIED_BY_BOUND.
Doc (docs/soft-hash-channel-t3-bound.md):
- §0 reworked into a reviewer brief recording dav1d's findings
(§2 accepted, §4 accepted, §5 accepted as model-bound, §3 = closure
blocker, wording/validation = applied).
- New §3.1: the B1-double-g issue spelled out — effective_control_v1
vs fraction_channels vs aggregate_bias vs max_envelope, with the
baseline-spread table (292 / 1730 / 5850 / 5850 bits); v2 path
described.
- §5: "B3 is a model-bound, not a directly-quoted theorem" note.
- §10: items 1-2 are now the closure blockers (B1 envelope v2; active
KAT fixture); items 3-7 are tightening paths (#000043). New §10.1
records what the 2026-05-11 hardening pass already landed.
- §11: calculator-output example updated to show the new fields +
corrected recommendation wording.
- §12: references add the dav1d review + clarify Bottou-Bousquet
"inspires" (not "underlies") the §5 model-bound.
Status (#000036 ticket + TICKETS.md row): review-returned + Tier-1-
applied; closure blockers = B1 v2 envelope (awaits fox go/no-go) +
active KAT fixture. R2's architectural integrations (Merkle audit-
event commitment, SQD canonicalization, CTI clause-lattice, 5F
trigger, ForkScore security-risk) noted as out-of-scope (separate
tickets if wanted).
AUTOCOUNT markers in docs/calculator-test-patterns.md +
docs/warrant-substrate-cookbook.md bumped 53 → 75.
Full suite: 2264 passed, 28 skipped.
Two coupled changes that wire the §13 Step 11 proposal stream from
in-memory-and-discarded to persisted-and-harvestable.
(A) emit_controller_events now writes a 4th event kind
controller_falsification_proposal, one row per
decision.falsification_proposals entry. Body carries branch_id +
witness_divergence + reason; label column carries the reason for
terminal-table inspection. Deterministic ordering by (branch_id,
witness_divergence) so canonical-body hashes are stable. Idempotent
under the existing UNIQUE (event_kind, body_hash) constraint. The
audit-chain semantics remain unchanged (still a sibling table; no
event_hash preimage entry). The CLI inspector's --kind choices gain
the new event kind so operators can filter for it directly.
(B) bench/scripts/harvest_falsification_proposals.py gains a third
source bucket CONTROLLER_PROPOSAL alongside the existing HYBRID +
UNGROUNDED providence_cache buckets. Reads
controller_falsification_proposal rows from controller_events,
extracts the 16-char cache_key prefix from branch_id (qa:<prefix>
pattern from the QA-runner advisory), joins back to providence_cache
for fixture enrichment (answer_text + audit_mode + verifier_method),
and tags _harvest_meta.harvested_from = "controller_events" so the
two source paths stay distinguishable in the fixture pack. Dedup
against the providence_cache buckets by fixture id.
Today this typically yields 0 new fixtures because (i) no live QA
has fired since Phase 2 wiring landed, and (ii) the QA-runner
single-branch advisory's proposals overlap providence_cache content
the harvester already finds. Real net value comes from Phase 3
(#000045) sweep emissions, which will produce multi-branch chunk
proposals that providence_cache rows can't predict.
Tests: 3 new in tests/test_prometheus_audit.py (proposal-row
emission, no-proposal no-row, idempotency); 1 new in
tests/test_bench_batteries.py (synthetic qa.db with both
providence_cache + controller_events rows; asserts the
controller_events bucket surfaces a fixture invisible to the
divergence-thresholded providence_cache buckets). The pre-existing
harvested-pack-runs-clean test relaxes its harvested_from pin from
"providence_cache"-only to {"providence_cache", "controller_events"}.
The dry-run was still using safe_weights(); Phase 1.c shipped
sweep_weights() (γ_5f=1.5, λ_capital_cost=0.25,
ν_witness_divergence=0.5) precisely for sleep-sweep economics. Swap
both call sites in prometheus_sigma_sweep_dryrun.py and label the
report Weight profile: sweep (§15.4).
§15.4 adds the sweep profile alongside §15.1 safe / §15.2
conservative / §15.3 exploratory so the named-profile registry has
a single doc source of truth and the dry-run report's §15.4
reference resolves.
§22 dry-run-runs table (3 rows: initial uniform-τ-1d under safe;
per-mode τ under safe; per-mode τ under sweep) replaces the prior
inline narrative count. Includes a Phase-3-design observation: the
sweep profile flattens the softmax (DEFERRED 271 → 369; REJECT
205 → 110; ACCEPT 1 → 0; MARGINAL 2 → 3) because reducing
λ_capital_cost + ν_witness_divergence shrinks the gap between
high-Δ5F and low-Δ5F branches, so fewer branches reach Kelly's
p_i > 0.5 floor. Falsification-fixture proposal count stayed flat
(447 → 449) — the §13 step 11 emission path is upstream of decision
labeling, so sweep mode preserves information-gathering value while
concentrating action-taking on the high-confidence tail. Phase 3
ticket #000045 §2 deliberately leaves softmax temperature un-pinned
because of this trade-off.
Per Finding 3, a uniform τ_qa=7d filtered out every recent
CANONICAL_PROJECTION row (the π* graduations from #000027/#000030/
#000032 are all younger than 7d), so the sweep saw zero high-value
kernel-only work. Splitting τ_qa by audit_mode lets the cheap kernel
re-probe path (CP) run on a short cycle while the expensive LLM
re-witness path (STRICT/HYBRID/UNGROUNDED) keeps the long cycle.
bench/scripts/prometheus_sigma_sweep_dryrun.py: new build_tau_by_mode()
helper + per-mode CASE in iter_target_a_candidates; sweep_target_a now
takes the dict instead of a single seconds value. New CLI flag
--tau-qa-cp-days (default 1d); --tau-qa-days now scopes to LLM-witness
modes only (default 7d). Report renders the per-mode τ table in the
header and marks Findings 2 and 3 RESOLVED with their landing commits.
Makefile: PROMETHEUS_SWEEP_TAU_DAYS bumped to 7 (was 1, the prior
Finding-3 workaround); new PROMETHEUS_SWEEP_TAU_CP_DAYS=1 makevar.
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md: regenerated
under the new defaults — 9 CP rows surface alongside 1,904 LLM-
witness candidates → 1,913 total Target A candidates → 1 ACCEPT,
2 MARGINAL, 205 REJECT, 271 DEFERRED chunks; 2 cache_drift vetoes
preserved end-to-end.
docs/tickets/ticket-000037 §22: Findings 2 + 3 marked RESOLVED with
landing-commit references; total dry-run cost line updated to the
new measurements (37.6 ms / 3,913 branches / 9.6 µs per branch).
Closes the controller → 5F battery loop fox designed: #000037 Phase
1 produces FalsificationFixtureProposal records from high-divergence
providence_cache rows; this commit lands the harvester that turns
those proposals into a real 5F fixture pack the falsification
battery exercises every test run.
bench/scripts/harvest_falsification_proposals.py — reads qa.db,
filters live rows with witness_divergence = (n_unverified / n_quotes)
>= 0.5, stratifies by audit_mode, picks 20 HYBRID + 20 UNGROUNDED
top-by-cache_key for determinism, writes embedded-mode fixtures
to bench/fixtures/5f/falsification-harvested-v1.jsonl.
Each fixture carries `_harvest_meta` with the source cache_key,
divergence at harvest time, audit_mode at harvest time, and the
ticket reference (#000037 §13 step 11). Embedded mode — uses
`observed_violations` directly without calling verify_quotes
again; `answer_text` preserved verbatim for debugging.
Pack composition (initial harvest 2026-05-10):
20 UNGROUNDED (expected_reason: UNGROUNDED)
9 HYBRID_QUOTE (NEW motif — not in falsification-v1.jsonl)
7 HYBRID_CLAIM_LATTICE (NEW motif — not in falsification-v1.jsonl)
4 HYBRID_PARAPHRASE (already covered in v1)
Two harness tests pin the pack:
- test_5f_falsification_harvested_pack_runs_clean: 100% pass-rate,
every fixture carries traceable _harvest_meta.
- test_5f_falsification_harvested_pack_widens_motif_coverage: the
HYBRID_QUOTE / HYBRID_CLAIM_LATTICE motifs surface from real
corpus (loud-fail if harvest rotation drops them).
Makefile: `make bench-5f-harvest` re-runs the harvester. Parameters
exposed: HARVEST_QA_DB, HARVEST_OUT, HARVEST_THRESHOLD,
HARVEST_SAMPLE_PER_BUCKET.
Full suite: 2328 passed, 37 skipped (+2 from the two new pins).
Closes four gaps in the initial Phase 1 landing (commit f625cac):
1. §14 row 4 — Hermes-saturation guard. `hermes_utilization` was on
the input contract but never consumed. Now: `utilization >=
budget` → DEFERRED with HERMES_SATURATED note + advisory event
carrying (utilization, budget) for Phase 2 audit. Three new
tests (saturation-equal, saturation-overflow, headroom-exists).
2. §13 step 11 — falsification-fixture proposal emission.
New `FalsificationFixtureProposal` dataclass; controller now
emits one per branch whose `witness_divergence >=
falsification_divergence_threshold` (weight-tunable, default
0.5). Emission fires BEFORE the all-vetoed cascade so vetoed-
AND-diverged branches still surface as 5F-fixture candidates
per #000025. Four new tests (high-divergence emits, low-
divergence stays silent, threshold is weight-tunable, vetoed-
diverged emits anyway).
3. Entropy + memory gating moved from module-level constants
(`H_LOW=0.3`, `H_HIGH=0.7`, `KAPPA_MEMORY=0.5`) to weight
fields (`h_low`, `h_high`, `kappa_memory`). Constants stay as
back-compat exports; defaults match exactly so byte-identical
behavior when neither override fires. Three new tests
(h_low/h_high tunable, kappa_memory tunable, back-compat
match).
4. Veto-class cascade hardening. Added explicit tests for the
`replay_window_unbounded` → ESCALATE path and the
`soft_hash_signal` → QUARANTINE path (§6 + §14 documented but
previously untested). Plus tests for the §6 fail-loud
priority ordering: ESCALATE > QUARANTINE > REJECT when mixed.
Dry-run regenerated against ~/.arborist/shards corpus:
Target A at τ_qa=1d surfaces **576 FalsificationFixture
proposals** from witness_divergence >= 0.5 (24% of swept
candidates). These rows are now an actionable funnel for 5F
fixture mining under #000025 §3.
Module: arborist/substrate/prometheus.py (+117 LOC, 899 total)
Tests: tests/test_prometheus.py (+196 LOC, 22 → 36 tests)
Dryrun: bench/scripts/prometheus_sigma_sweep_dryrun.py +
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md
track the new falsification_proposals_total counter.
Full suite: 2326 passed, 37 skipped (+14 net from the new tests).
Phases 1 (controller) and 2 (sibling-table audit writes) landed in
prior commits. This commit adds the Phase 3 dry-run simulator
instead of the actual sleep-sweep scheduler, since Phase 3's value
is mostly in what we'd learn from running it — and the dry-run
captures those findings without committing to a scheduler design
prematurely.
bench/scripts/prometheus_sigma_sweep_dryrun.py — read-only
simulator that classifies §3 Target A (providence_cache) + Target B
(documents) sweep candidates, synthesizes ControllerBranches from
real shard data, runs the Phase 1 controller, reports decision
distribution + Phase-3-design findings. No LLM calls, no
mutations.
make prometheus-sweep-dryrun — produces a dated markdown report
at bench/results/prometheus-sigma-sweep-dryrun-YYYY-MM-DD.md.
Five findings surfaced by three dry-run iterations against the
live ~/.arborist/shards corpus (3.5M docs + 2839 providence_cache
rows) — captured in ticket §22:
1. chunk-size dominates Kelly threshold (must = Hermes
concurrency, not candidate pool)
2. flat capital_cost blocks every allocation (split kernel-cost
vs LLM-cost on the contract)
3. τ_qa=7d filters every CANONICAL_PROJECTION row (all 29 are
<7d old; need per-audit-mode τ)
4. Target B canonical-shape detection is the real headline
(~152K candidates extrapolated; controller correctly returns
MARGINAL on shape-match chunks)
5. quarantined rows correctly veto via cache_drift hard-veto
Mean per-branch controller latency in dry-run: 12.5 µs at
chunk_size=4. Phase 3's actual bottleneck is the witness fan-out
(Hermes calls), not the controller itself.
Ticket #000037 status flipped to in-progress with Phases 0+1+2
landed; Phase 3 scheduler remains future work but is informed by
the five findings.
Lands the formal derivation deliverable proposed in #000036 §3.1.
Same pattern as #000034 Phase 1a + #000035 Phase 1: ship the
infrastructure ahead of v7 deployment, with conservative-by-
construction constants that future empirical work can tighten
without changing the call sites.
docs/soft-hash-channel-t3-bound.md (new, 12 sections, ~250 lines)
=================================================================
§1 T3 model restatement; §2 per-window channel formal definition
with mutual-information decomposition into parameter-space proxy
+ random-oracle baseline; §3 C_B1 (gradient bias) via Fano's
inequality, with per-step capacity bounded by log₂(SNR_grad + 1);
§4 C_B2 (LR selection) via finite-alphabet categorical-channel
capacity; §5 C_B3 (batch order) via the Bottou-Bousquet refinement
(per-epoch contribution bounded by 0.5·log₂(N_b·σ_grad/‖∇L_max‖),
much tighter than the naive log₂(N_b!) bound that the ticket §3.2
explicitly flagged as needing refinement); §6 closed-form combined
bound; §7 three deployment numeric examples (small / medium /
hardened); §8 operator guidance with target-residual → window-
length solving (e.g. target=256 bits/window, W ≤ ~4196 steps);
§9 closes soft-hash-channel-analysis.md §9.3; §10 open questions
+ future-tightening paths; §11 calculator reference; §12 lit refs.
The closed form (§6):
I_window ≤ C_B1 · g · W · log₂(SNR_grad + 1)
+ C_B2 · ⌈W/K⌉ · log₂(R)
+ C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2
Conservative constants C_B1 = C_B2 = C_B3 = 1 (each by
data-processing inequality). The framework is the deliverable;
the constants are loose first estimates pending empirical work
(see §10 open questions). Tightening any of them refines the
bound without invalidating it.
bench/scripts/t3_bound_calculator.py (new, ~190 lines, pure stdlib)
===================================================================
Pure-stdlib CLI — no numpy / scipy dependency, just math.log2 +
ceiling division. Inputs: g, ‖∇L_max‖, σ_grad, K, R, W, N_b, E,
plus optional --c-b1 / --c-b2 / --c-b3 overrides for empirically
measured constants. Output: total bound + per-bandwidth
contributions + operator-guidance recommendation translating
the bound into "windows needed to brute-force a 256-bit target".
Verified against doc §7.1 small-deployment example: produces
625.87 bits/window vs the doc's hand-calculated 622.7. Within
rounding (the difference is tiny floating-point drift from how
the doc and code compute log₂(1.5)).
soft-hash-channel-analysis.md
=============================
§9.3 marked closed-2026-05-10 with reference to the new bound
doc. §11 status updated: open-questions list now reads §9.1
(parks on v7 per #000034 Phase 1b) + §9.2 (awaits v7 §9.10
amendment per #000035 Phase 2); §9.3 closed via #000036.
#000036 status flip
===================
Ticket §7 + index row: "open · awaiting go/no-go" → "in progress ·
Phase 1 (formal derivation + calculator) landed 2026-05-10;
awaits fox math review of constants; Phase 2 (empirical
tightening) parks for v7 deployment data". Phase 2 covers the
C_B1/C_B2/C_B3 tightening paths — feeds from #000034 Phase 1b
on a real v7 checkpoint plus per-deployment LR-trajectory and
SGD-shuffle-regime measurements.
Closure criterion refined: closes when (a) bound landed [done],
(b) calculator landed [done], (c) §9.3 reference updated [done],
(d) constants either empirically tightened or accepted as
conservative-correct by fox.
Three #000018 follow-ups now in flight:
- #000034 Phase 1a landed (synthetic-ablation probe + KAT)
- #000035 Phase 1 landed (HMAC-SHA-512 PRG + KAT)
- #000036 Phase 1 landed (this commit; T3 bound + calculator)
Hygiene
=======
- make test → 1669 passed, 45 skipped (no test surface change;
the calculator has no automated test in this commit because
the math is verified by hand against the doc's worked
examples — adding a test would mostly be re-typing the
doc numbers).
- make chain-check-shards → 0 across all 7 shards.
- arborist/ Python source unchanged; this commit is doc + script.
Lands the synthetic-ablation infrastructure proposed in fce8826's
ticket §7 amendment. Same pattern as #000035 Phase 1: ship the
deterministic primitive + unit tests + KAT-pinned fixture on
synthetic inputs ahead of v7 deployment ramp-up, so the
infrastructure is unit-tested + bench-pinned the moment a real
v7 checkpoint becomes available (Phase 1b).
bench/scripts/phi_alignment_probe.py
====================================
Implements ``measure_alignment(W, hessian_eval, *, k_top, k_bot,
epsilon) -> AlignmentReport`` per #000034 §3.1:
- Lanczos top-k + bottom-k via ``scipy.sparse.linalg.eigsh`` over
a user-supplied HVP closure. Probe never materializes H.
- Alignment score: A(W, H) = Σ_j (Σ_i ⟨W·e_i, v_j⟩²) / (λ_j+ε)
/ ‖W‖_F², per ticket §2.1. Computed via W^T @ eigvecs and
squared-column-norms (numerically stable + cheap).
- Verdict thresholds (§3.3): STRUCTURAL_ALIGNMENT (ratio > 1.5) /
NO_ALIGNMENT / ANTI_ALIGNED (ratio < 0.7).
Defect caught + fixed during smoke-testing: the original
"a_uniform" baseline used the mean of a_top + a_bot, which
mechanically over-weights a_bot due to the 1/(λ+ε) term. Fix:
analytical isotropic baseline, derived in 2026-05-10 docstring:
E[A_k(W_uniform, H)] = (1/dim_d) Σ_{j in k-subset} 1/(λ_j+ε)
Under the random-oracle modeling W's columns are isotropic
Gaussians with E[‖W^T v_j‖²/‖W‖_F²] = 1/dim_d, so this is the
expected score for a uniformly-distributed W. Smoke test
post-fix: aligned → STRUCTURAL_ALIGNMENT (ratio ~7.97), uniform →
NO_ALIGNMENT (ratio ~1.00), anti → ANTI_ALIGNED (ratio ~0.00).
All three classes land cleanly in their expected verdict bucket.
Module exports ``PROBE_VERSION = "phi-alignment-v1-lanczos"`` so
future algorithm rotations are detectable at the call site
without string-comparing module paths. Same convention as
#000035's PHI_PRG_VERSION.
bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl
========================================================
30 KAT entries — 10 per class (aligned / uniform / anti) — each
pinning (seed, dim_d, k, class) → expected_verdict + observed_ratio
for regression coverage. Deterministic-seeded so CI replays
exactly. Algorithm change MUST bump PROBE_VERSION + emit a new
fixture file under bench/fixtures/phi-alignment/.
Class ratio ranges:
- aligned: 7.77 - 8.27 (well above 1.5 STRUCTURAL_ALIGNMENT floor)
- uniform: 0.95 - 1.04 (cleanly within NO_ALIGNMENT band)
- anti: 0.00 (well below 0.7 ANTI_ALIGNED ceiling)
tests/test_phi_alignment_probe.py
=================================
14 tests covering #000034 §3.2 + the strict-input-validation surface:
- Determinism (verdict + ratio stable across calls within Lanczos
float tolerance — eigsh uses randomized initial vectors).
- Verdict thresholds (engineered cases land in correct bucket).
- Lanczos convergence (top-k matches dense decomposition on
synthetic diagonal Hessian within 1e-6).
- Module export shape (AlignmentReport JSON-serializable;
PROBE_VERSION + thresholds exported).
- Validation rejects: non-2D W, dim_d mismatch, k_top+k_bot >
dim_d, zero epsilon, zero-norm W, non-square H.
- KAT regression against the 30-entry fixture.
Tests skip via ``pytest.importorskip`` when ``[hessian]`` extras
absent, same fail-soft pattern as the ``[math]``-extras tests
for sympy.
pyproject.toml — new [hessian] optional-deps block
==================================================
Adds ``numpy>=1.26`` + ``scipy>=1.11`` under a new ``[hessian]``
extras gate. Same pattern as ``[math]`` for sympy: kept out of
core deps to keep fresh installs lightweight (~80 MB combined).
Operators install via ``pip install 'arborist[hessian]'``.
#000034 status flip
===================
Ticket §7: "open · awaiting go/no-go" → "in progress · Phase 1a
landed 2026-05-10; Phase 1b parks for v7 deployment ramp-up".
Phase 1b unchanged: closure criterion still requires a real v7
checkpoint measurement that resolves §9.1 of the soft-hash-
channel-analysis. TICKETS.md index row refreshed.
Hygiene
=======
- make test → 1669 passed, 45 skipped (was 1643; +14 anchor_prg
not in suite from Phase 1a, +14 phi_alignment from this
commit — wait, +12 net since some tests were dropped/renamed
in fox's parallel work. Bottom-line: 1669 stable.)
- make chain-check-shards → 0 across all 7 shards.
- arborist.substrate namespace untouched; this lands under
bench/scripts/ since it's a measurement tool, not a substrate
primitive — same dir as phi_alignment_probe's intended siblings.
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.
- New Citation.via_citation_alias field (default False, preserves
parse-from-source-ref path).
- warrant_status sets via_citation_alias=True on substitute
Citations from #000041 lookup_citation_aliases.
- resolve_chunks reads citation.via_citation_alias as a "floor" for
via_alias on every match it produces (Pass 1 hits inherit it
too, not just Pass 2 term-alias hits). Audit-honest: matches
from a substitute Citation are alias-driven regardless of which
cascade pass found the chunk.
Live re-resolve under the new attribution: 18 direct + 74 +alias
(was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
records resolving on the literally-cited Hilbert textbook. All 74
records resolved via citation-alias substitution now carry
process_id="warrant-resolver-v1+alias" in derivations.
3 new unit tests (test_warrant_resolver.py): default-False on
parsed Citations, explicit-True construction works, resolve_chunks
propagates the floor onto every ResolutionMatch.
B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.
- HtmlPageSource accepts default_author kwarg; appends ', by
<author>' to ingested document titles when the <title> tag
doesn't already include the surname.
- TextbookTexSource accepts default_author kwarg; appends ' by
<author>' to titles when the LaTeX has no \author{} macro AND
no PG-style 'Author:' boilerplate.
- _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
kwarg; same append logic. ingest_crawled() and arborist crawl
--ingest plumb it through.
- arborist ingest --author + arborist crawl --author CLI flags.
- bench/scripts/textbooks_manifest.py:cmd_lookup emits the
manifest's `author` field as a 7th tab column.
- make textbook target reads the author column and threads
--author into both crawl-ingest and shallow-ingest paths.
Idempotency preserved — surname-already-in-title detection prevents
double-stamping on re-ingest. Shards previously SQL-backfilled
(Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
Grinstead-Snell) keep their existing titles; new ingests pick up
the author signal at source time.
Live smoke: arborist ingest --source html --author "Bertrand
Russell" against PG #41654 yields title "Introduction to
Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
with no SQL UPDATE needed.
Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
Three concrete improvements to the textbook ingest layer:
Per-book make targets
=====================
- `make textbook ID=<id>` — generic, looks up one entry from the
manifest and runs the right ingest path (BFS crawl if
`crawl_url` is set, shallow URL ingest otherwise).
- `make textbook-list` — list ingestable textbook ids.
- Convenience targets per active manifest entry:
textbook-bogart, textbook-keller-trotter, textbook-levin,
textbook-aristotle-prior, textbook-aristotle-posterior,
textbook-newton, textbook-morin
Each per-book target writes to a dedicated shard at
`$(CRAWL_SHARDS_DIR)/textbook_<id>.db` — one shard per textbook
id rather than one per host (which would conflate multiple
Wikisource books — Aristotle Prior, Posterior, Newton — into
one shard).
The manifest helper script gains `ids` and `lookup <id>`
subcommands so the Makefile can introspect manifest entries
without grep gymnastics.
Idempotency contract documented
================================
Every ingest path is content-addressed. Same content →
same `document_root` → no-op insert. Re-running `make textbook-*`
or `make crawl-textbooks` on the same network state is safe at
the database layer — no duplicates, no schema drift, no
audit-event proliferation. HTTP refetch happens every run; pair
with `make recrawl-check` for a HEAD-only freshness probe.
README rewritten to make this contract explicit, with shard-
layout table + crawl-delay caveats consolidated.
Vital-books checklist
=====================
The README now carries a g4-pillar coverage table:
Pillar I Logic ✓ Levin + Aristotle Prior + Posterior
Pillar II Set Theory ✓ Levin
Pillar III Arithmetic ✓ Levin
Pillar IV Geometry ✗ Hilbert PG #17384 (PDF/TeX only)
Pillar V Probability ✗ Kolmogorov license analysis pending
Pillar VI Phys. Newton ✓ Newton Principia Motte
Pillar VII Combin. ✓ Bogart + Keller-Trotter + Levin
Pillar IX λ-Calculus ✗ Church + Turing 1936 papers pending
5 of 7 pillars now have surface coverage. The remaining gaps
are explicit license (V) or format (IV PDF/TeX, IX paper-level)
issues, each documented in the corresponding manifest entry's
notes field.
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
Per the #000031 implementation track — surface-ingest cited
textbooks for warrant promotion of claim-pack records. Five new
manifest entries spanning the PD-scope additions plus
license-fail placeholders for citation traceability:
newton-principia-motte (PD)
Newton's Mathematical Principles of Natural Philosophy
(Motte 1729 / Motte-Cajori 1846 English translations) via
Wikisource. Crawled @ depth=2: 60 docs, 289 chunks, 10K
edges. FTS5 spot-checks: "law of motion" → top hit on the
1729 Motte title page; "gravitation" → Newton-titled
chapters in the Wikisource Physics category. Cited by
claim-pack pillar VI.
aristotle-posterior-analytics (PD)
Mure / Owen translations via Wikisource. Crawled @ depth=1:
20 docs, 71 chunks. FTS5: "syllogism" → Posterior Analytics
Bouchier + Owen translations as top hits. Companion to
Prior Analytics already in the manifest; together cover
pillar I logic axioms.
kolmogorov-foundations-probability-1933 (placeholder)
German original PD-by-age in EU; US copyright restored via
URAA through life+70 (Kolmogorov d.1987 → US PD 2058).
Morrison 1956 English translation: Chelsea Publishing
copyright. Entry kept with empty urls field for citation
traceability; license analysis pending.
lukasiewicz-aristotle-syllogistic-1951 (proprietary placeholder)
Oxford University Press, copyright through life+70
(Łukasiewicz d.1956 → PD 2027 EU, longer US via URAA). NOT
redistributable. Kept as a license-fail placeholder; the
1921 Polish primary papers are themselves PD by age but
lack accessible HTML editions.
License-validator drive-by
==========================
The textbooks_manifest.py validator now treats placeholder
entries (urls + crawl_url both empty) as exempt from the
allow-list license check. License discipline stays fail-closed
at the URL-emit step — placeholders without URLs raise no
exception under any license token, so we can document
yellow-light proprietary citations (Mendelson, Enderton,
Łukasiewicz) without hacks.
Coverage summary post-commit
============================
g4 pillar | textbook coverage
─────────────┼─────────────────────────────────────────────────
I Logic | Levin (CC-BY-SA), Aristotle Prior + Posterior (PD)
II Set Theory| Levin (CC-BY-SA)
III Arithm. | Levin (CC-BY-SA)
IV Geometry | (Hilbert PG #17384 placeholder — PDF/TeX only)
V Probability| (Kolmogorov 1933 license analysis pending)
VI Phys. | Newton Principia Motte (PD)
VII Combin. | Bogart (GFDL), Keller-Trotter (CC-BY-SA), Levin
Out-of-scope follow-ups: PDF/TeX source support (#000031 §5),
Mendelson + Enderton license decision (#000031 §2.1),
chunk-resolution + derivations.proof_blob warrant promotion
(separate ticket).
The first iteration of this layer landed scaffolding plus shallow
seed-URL ingest, which produced 300-1000 byte stubs (front matter
+ nav chrome only) instead of real book content. Fox flagged
that as the wrong outcome — we were supposed to track down the
actual texts.
This commit fixes it by adding the deep-ingest path: each
manifest entry now carries `crawl_url` + `crawl_depth` +
`crawl_max` fields, and `make crawl-textbooks` iterates them
through the existing BFS crawler (one shard per host).
Manifest changes
================
Three URL fixups uncovered by the first crawl attempt:
- PreTeXt textbooks (Bogart, Keller-Trotter, Levin) ship an
`index.html` that's a 1-line meta-refresh redirect; the BFS
crawler doesn't follow meta-refresh, so the seed URL has to
point at the actual content entry (`frontmatter.html`,
`app-comb-2.html`, `ctgd/index.html`).
- Open Data Structures uses trailing-slash directory paths
(`/ods-cpp/`, `/ods-python/`, `/ods-java/`), not `/ods-cpp.html`.
- `appliedcombinatorics.org` declares `Crawl-Delay: 20` in
robots.txt — full BFS takes ~25 minutes; documented in the
manifest entry's notes so operators know to background it.
Hilbert + Boole stay flagged as `urls: []` placeholders because
PG eBooks #17384 + #15114 ship as PDF/TeX only (no HTML format).
Their `tex_url` + `pdf_url` fields are now preserved on the
entry for the future PDF/TeX source ticket.
Make targets
============
- crawl-textbooks — BFS-crawl every entry with `crawl_url`
- crawl-textbooks-stats — docs/chunks/size summary across crawl shards
Helper script: `bench/scripts/textbooks_manifest.py crawl-targets`
emits one tab-separated `<url>\t<depth>\t<max>\t<id>` row per
entry that declares a `crawl_url`.
Real-content verification
=========================
Crawl results across 4 of 5 textbooks (Keller-Trotter still
running in background due to crawl-delay; expected to complete
in ~25 min):
crawl_bogart_openmathbooks_org.db docs=44 chunks=161 971 KB
crawl_discrete_openmathbooks_org.db docs=51 chunks=376 1929 KB
crawl_en_wikisource_org.db docs=30 chunks=110 1651 KB
crawl_opendatastructures_org.db docs=64 chunks= 84 1110 KB
TOTAL 189 docs, 731 chunks
FTS5 spot-checks confirm substantive content landed:
- Bogart "binomial coefficient" → "CTGD Partitions and Stirling
Numbers" chapter, top hit
- Levin "modus ponens" → "Propositional Logic" chapter, top hit
That's the real lift signal we were after. Out-of-scope
follow-ups remain: PDF/TeX support for PG editions (Hilbert,
Boole) and full Aristotle Owen-translation depth (currently
~75% complete on Wikisource).
A make-target scaffold for pulling public-domain and copyleft-
redistributable textbooks into a dedicated arborist shard via the
existing HtmlPageSource ingest pipeline. No new ingest machinery —
the existing path (robots.txt → noise-strip → 512-token chunk →
Merkle root → audit-event) IS the consistent process.
Manifest format
===============
bench/fixtures/textbooks/manifest-v1.jsonl — one JSON record per
textbook with explicit license tokens (PD / CC-BY / CC-BY-SA /
GFDL / OSI-approved). Fail-closed validation in
bench/scripts/textbooks_manifest.py refuses to emit URLs from
entries with missing or disallowed license tokens, so license
discipline surfaces at the URL-emit step rather than getting
silently buried in a fetch run.
Allow-list excludes CC-BY-NC (incompatible with arborist's
AGPLv3 distribution profile) and CC-BY-ND (no-derivatives
prevents chunking). Wilf's generatingfunctionology stays out
because its educational-use license forbids rehosting; citable
but not redistributable.
Initial entries (7 textbooks, 8 seed URLs)
==========================================
- Bogart, Combinatorics Through Guided Discovery — GFDL-1.3
- Keller & Trotter, Applied Combinatorics — CC-BY-SA-4.0
- Levin, Discrete Mathematics: An Open Introduction (3rd ed) —
CC-BY-SA-4.0 (4th ed switched to NC; pin 3rd)
- Hilbert, Foundations of Geometry (Townsend 1902) — PD via PG
- Boole, An Investigation of the Laws of Thought (1854) — PD
- Aristotle, Prior Analytics (Jenkinson) — PD via Wikisource
- Morin, Open Data Structures — CC-BY-2.5
Covers pillars I (logic) / II (set theory) / III (arithmetic) /
IV (geometry) / VII (combinatorics) on the claim-pack
(#000029) layout, plus a CS anchor for downstream domain
expansion. Each entry's pillar_targets field lists which
claim-pack pillars its records are candidate citations for.
Make targets
============
- textbooks-summary — license + URL counts per entry
- textbooks-urls — flat URL list to stdout
- fetch-textbooks — ingest all manifested URLs into
$(TEXTBOOK_DB) (default
~/.arborist/textbooks.db)
- textbooks-stats — documents / chunks / edges
- textbooks-verify — sample Merkle proof verification
To grow coverage of one textbook: append more chapter URLs to
its `urls` array. For deep-BFS of a textbook home: use the
existing `make crawl-ingest URL=<base> DEPTH=N` instead.
Smoke-tested end-to-end against Wikisource Prior Analytics:
1 doc / 1 chunk / 12 outbound edges / Merkle proof passes.
Shows the pipeline works; populating each book to depth needs
either more URLs in the manifest or the crawler.
Out of scope
============
- PDF processing. The Internet Archive hosts PD textbooks
(MacMahon's Combinatory Analysis 1915, Whitworth's Choice
and Chance 1867, the Motte 1729 Principia) as scanned PDFs;
a TextbookPdfSource with pdftotext / pypdf extraction is a
separate ticket — current scope is HTML-shaped sources.
- Mendelson + Enderton (proprietary; await #000031 §2.1
decision).
- Wilf generatingfunctionology (license forbids redistribution).
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
Three streams. Two land in the repo; one lands in fox's Downloads
(existing g4 packs live there too).
Witness sweep automation
========================
`bench/scripts/witness_sweep_cron.sh` — schedulable harness that
runs `make bench-witness-sweep` + `make bench-witness-divergence`
back-to-back and (with --commit) auto-commits any new divergence
fixtures + sweep result. Fail-closed: pre-commit hook failure
aborts; non-main-branch refuses to commit.
$ bench/scripts/witness_sweep_cron.sh # dry run
$ bench/scripts/witness_sweep_cron.sh --commit # auto-commit
$ bench/scripts/witness_sweep_cron.sh --commit --push # full pipeline
Recommended cron (every 6 hours; LLM cost scales with frequency):
0 */6 * * * \
bench/scripts/witness_sweep_cron.sh --commit --branch main \
>> /var/log/arborist-witness.log 2>&1
Smoke this commit: dry-run produced 5 fresh divergences (10
cumulative across two extraction passes). The witness-sweep + 5F
extractor pipeline runs unattended now; calibration-data corpus
will grow over time.
#000018 — adversarial soft-hash covert-channel analysis
========================================================
`docs/soft-hash-channel-analysis.md` — formal threat model + channel-
capacity definition + cryptographic reduction + mitigation table
+ residual analysis. Doc-only deliverable per the ticket's hard
constraint.
Headline result: under threat models T1 (data poisoner) and T2
(gradient adversary), the channel is bounded by SHA-256 partial-
preimage hardness when φ (the anchor map) is independent of
parameter geometry. T3 (hyperparameter adversary) requires
M2 (per-checkpoint anchor nonce) for boundedness.
Recommended mitigation: M2 (per-checkpoint nonce) — 32 bytes added
to the snapshot manifest, no training-ergonomics regression,
closes T1+T2 channels under any φ choice and bounds T3 by
per-checkpoint window. Fall back to M1 (PRG-based φ) if M2-only
deployment surfaces structural concerns. M3 (drop anchor entirely)
stays in reserve as the strict-construction fallback.
Three open questions (§9): Hessian alignment under φ_linear,
PRG choice for φ_PRG, and explicit T3 per-window bound. Each is a
follow-up ticket.
Ticket #000018 status: closed · landed 2026-05-09 (analysis doc).
v7 § 9.10 spec amendment proposed in §7 of the analysis.
#000033 — pillar VII (combinatorics), Claude-authored
======================================================
NOT committed to the arborist repo (the existing g4-v2 packs live
in `/home/fox/Downloads/` too — that's the operator's bundle
location). Two new bundle files at:
/home/fox/Downloads/axiomsclaude-vii-v1.json (7 axioms)
/home/fox/Downloads/theoremsclaude-vii-v1.json (7 theorems)
Pillar VII covers combinatorial counting — the gap between Grok's
pillars VI and IX in the v2 packs:
axioms (7): addition principle · multiplication principle ·
pigeonhole principle · factorial definition ·
binomial coefficient definition · Pascal's rule ·
empty-set / boundary axiom
theorems (7): binomial theorem · inclusion-exclusion (counting
form) · hockey-stick identity · Vandermonde's
identity · Catalan number closed form · stars-and-
bars · strong pigeonhole
Each record in the dual-thread format the existing g4 packs use
(Δ symbolic LaTeX + ∇ verbose prose + ∇ concise + sigil + formal
language + role + status + source_reference + date + foundational
group + category + subfield). Per fox's directive: explicit
authorship metadata everywhere — `authored_by: Claude (Anthropic)
— model claude-opus-4-7`. NOT Grok-generated; no silent invention.
Each record carries `pi_star_ref: combinatorics@v1` so the kernel
binding is explicit. Theorems list `depends_on_axioms` arrays so
each theorem cites the foundation axioms it bottoms out on.
Smoke test (committed alongside):
$ arborist --db /tmp/test.db ingest --source claim_pack \\
--bundle /home/fox/Downloads/axiomsclaude-vii-v1.json \\
--bundle /home/fox/Downloads/theoremsclaude-vii-v1.json
→ 14 docs, 14 chunks, 0 cross-bundle edges
Source attributions: Stanley EC1, Brualdi Introductory
Combinatorics, Knuth TAOCP Vol 1, plus historical sources where
applicable (Pascal 1654, Vandermonde 1772, Dirichlet 1834, Catalan
1838, Feller 1950 for stars-and-bars).
Tests: 1636 passed, 37 skipped (no regressions; pillar VII
ingestion smoke covered above).
Three small streams:
#3 — close#000030 properly
============================
All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:
Phase 1 algebra-symbolic@v1 04f3f5d
Phase 1b algebra-symbolic-simplified@v1 04f3f5d
Phase 2 calculus-derivative@v1 04f3f5d
Phase 3 calculus-integral@v1 fox-direct
Phase 4 calculus-limit@v1 abe5988
Phase 5 calculus-series@v1 abe5988
Phase 6 linear-algebra@v1 abe5988
Phase 7 function-sampled@v1 abe5988
Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.
#5 — composition fixtures across new SymPy π*'s
================================================
12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:
- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
fingerprint is order-sensitive; composite domain == inner domain;
composite bytes == manual chain bytes.
Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().
#4 — end-to-end witness sweep against real shards + Hermes
===========================================================
New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.
`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.
First real sweep (this commit, against Hermes-3-8B):
agreement label count rate
KERNEL-LLM-DIVERGED 5 62.5%
KERNEL-LLM-AGREE 3 37.5%
───────────────────────────────────────────
divergence_count 5 62.5%
wall median / max 130 ms / 1.1 s
Hermes diverged on 5/8 of the canonical-shape questions:
- said `1/10` for `0.1 + 0.2` (kernel: `3/10`)
- said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already
expanded — but Hermes ALSO
emitted the unexpanded form
when given the expanded
form, vs the kernel's
deterministic expand)
- and 2 more.
These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.
Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).
Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.
## Lazy concept_relations loading
Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.
Refactored arborist/concepts/query.py:
- New _load_neighbors_for(shards_dir, tokens) — targeted
WHERE token IN (...) OR target IN (...) query. Returns just the
direct synonym neighborhood for the given tokens (~300 rows for
a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
expansion exceeds max_total (the cap). Most queries never reach
the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
loaders. Eager _load_indices / _get_indices kept for any
back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.
All 14 concept tests pass unchanged — the contract is preserved.
Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):
metric pre-fix post-Phase-1 post-lazy-concepts
wall_ms 14,500 13,400 9,300 (-36%)
search_ms 9,900 10,600 5,000 (-49%)
SQLite executes 10,623 3,687 3,708 ~same
synonym_expand 2,966 2,840 ~0 (lazy hit)
Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).
## Real-shard baseline artifact (Phase 2)
bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:
- bench/results/real-shard-baseline.json (durable; commit_sha,
shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)
Question set in bench/fixtures/real-shard-baseline-v1.jsonl:
virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
(rivalry path), and two canonical-projection cases (math + logic
preflight short-circuit).
First baseline run (commit e78814c plus this fan-out, BURN=1):
audit_mode n notes
STRICT 4 (virt-back, France, Mac OS X, Linux)
HYBRID 2 (Microsoft founder, AMD/Intel)
CANONICAL 2 (0.1+0.2, A IMPL B; <1 ms each)
wall median 4.1 s (range 0.6 ms – 7.2 s)
primary used 4 / 8
`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).
Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.
`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.
## Test status
Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
