Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl.
The parser was line/bullet-only — one line ⇒ one claim — so a line
that crammed several pointered claims onto one row ("Water is wet
[E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became
one monolithic claim with all the pointers, and a wrapped bullet
became two.
arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';',
sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes
(' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since
'/' while ', inline '(N)' enumeration markers, and commas — with
'(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it.
_segment_line keeps the split ONLY IF every resulting non-empty
segment is a well-pointered claim — a legit single claim ("The cat
is black and white [E1].", "The cast: A, B, C [E1].") is never
broken because splitting it would manufacture pointer-less prose
fragments → guard rejects; a leading colon-terminated header with no
pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a
wrapped-bullet join: a continuation line (leading whitespace then a
lowercase letter, no bullet glyph) folds its text + pointers into the
previous claim.
Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling
— a harder Formulate tier would re-open below-ceiling headroom; a
#000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE
recombinations in falsification-hard (Mercury, Einstein — step 2.2).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims
feeds the 450 claim_lattice_pointer + claim_lattice cells) after
(bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline
(...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55,
pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise
floor. Per-row diff: the segmenter changed the parsed-claim count on
the SAME answer text for 7 of the 450 lattice cells (0 in
claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an
audit_mode change — both correct: a wrap-join recovered an answer's
intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID)
into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1
monolithic claim, all pointers → STRICT) split into 8 claims, some
not individually verifying → HYBRID (the honest verdict — false-
positive STRICT was the corruption). Every other lattice/quote delta
is LLM re-answer variance. No regression — the segmenter's only
visible effects on real traffic are honest improvements. Summarized
in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4.
Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction
splits; pointerless-fragment + cast-list guards; leading-colon-header
drop; wrapped-bullet join; pointer-order/multi-pointer); existing
parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack
re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped.
#000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes updated.
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in
falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a
multi-word proper noun matches the source — but "Insulin was
discovered by Alexander Fleming" against "Penicillin was discovered by
Alexander Fleming" matches on the shared "Alexander Fleming" while the
swapped subject "Insulin" (the falsehood) is ignored.
arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx)
flags a >4-char Capitalized content token (stopword-filtered) or a
digit-number in the answer absent from the source.
_is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in
verify_quotes' entity branch (proximity policy) in the weakest-grounding
slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence
AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is
what keeps a structured multi-claim summary untouched — the Matrix cast
list (many entities, a tight cluster) and the TMNT answer (a numbered
list with parenthetical nicknames the source omits): model-added
accurate detail in a real summary isn't a contradiction, only the
single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid
entity-path regression tests still pass, pinned untouched.
Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin
/ 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it
newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard
pack around) and 5f-fal-live-028 — had expected_reason updated
HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes
actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE
recombinations (Mercury, Einstein — step 2.2) + 8 Formulate
mis-segments (step 2.4).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after
(bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline
(...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54,
pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise
floor. Per-row diff (675 common cells, 30 quote-mode rows changed
audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity
path — the gate fired on 0 legitimate QA answers in the whole bench.
Every transition was LLM re-answer variance (verifier quote→quote with
the verdict flipping); pointer/lattice deltas are noise too (the gate
is in verify_quotes / quote mode, not the claim-lattice verifier). No
regression — the gate is provably narrow on real traffic. Summarized in
qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1.
Tests: 4 new in test_verify.py (_is_single_sentence helper,
_entity_salient_disagrees helper, swapped-subject → UNGROUNDED,
gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling
re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated
(positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5
dilution; ACCEPT via a degraded-parent sub-scenario).
make test 2343 passed, 28 skipped.
#000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes / baseline JSON updated.
`make chain-check-shards` / `chain-check` previously only counted dangling
prev_event_hash references — it missed forks (two events chained off the
same head; qa.db seq 7724/7725 was that, and the cheap check reported 0).
CHAIN_CHECK_SQL now reports the sum of: dangling prev refs + forked
parents (a prev_event_hash claimed by >1 row) + extra genesis rows
(>1 row with prev_event_hash IS NULL). 0 = a single linear chain.
(Working-tree Makefile also carried in-flight bench-5f-* target additions
from a concurrent session; they ride along in this commit.)
Closes the harder-fixture-tier ticket: a real verify_quotes
tightening lifts the falsification-hard rate 4/12 → 6/12, bench-gated,
ForkScore's bench-Δ goes positive on it — the loop is closed
end-to-end.
arborist/qa/verify.py: _numeric_signature(text) extracts comma-
stripped digit-runs ('8,849' and '8849' collapse; '300' stays
distinct from '300000' ← '300,000'). _check_each_with_paraphrase
gains a gate: a span that token-covers the source ≥ paraphrase_coverage
but asserts a digit-number the source lacks (modulo thousands-comma)
is no longer paraphrase-grounded — it goes to unverified. Catches the
near-miss the lexical coverage check is blind to ("Water boils at 50
degrees" against a source saying 100 token-covers 100% because
'50'/'100' aren't >4-char content tokens). Narrow by construction:
fires only on the paraphrase fallback (verbatim/span/entity/claim-
lattice paths untouched), only on a digit-number. A rounding-
paraphrase demoting here is the honest verdict — it isn't a verbatim
grounding.
Hard pack: lifts 5f-fal-hard-004 (50 vs 100) and -007 (300 vs
300,000) to UNGROUNDED → falsification-hard rate 4/12 → 6/12 = 0.5.
The other 6 over-grounds (4 HYBRID_ENTITY + 2 recombined-no-number
STRICT_PARAPHRASE) and the Formulate hard pack are unaffected —
headroom for a bigger order/dependency-aware verifier upgrade, an
optional follow-up, not a #000046 blocker.
Bench gate: make bench-qa (n=3 × 75 questions × 3 modes = 675 cells)
before / after. STRICT-rate quote 0.53→0.50, pointer 0.23→0.25,
lattice 0.44→0.45 — all within the 5-pp noise floor. Per-row diff
(675 common cells, 74 changed audit_mode): the only clearly
gate-attributable QA shift was the fictional "our cold fusion
breakthrough" year-claim demoting STRICT→HYBRID (×3 samples) — a
correct demotion; every other transition was quote→quote /
claim_lattice→claim_lattice LLM re-answer variance. No regression on
legit answers. Artifacts: bench/qa_results/2026-05-11T13-42-38Z (before)
and ...T14-19-51Z (after) — gitignored; summarized in qa-modes-bench.md
Addendum 5 + ticket-000046 §5 Phase 3.
Worked example: fork_score on the real change — parent {5f/
falsification: 4/12} → child {5f/falsification: 6/12} → γ·Δ5f =
(1/6)/5 ≈ +0.033 > 0 (positive; a single improvement of this size is
MARGINAL by the ÷5 dilution, the test pins the full-lift-to-1.0 case
at ACCEPT).
Tests: 5 new in tests/test_verify.py (numeric_signature normalization
+ subset-matches-comma-variant + disagreement-rejected +
gate-is-narrow + number-present-still-verifies);
test_5f_falsification_hard_pack_below_ceiling re-pinned 4/12 → 6/12.
make test 2339 passed, 28 skipped.
#000046 → closed; #000012 §8 §4 + TICKETS.md row + falsification-hard
_meta / hard-004,007 notes + Makefile 8/12→6/12 comments updated.
(Makefile also carries an uncommitted chain-check-SQL improvement from
the concurrent session — NOT in this commit; staged only the
#000046-comment hunks.)
Extends the #000046 hard tier to the Formulate sub-battery.
bench/fixtures/5f/formulate-hard-v1.jsonl — 12 prose inputs that
arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a
particular claim lattice (recorded in expected_lattice). The parser
is line/bullet-based — one line ⇒ one claim, [E#] tokens attach to
it — so 8 of 12 it mis-segments: merges and-/semicolon-/dash-joined
or (1)(2)-enumerated multi-claim lines into one claim with all the
pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD
on claim-count mismatch; the other 4 are well-formed bullet/numbered
lists / single claims the parser handles right. Rate at HEAD = 4/12
= 0.333, stable (parse_pointer_claims is deterministic). A
claim-lattice parser that does sentence/clause segmentation (split on
'. ', ';', subordinating conjunctions, inline enumerations) + joins
wrapped bullets lifts the rate toward 1.0 → positive γ·Δ5f for that
child fork.
make bench-5f-formulate-hard runs it (|| true past the runner's
nonzero-on-failures exit). Not in `make bench-5f` / `runner --all`.
tests/test_bench_batteries.py — test_5f_formulate_hard_pack_below_ceiling
(pins rate 4/12, source=live, the 8 fails are claim-count
mis-segments).
#000046 → "Phase 1 + Phase 2 landed"; two below-ceiling 5F subs now
exist (falsification, formulate). Closure still pending an actual
surface improvement (verify_quotes tightening — bench-gated — or
parse_pointer_claims segmentation) that lifts a rate. §5 + §6 +
TICKETS.md row updated.
(Makefile also carries an uncommitted chain-check-SQL improvement
from the concurrent #000039 session — NOT included in this commit;
staged only the bench-5f-formulate-hard hunk + the .PHONY line.)
Closes the "everything is at rate 1.0 so fork_score's bench-Δ terms
are inert" gap the #000025 §10.14 calibration surfaced — at least on
the 5F/falsification axis.
bench/fixtures/5f/falsification-hard-v1.jsonl — 12 near-misses, each a
FALSE/unsupported claim whose correct verdict is UNGROUNDED (recorded
in expected_reason). 8 of 12 are over-grounded by
arborist.qa.verify.verify_quotes at HEAD — its paraphrase
token-coverage strategy returns STRICT_PARAPHRASE, its entity-proximity
strategy returns HYBRID_ENTITY, both matching on incidental overlap
(shared entities/numbers, the same key terms stated in the opposite
direction) — so those tasks fail by design; the other 4 the verifier
handles correctly. Rate at HEAD = 4/12 = 0.333, stable (verify_quotes
is pure-lexical / deterministic). Built around the pre-documented gap
5f-fal-live-003.
make bench-5f-falsification-hard runs the pack; make
bench-fork-baseline-hard pins it to
bench/results/baseline-falsification-hard.json. Both targets `|| true`
past the runner's nonzero-on-failures exit (8 fixtures fail by design;
the JSON is still written).
tests/test_bench_batteries.py — test_5f_falsification_hard_pack_below_ceiling
(pins rate 4/12, source=live, every fixture asserts UNGROUNDED, the 8
fails are over-grounds not abstentions) +
test_fork_score_positive_gamma_5f_on_hard_falsification_improvement
(the worked example: fork_score(parent={5f/falsification: 1/3},
child={5f/falsification: 1.0}) → gamma*Delta5f ≈ +0.133 > 0, verdict
ACCEPT, no regression flags — the bench Δ-rate carrying signal it
can't carry while every canonical pack is at ceiling).
NOT in `make bench-5f` / `make bench-5s5t5f` / `make
bench-fork-baseline` / `runner --all` — the hard pack is a separate,
deliberately-failing artifact pinned on its own.
#000046 flipped to "in progress · Phase 1 landed"; closure pending an
actual verify_quotes tightening that lifts the rate (a separate,
larger task). ticket-000012 §8 §4 + TICKETS.md row updated.
Full suite: 2314 passed, 28 skipped.
Closes the three open Phase-1b items of #000025; every §10 closure
criterion is now met, so the ticket flips to closed.
§10.14 — ForkScore threshold-calibration handoff to #000012.
bench/scripts/fivef_threshold_calibration.py (make bench-5f-threshold-
calibration) runs the canonical 5S/5T/5F packs + the 5F live packs and
reports baseline rates, observability granularity (1/n), and fork_score
verdicts on the parent vs synthetic child perturbations →
bench/results/5f-threshold-calibration-2026-05-11.md. Findings written
into ticket-000012 §8: keep SIGNAL_FLOOR / HARD_REGRESSION_FLOOR at
0.05; the small 5S packs (syntax n=10, semantics n=8) are coarser than
the floors so any regression there trips hard-reject (intended zero-
tolerance); the 5x averaging dilution in _delta_*; ceiling saturation
(every pack at 1.0 -> delta-rate terms <= 0). No constant change
shipped. 6 tests in tests/test_fivef_threshold_calibration.py.
§10.13 — feedback latency / efficiency on real workload.
run_feedback_loop now computes feedback_latency (listed in §5.5 since
Phase 1a, never implemented) — wall-clock seconds to apply a live
chain against its temp shard, surfaced per-task
(feedback_latency_seconds) + battery (feedback_latency_mean_seconds,
feedback_live_task_count). For live chains feedback_efficiency's cost
denominator switched from len(chain) (count of requested ops) to the
persisted footprint _persisted_cost = audit-event rows the chain
actually wrote + their body bytes / 1e6. Embedded chains keep
len(chain) and report feedback_latency_seconds = None. Latency is a
wall-clock field (run-to-run variable, like BatteryResult.timestamp)
and is not a fork_score input. 3 tests in tests/test_bench_batteries.py.
§10.11 — real selfmodel finetuning chains.
bench/scripts/selfmodel_chain_snapshot.py (make bench-5f-selfmodel-
snapshot) appends one chained SelfModel snapshot per run to a
persistent shard (~/.arborist/shards/selfmodel-chain.db, override via
ARBORIST_SELFMODEL_CHAIN_DB) with one CapabilityClaim per sub-battery
(metric = "5S-syntax" etc., measured_value = that pack's rate,
eval_digest = the pack's fixture digest, threshold = SIGNAL_FLOOR).
snapshot() auto-parents, so each snapshot is a distinct root and the
lineage grows by one per run. run_finetuning gains a third dispatch
mode — shard-chain (gated on a task's selfmodel_shard key) — via
_chain_finetuning_measure: reads the two most-recent snapshots
(latest() = child, its parent_selfmodel_root = parent) and measures
improvement on target_capability between them. This is the real
lineage replacing Phase-1a's synthetic parent->child pairs; the
chained delta reflects genuine cross-run drift (0.0 today — the
embedded packs are at ceiling). Operator pack
bench/fixtures/5f/finetuning-shardchain-v1.jsonl (6 tasks) + make
bench-5f-finetuning-shardchain; not in `make bench-5f`, `make test`,
or a fresh checkout (a missing/too-short chain fails honestly). The
real chain shard was bootstrapped 2-deep on 2026-05-11; make
chain-check-shards reports 0 breaks on it (and all other shards).
10 tests in tests/test_selfmodel_chain.py.
Full suite: 2311 passed, 28 skipped.
Per Finding 3, a uniform τ_qa=7d filtered out every recent
CANONICAL_PROJECTION row (the π* graduations from #000027/#000030/
#000032 are all younger than 7d), so the sweep saw zero high-value
kernel-only work. Splitting τ_qa by audit_mode lets the cheap kernel
re-probe path (CP) run on a short cycle while the expensive LLM
re-witness path (STRICT/HYBRID/UNGROUNDED) keeps the long cycle.
bench/scripts/prometheus_sigma_sweep_dryrun.py: new build_tau_by_mode()
helper + per-mode CASE in iter_target_a_candidates; sweep_target_a now
takes the dict instead of a single seconds value. New CLI flag
--tau-qa-cp-days (default 1d); --tau-qa-days now scopes to LLM-witness
modes only (default 7d). Report renders the per-mode τ table in the
header and marks Findings 2 and 3 RESOLVED with their landing commits.
Makefile: PROMETHEUS_SWEEP_TAU_DAYS bumped to 7 (was 1, the prior
Finding-3 workaround); new PROMETHEUS_SWEEP_TAU_CP_DAYS=1 makevar.
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md: regenerated
under the new defaults — 9 CP rows surface alongside 1,904 LLM-
witness candidates → 1,913 total Target A candidates → 1 ACCEPT,
2 MARGINAL, 205 REJECT, 271 DEFERRED chunks; 2 cache_drift vetoes
preserved end-to-end.
docs/tickets/ticket-000037 §22: Findings 2 + 3 marked RESOLVED with
landing-commit references; total dry-run cost line updated to the
new measurements (37.6 ms / 3,913 branches / 9.6 µs per branch).
Closes the controller → 5F battery loop fox designed: #000037 Phase
1 produces FalsificationFixtureProposal records from high-divergence
providence_cache rows; this commit lands the harvester that turns
those proposals into a real 5F fixture pack the falsification
battery exercises every test run.
bench/scripts/harvest_falsification_proposals.py — reads qa.db,
filters live rows with witness_divergence = (n_unverified / n_quotes)
>= 0.5, stratifies by audit_mode, picks 20 HYBRID + 20 UNGROUNDED
top-by-cache_key for determinism, writes embedded-mode fixtures
to bench/fixtures/5f/falsification-harvested-v1.jsonl.
Each fixture carries `_harvest_meta` with the source cache_key,
divergence at harvest time, audit_mode at harvest time, and the
ticket reference (#000037 §13 step 11). Embedded mode — uses
`observed_violations` directly without calling verify_quotes
again; `answer_text` preserved verbatim for debugging.
Pack composition (initial harvest 2026-05-10):
20 UNGROUNDED (expected_reason: UNGROUNDED)
9 HYBRID_QUOTE (NEW motif — not in falsification-v1.jsonl)
7 HYBRID_CLAIM_LATTICE (NEW motif — not in falsification-v1.jsonl)
4 HYBRID_PARAPHRASE (already covered in v1)
Two harness tests pin the pack:
- test_5f_falsification_harvested_pack_runs_clean: 100% pass-rate,
every fixture carries traceable _harvest_meta.
- test_5f_falsification_harvested_pack_widens_motif_coverage: the
HYBRID_QUOTE / HYBRID_CLAIM_LATTICE motifs surface from real
corpus (loud-fail if harvest rotation drops them).
Makefile: `make bench-5f-harvest` re-runs the harvester. Parameters
exposed: HARVEST_QA_DB, HARVEST_OUT, HARVEST_THRESHOLD,
HARVEST_SAMPLE_PER_BUCKET.
Full suite: 2328 passed, 37 skipped (+2 from the two new pins).
Phases 1 (controller) and 2 (sibling-table audit writes) landed in
prior commits. This commit adds the Phase 3 dry-run simulator
instead of the actual sleep-sweep scheduler, since Phase 3's value
is mostly in what we'd learn from running it — and the dry-run
captures those findings without committing to a scheduler design
prematurely.
bench/scripts/prometheus_sigma_sweep_dryrun.py — read-only
simulator that classifies §3 Target A (providence_cache) + Target B
(documents) sweep candidates, synthesizes ControllerBranches from
real shard data, runs the Phase 1 controller, reports decision
distribution + Phase-3-design findings. No LLM calls, no
mutations.
make prometheus-sweep-dryrun — produces a dated markdown report
at bench/results/prometheus-sigma-sweep-dryrun-YYYY-MM-DD.md.
Five findings surfaced by three dry-run iterations against the
live ~/.arborist/shards corpus (3.5M docs + 2839 providence_cache
rows) — captured in ticket §22:
1. chunk-size dominates Kelly threshold (must = Hermes
concurrency, not candidate pool)
2. flat capital_cost blocks every allocation (split kernel-cost
vs LLM-cost on the contract)
3. τ_qa=7d filters every CANONICAL_PROJECTION row (all 29 are
<7d old; need per-audit-mode τ)
4. Target B canonical-shape detection is the real headline
(~152K candidates extrapolated; controller correctly returns
MARGINAL on shape-match chunks)
5. quarantined rows correctly veto via cache_drift hard-veto
Mean per-branch controller latency in dry-run: 12.5 µs at
chunk_size=4. Phase 3's actual bottleneck is the witness fan-out
(Hermes calls), not the controller itself.
Ticket #000037 status flipped to in-progress with Phases 0+1+2
landed; Phase 3 scheduler remains future work but is informed by
the five findings.
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.
- New Citation.via_citation_alias field (default False, preserves
parse-from-source-ref path).
- warrant_status sets via_citation_alias=True on substitute
Citations from #000041 lookup_citation_aliases.
- resolve_chunks reads citation.via_citation_alias as a "floor" for
via_alias on every match it produces (Pass 1 hits inherit it
too, not just Pass 2 term-alias hits). Audit-honest: matches
from a substitute Citation are alias-driven regardless of which
cascade pass found the chunk.
Live re-resolve under the new attribution: 18 direct + 74 +alias
(was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
records resolving on the literally-cited Hilbert textbook. All 74
records resolved via citation-alias substitution now carry
process_id="warrant-resolver-v1+alias" in derivations.
3 new unit tests (test_warrant_resolver.py): default-False on
parsed Citations, explicit-True construction works, resolve_chunks
propagates the floor onto every ResolutionMatch.
B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.
- HtmlPageSource accepts default_author kwarg; appends ', by
<author>' to ingested document titles when the <title> tag
doesn't already include the surname.
- TextbookTexSource accepts default_author kwarg; appends ' by
<author>' to titles when the LaTeX has no \author{} macro AND
no PG-style 'Author:' boilerplate.
- _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
kwarg; same append logic. ingest_crawled() and arborist crawl
--ingest plumb it through.
- arborist ingest --author + arborist crawl --author CLI flags.
- bench/scripts/textbooks_manifest.py:cmd_lookup emits the
manifest's `author` field as a 7th tab column.
- make textbook target reads the author column and threads
--author into both crawl-ingest and shallow-ingest paths.
Idempotency preserved — surname-already-in-title detection prevents
double-stamping on re-ingest. Shards previously SQL-backfilled
(Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
Grinstead-Snell) keep their existing titles; new ingests pick up
the author signal at source time.
Live smoke: arborist ingest --source html --author "Bertrand
Russell" against PG #41654 yields title "Introduction to
Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
with no SQL UPDATE needed.
Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
Caught while sweeping for remaining "arborist v8" CLI invocations
after bae5caf renamed the subcommand. Makefile target
``bench-fork-score`` (line 430) still called
``$(ARBORIST) v8 score`` directly — would have failed loudly on
next ``make bench-fork-score`` run with an argparse error
("invalid choice: 'v8'").
Same kind of subtle breakage I caught in fork_score.py earlier
today (85be5eb): a refactor's text-search misses a callsite that
isn't picked up by Python imports. Lesson: after renaming a CLI
subcommand, grep ``$(ARBORIST) <oldname>`` AND ``arborist <oldname>``
in Makefile, scripts/, and bench/ — argparse errors don't surface
until someone runs the target.
Hygiene
=======
- ``grep -rn "arborist v8\\b" Makefile bench scripts docs`` → only
the historical-note comment in cli.py:5131 remains.
- ``make help | grep substrate`` → target descriptions accurate.
- ``.venv/bin/arborist substrate score --help`` still 0.
Under fox "work all paths" directive, four orthogonal lifts:
(1) 14 Levin alias rows for pillar VII compound citations — each
Stanley/Brualdi/Knuth ref now has BOTH Bogart AND Levin as
substitute. Lifted +2 (hockey-stick + stars-and-bars).
(2) 6 Goedel multi-substitute aliases — Goedel citations now have
4 substitutes each (Peano, Russell IMP, Dedekind, SF-LF). Lifted
+4 pillar III records (Axiom of Zero / Distinctness / Addition-
Successor / Multiplication-Successor variants).
(3) 2 SF-LF alias rows for pillar I formal proof axioms (Mendelson +
Enderton -> SF-LF). SF Logic.v has 12 hits "excluded middle",
40 "existential", 1 "double negation". Lifted +2 (Existential
Introduction, Double Negation Elimination).
(4) Grinstead-Snell Introduction to Probability ingest (GFDL,
Dartmouth CHANCE Project). Initial LibreTexts BFS crawl thin
(24 chunks nav-only at depth=2). Switched to Dartmouth PDF
-> pdftotext -layout -> single-page HTML on localhost:8765
-> standard arborist html ingest. Result: 339 chunks of
substantive probability content. Lifted +1 (Inclusion-Exclusion).
New ingest pattern useful for any PDF-only textbook.
Per-pillar:
I: 10/13 -> 12/13 (+2)
II: 9/10 stable
III: 7/13 -> 11/13 (+4 — biggest win)
IV: 18/18 stable
V: 1/5 -> 2/5 (+1)
VI: 3/5 stable
VII: 9/14 -> 11/14 (+2)
IX: 11/14 stable
Total: 77/92 (84%)
7 term aliases registered (physics Newton-vocab + arithmetic Peano-
vocab) but most didn't lift — cascade picks different tokens than
my aliases target. Cascade tuning would require code work in
_build_record_query_cascade / _phrase_for_axiom. Deferred.
Cumulative day-of-2026-05-10: 18 -> 77 / 92 (+59 records, +64 pp).
15 records still stuck. Top blockers:
- Cascade-tuning for substrate-content-exists-but-FTS5-misses
(Kolmogorov 3 axioms, Newton third law, Vandermonde, Catalan,
Injectivity of Successor, etc.) — code work
- PM *24+ not transcribed (Russell's Paradox Resolution)
- Boehm-Jacopini esoteric (no PD substitute)
Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification
controller) is doc-only and gates Phase 1 on §12 measured-pressure
triggers. This commit lands the empirical-evidence harness fox needs
for the go/no-go decision, plus the §16.2 test scaffolding so the
contract is discoverable from the test runner today.
bench/prometheus_sigma_trigger_probe.py
=======================================
Read-only walk of audit_events + capital_ledger across shards.
Reports each §12 trigger:
- Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a
fork_score branch-set table; reports "no data — single-validator
ForkScore Phase 1a" when absent. Trigger structurally cannot fire
until #000012 multi-branch persistence lands.
- Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10):
aggregates providence_canonical_witness audit-event bodies,
maps each agreement_label per #000028 §1.2 to a binary
LLM-divergence score, computes mean/stddev/ratio. Defensive on
the max(mean,ε) guard from §12 itself.
- Trigger 3 (witness cost share > 0.30): aggregates capital_ledger
rows; uses `material` (kWh proxy) as the canonical compute axis;
surfaces both `material` and `financial` so fox can pick a
different form if needed. Tags the report with the caveat that
the current ledger reflects ad-hoc activity, not a controlled
#000026 sweep.
- Trigger 4 (operator mission need): n/a — operator decision.
Pure measurement, no LLM, no schema change, no mutation. Output is
a markdown report at $(PROMETHEUS_PROBE_OUT) (default
bench/results/prometheus-sigma-triggers-<utc-date>.md).
Wired via `make prometheus-trigger-probe`.
bench/results/prometheus-sigma-triggers-2026-05-10.md
=====================================================
First captured baseline. Verdict on current shards:
Trigger 1: NO (no fork_score branch-set table; #000012 Phase 1a
is single-validator)
Trigger 2: NO (16 samples; N_min=30. But mean=0.625, σ=0.5,
ratio=0.8 — both ratio AND abs floors would fire
if N reaches 30. Signal is there; just needs more
samples.)
Trigger 3: NO (witness/total material = 0.005, well under 0.30
threshold. Caveated: ledger is ad-hoc, not a
controlled #000026 sweep.)
Trigger 4: n/a (operator-stated)
Empirical answer: no §12 measured-pressure trigger has fired yet.
Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4.
tests/test_prometheus_sigma.py
==============================
17 skip-stubs pinning the §16.2 acceptance contract. Each test:
- Collects today via `pytest --collect-only` so the test surface is
discoverable.
- Skips with reason "Phase 1 not landed; controller module
arborist/v9/prometheus.py absent" until that import succeeds.
- Carries a one-sentence intent line tying it back to a numbered
ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4).
When Phase 1 lands, the implementer adds the controller module
under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE
becomes True; each test gets its body filled in. The names, intents,
and skip-reason strings are the durable contract.
Hygiene
=======
- make test → 1623 passed, 45 skipped (was 28; +17 new skips).
- make chain-check-shards → 0 breaks across all 7 shards.
- No code touched outside the new probe + scaffolding files +
Makefile target wiring. fox's in-flight #000037 ticket
modifications left untouched.
Three orthogonal lifts under fox 2026-05-10 "all of the above"
directive:
(1) 7 medium-confidence citation aliases per #000041:
Mendelson -> Russell IMP + De Morgan FNL + Boole Laws of Thought
Enderton -> Russell IMP + De Morgan FNL
Jech -> Cantor 1915 Contributions
Landau -> Russell IMP
Each row carries partial-coverage caveat in decision_rationale.
(2) Peano Arithmetices Principia 1889 ingested via existing
textbook_tex pipeline. Source: mdnahas/Peano_Book GitHub
repo (CC-BY-SA-4.0, Verheyen+Nahas English+Latin LaTeX).
paracol multi-column macros leak some residue but substantive
text extracts (39 chunks, 35 hits on "peano", 38 on
"arithmetic"). Make target textbook-peano. Manifest entry
peano-arithmetices-principia-1889. Plus 3 substitute-citation
aliases:
Landau -> Peano (lifts 2 pillar III records)
Goedel -> Peano (2 variant strings)
(3) Title-from-author backfill on Cantor / Russell IMP / Bogart /
Judson / Peano shards. Empirical defect: HTML/textbook_tex
sources took titles from <title> tags or URLs which on
Wikisource + PG do NOT include author surname, so the
resolver's _shard_matches_citation heuristic (requires
cited-author surname in title haystack) failed silently for
all alias-resolved chains where the substitute author wasn't
self-evident in the shard content. Surgical SQL UPDATE on
documents.title fixed it. Single biggest lift in this turn:
+13 records.
Per-pillar:
I: 4/13 -> 7/13 (+3, Russell IMP / Boole / De Morgan)
II: 0/10 -> 5/10 (+5, Cantor)
III: 0/13 -> 7/13 (+7, Peano + Russell IMP)
IV: 18/18 stable
V: 0/5 stable (Kolmogorov URAA-blocked)
VI: 3/5 stable
VII: 9/14 stable
IX: 5/14 stable
Total: 54/92 (59%)
Title backfill is a one-off SQL; the proper long-term fix is in
the source ingest pipelines (HTML / textbook_tex should pull
author from manifest's `author` field at ingest time and prepend
to document.title). Tracked under #000031 Phase 1 follow-up.
Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of
Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy
(PG #41654, PD), and Judson Abstract Algebra Theory and Applications
(GFDL-1.3) to the textbook manifest, plus per-book make targets +
a `textbooks-base-knowledge` bulk target. Idempotent at DB layer.
These four primaries cover the proprietary citations across pillars
I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote)
that dominate the 74 unresolved claim-pack records. They land as
honest "no match" under the existing resolver `_shard_matches_citation`
heuristic (cited-author surname required) — full lift gates on
fox citation-alias decisions per #000041.
Total ingest delta: 43 docs / 295 chunks across four crawl shards.
Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
records per-book license rationale, projected lift table, and the
remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau,
Gödel, Kolmogorov, Church) for which no clean PD HTML edition was
located in this session.
Pipeline was disabled 2026-05-02 because CI runners overlapped
with bench infrastructure. Today the substrate is in a good
state (1,684 passing locally) and the gate has been off long
enough that drift would land silently if reintroduced.
Re-enables the workflow + adds a `test-ci` Makefile target that
skips the wikipedia ingest test files (test_wikipedia_old.py,
test_wikipedia_xml.py). Their runtime path is already exercised
end-to-end by `make ingest-*` against real shards, so the
synthetic fixtures duplicate coverage without adding signal CI
should gate on.
`make test` stays comprehensive for the local dev loop —
1,684 passed there, 1,560 passed under `make test-ci` (the
124-test delta is the wikipedia ingest fixtures).
PYTEST_XDIST_AUTO_NUM_WORKERS=4 still caps the runner's
parallelism so a re-enabled CI doesn't oversubscribe the
shared host or skew live bench latencies on the same pool.
Three concrete improvements to the textbook ingest layer:
Per-book make targets
=====================
- `make textbook ID=<id>` — generic, looks up one entry from the
manifest and runs the right ingest path (BFS crawl if
`crawl_url` is set, shallow URL ingest otherwise).
- `make textbook-list` — list ingestable textbook ids.
- Convenience targets per active manifest entry:
textbook-bogart, textbook-keller-trotter, textbook-levin,
textbook-aristotle-prior, textbook-aristotle-posterior,
textbook-newton, textbook-morin
Each per-book target writes to a dedicated shard at
`$(CRAWL_SHARDS_DIR)/textbook_<id>.db` — one shard per textbook
id rather than one per host (which would conflate multiple
Wikisource books — Aristotle Prior, Posterior, Newton — into
one shard).
The manifest helper script gains `ids` and `lookup <id>`
subcommands so the Makefile can introspect manifest entries
without grep gymnastics.
Idempotency contract documented
================================
Every ingest path is content-addressed. Same content →
same `document_root` → no-op insert. Re-running `make textbook-*`
or `make crawl-textbooks` on the same network state is safe at
the database layer — no duplicates, no schema drift, no
audit-event proliferation. HTTP refetch happens every run; pair
with `make recrawl-check` for a HEAD-only freshness probe.
README rewritten to make this contract explicit, with shard-
layout table + crawl-delay caveats consolidated.
Vital-books checklist
=====================
The README now carries a g4-pillar coverage table:
Pillar I Logic ✓ Levin + Aristotle Prior + Posterior
Pillar II Set Theory ✓ Levin
Pillar III Arithmetic ✓ Levin
Pillar IV Geometry ✗ Hilbert PG #17384 (PDF/TeX only)
Pillar V Probability ✗ Kolmogorov license analysis pending
Pillar VI Phys. Newton ✓ Newton Principia Motte
Pillar VII Combin. ✓ Bogart + Keller-Trotter + Levin
Pillar IX λ-Calculus ✗ Church + Turing 1936 papers pending
5 of 7 pillars now have surface coverage. The remaining gaps
are explicit license (V) or format (IV PDF/TeX, IX paper-level)
issues, each documented in the corresponding manifest entry's
notes field.
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
Pillar VII bundle (axiomsclaude-vii-v1.json +
theoremsclaude-vii-v1.json) ingested via the existing claim_pack
source into ~/.arborist/shards/000.db. 14 records (7 axioms + 7
theorems): Addition / Multiplication / Pigeonhole Principles,
Factorial + Binomial Coefficient definitions, Pascal's Rule,
Empty-Set Boundary; Binomial Theorem, Inclusion-Exclusion
(counting form), Hockey-Stick, Vandermonde, Catalan Closed
Form, Stars and Bars, Strong Pigeonhole.
Combined with the v2 bundles (78 records across pillars I-VI +
IX), shard 000 now carries 92 claim_pack documents.
Retrieval lift verified on representative combinatorics queries:
- Pascal's rule → claim-pack record at #2
- pigeonhole → Strong Pigeonhole Principle at #2
- Modus Tollens → claim-pack record at #3
Authorship metadata: Claude blackops draft + cite-check against
Stanley / Brualdi / Wilf / Knuth (option C from #000033 §2.1).
Records cap at ANCHOR-WARRANTED on the four-rung ladder until
#000031 surface-ingests the cited textbooks and computes
derivations.proof_blob — that's the warrant-promotion track.
Drive-by Makefile fix
=====================
Crawl shards now land in $(CRAWL_SHARDS_DIR) ($(HOME)/.arborist/crawl)
by default, separate from $(SHARDS_DIR) ($(HOME)/.arborist/shards).
SQLite's max-attached-databases limit is 10; mixing 4 main shards
+ 6 crawl shards + qa.db + snapshots.db put us at 12 and broke
cross-shard queries. Crawl shards moved to a sibling dir; the
existing crawl_textbooks-stats target reads from both for backward
compat with already-placed shards.
The first iteration of this layer landed scaffolding plus shallow
seed-URL ingest, which produced 300-1000 byte stubs (front matter
+ nav chrome only) instead of real book content. Fox flagged
that as the wrong outcome — we were supposed to track down the
actual texts.
This commit fixes it by adding the deep-ingest path: each
manifest entry now carries `crawl_url` + `crawl_depth` +
`crawl_max` fields, and `make crawl-textbooks` iterates them
through the existing BFS crawler (one shard per host).
Manifest changes
================
Three URL fixups uncovered by the first crawl attempt:
- PreTeXt textbooks (Bogart, Keller-Trotter, Levin) ship an
`index.html` that's a 1-line meta-refresh redirect; the BFS
crawler doesn't follow meta-refresh, so the seed URL has to
point at the actual content entry (`frontmatter.html`,
`app-comb-2.html`, `ctgd/index.html`).
- Open Data Structures uses trailing-slash directory paths
(`/ods-cpp/`, `/ods-python/`, `/ods-java/`), not `/ods-cpp.html`.
- `appliedcombinatorics.org` declares `Crawl-Delay: 20` in
robots.txt — full BFS takes ~25 minutes; documented in the
manifest entry's notes so operators know to background it.
Hilbert + Boole stay flagged as `urls: []` placeholders because
PG eBooks #17384 + #15114 ship as PDF/TeX only (no HTML format).
Their `tex_url` + `pdf_url` fields are now preserved on the
entry for the future PDF/TeX source ticket.
Make targets
============
- crawl-textbooks — BFS-crawl every entry with `crawl_url`
- crawl-textbooks-stats — docs/chunks/size summary across crawl shards
Helper script: `bench/scripts/textbooks_manifest.py crawl-targets`
emits one tab-separated `<url>\t<depth>\t<max>\t<id>` row per
entry that declares a `crawl_url`.
Real-content verification
=========================
Crawl results across 4 of 5 textbooks (Keller-Trotter still
running in background due to crawl-delay; expected to complete
in ~25 min):
crawl_bogart_openmathbooks_org.db docs=44 chunks=161 971 KB
crawl_discrete_openmathbooks_org.db docs=51 chunks=376 1929 KB
crawl_en_wikisource_org.db docs=30 chunks=110 1651 KB
crawl_opendatastructures_org.db docs=64 chunks= 84 1110 KB
TOTAL 189 docs, 731 chunks
FTS5 spot-checks confirm substantive content landed:
- Bogart "binomial coefficient" → "CTGD Partitions and Stirling
Numbers" chapter, top hit
- Levin "modus ponens" → "Propositional Logic" chapter, top hit
That's the real lift signal we were after. Out-of-scope
follow-ups remain: PDF/TeX support for PG editions (Hilbert,
Boole) and full Aristotle Owen-translation depth (currently
~75% complete on Wikisource).
A make-target scaffold for pulling public-domain and copyleft-
redistributable textbooks into a dedicated arborist shard via the
existing HtmlPageSource ingest pipeline. No new ingest machinery —
the existing path (robots.txt → noise-strip → 512-token chunk →
Merkle root → audit-event) IS the consistent process.
Manifest format
===============
bench/fixtures/textbooks/manifest-v1.jsonl — one JSON record per
textbook with explicit license tokens (PD / CC-BY / CC-BY-SA /
GFDL / OSI-approved). Fail-closed validation in
bench/scripts/textbooks_manifest.py refuses to emit URLs from
entries with missing or disallowed license tokens, so license
discipline surfaces at the URL-emit step rather than getting
silently buried in a fetch run.
Allow-list excludes CC-BY-NC (incompatible with arborist's
AGPLv3 distribution profile) and CC-BY-ND (no-derivatives
prevents chunking). Wilf's generatingfunctionology stays out
because its educational-use license forbids rehosting; citable
but not redistributable.
Initial entries (7 textbooks, 8 seed URLs)
==========================================
- Bogart, Combinatorics Through Guided Discovery — GFDL-1.3
- Keller & Trotter, Applied Combinatorics — CC-BY-SA-4.0
- Levin, Discrete Mathematics: An Open Introduction (3rd ed) —
CC-BY-SA-4.0 (4th ed switched to NC; pin 3rd)
- Hilbert, Foundations of Geometry (Townsend 1902) — PD via PG
- Boole, An Investigation of the Laws of Thought (1854) — PD
- Aristotle, Prior Analytics (Jenkinson) — PD via Wikisource
- Morin, Open Data Structures — CC-BY-2.5
Covers pillars I (logic) / II (set theory) / III (arithmetic) /
IV (geometry) / VII (combinatorics) on the claim-pack
(#000029) layout, plus a CS anchor for downstream domain
expansion. Each entry's pillar_targets field lists which
claim-pack pillars its records are candidate citations for.
Make targets
============
- textbooks-summary — license + URL counts per entry
- textbooks-urls — flat URL list to stdout
- fetch-textbooks — ingest all manifested URLs into
$(TEXTBOOK_DB) (default
~/.arborist/textbooks.db)
- textbooks-stats — documents / chunks / edges
- textbooks-verify — sample Merkle proof verification
To grow coverage of one textbook: append more chapter URLs to
its `urls` array. For deep-BFS of a textbook home: use the
existing `make crawl-ingest URL=<base> DEPTH=N` instead.
Smoke-tested end-to-end against Wikisource Prior Analytics:
1 doc / 1 chunk / 12 outbound edges / Merkle proof passes.
Shows the pipeline works; populating each book to depth needs
either more URLs in the manifest or the crawler.
Out of scope
============
- PDF processing. The Internet Archive hosts PD textbooks
(MacMahon's Combinatory Analysis 1915, Whitworth's Choice
and Chance 1867, the Motte 1729 Principia) as scanned PDFs;
a TextbookPdfSource with pdftotext / pypdf extraction is a
separate ticket — current scope is HTML-shaped sources.
- Mendelson + Enderton (proprietary; await #000031 §2.1
decision).
- Wilf generatingfunctionology (license forbids redistribution).
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
Three small streams:
#3 — close#000030 properly
============================
All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:
Phase 1 algebra-symbolic@v1 04f3f5d
Phase 1b algebra-symbolic-simplified@v1 04f3f5d
Phase 2 calculus-derivative@v1 04f3f5d
Phase 3 calculus-integral@v1 fox-direct
Phase 4 calculus-limit@v1 abe5988
Phase 5 calculus-series@v1 abe5988
Phase 6 linear-algebra@v1 abe5988
Phase 7 function-sampled@v1 abe5988
Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.
#5 — composition fixtures across new SymPy π*'s
================================================
12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:
- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
fingerprint is order-sensitive; composite domain == inner domain;
composite bytes == manual chain bytes.
Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().
#4 — end-to-end witness sweep against real shards + Hermes
===========================================================
New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.
`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.
First real sweep (this commit, against Hermes-3-8B):
agreement label count rate
KERNEL-LLM-DIVERGED 5 62.5%
KERNEL-LLM-AGREE 3 37.5%
───────────────────────────────────────────
divergence_count 5 62.5%
wall median / max 130 ms / 1.1 s
Hermes diverged on 5/8 of the canonical-shape questions:
- said `1/10` for `0.1 + 0.2` (kernel: `3/10`)
- said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already
expanded — but Hermes ALSO
emitted the unexpanded form
when given the expanded
form, vs the kernel's
deterministic expand)
- and 2 more.
These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.
Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).
Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
A new π* kernel that canonicalizes pure-integer counting
expressions and FAILS CLOSED on any input whose result isn't a
non-negative sp.Integer. Tighter domain than algebra-symbolic@v1,
which already accepts the same input surface but happily returns
symbolic / negative / non-integer outputs.
Distinguishing feature versus algebra-symbolic@v1:
algebra-symbolic@v1: binomial(n, k) → "binomial(n, k)" (symbolic
passthrough)
combinatorics@v1: binomial(n, k) → PiStarError (fail-closed
on free-symbol output)
algebra-symbolic@v1: binomial(Rational(1,2), 3) → 1/16 (rational)
combinatorics@v1: binomial(Rational(1,2), 3) → PiStarError
(output not Integer)
Boundary kept explicit: binomial(-3, 2) = 6 IS accepted because the
output is an integer 6. The fail-closed rule is on output shape
(Integer ≥ 0), not input range. Documented as
test_generalized_binomial_negative_args_accepted_when_integer.
Output format: plain decimal literal (b"10", b"5040"). Composes
with arithmetic@v1 for byte-identical agreement with the rational
route (b"10/1") so the multi-modality witness (#000028) can pin
equivalence-class agreement when both routes fire on the same
question.
Allowed surface (via SymPy primitives): binomial, factorial, ff /
rf (falling/rising), catalan, bell, partition, stirling, plus
arithmetic compositions over those primitives
(3*binomial(5,2) + factorial(4) = 54).
Coverage:
- 43 unit tests including binomial symmetry C(n,k)=C(n,n-k),
Pascal's rule C(n,k)=C(n-1,k-1)+C(n-1,k), the C(n,k) =
factorial(n)/(factorial(k)·factorial(n-k)) identity,
fail-closed paths (symbolic/negative/non-integer/relational/
parse), round-trip idempotence, composition with arithmetic@v1.
- 10 syntax + 12 semantics bench fixtures, 100% pass.
- bench/batteries/base.py PHASE_1_CARRIERS gains "combinatorics".
- Makefile bench-5s-combinatorics target.
All gate on pytest.importorskip("sympy") so a sympy-less suite
stays green. Full make test: 1537 passed / 28 skipped.
Sequencing rationale honored: this kernel lands FIRST so that
#000033 (claim-pack pillar VII for combinatorics) can bind its
records to the tighter integer kernel from day one — avoids
rebind churn on pi_star_ref fields.
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
tabular-pinned@v1 + calculus-limit@v1 + calculus-series@v1 +
linear-algebra@v1 + function-sampled@v1 — all reserved stubs
graduated; the π* registry is now 15 concrete kernels with no
remaining reserved-stub entries.
#000030 Phase 4 — calculus-limit@v1
====================================
sp.limit with thread-timeout. One-sided dir support (+/-/+-).
Pinned spelling for infinity cases: b"+oo" / b"-oo" / b"zoo"
(complex infinity) — bypasses sp.expand since Infinity isn't
algebraic. Finite results re-canonicalize through algebra-symbolic
recipe (sp.expand + sp.srepr). Unevaluated cases / timeouts emit
b"unevaluated:" + sp.srepr(<Limit>) sentinel, mirroring
calculus-integral's pattern.
#000030 Phase 5 — calculus-series@v1
=====================================
sp.series(f, x, x0, n).removeO() → sp.expand → sp.srepr. Drops
O(x**n) remainder explicitly so the canonical form is finite-byte.
Sentinel format mirrors limit/integral: b"unevaluated:Series(...)"
on timeout. n must be a positive int; 0 / float / negative rejected.
#000030 Phase 6 — linear-algebra@v1
====================================
Single π* covers the whole linear-algebra surface via {op, matrix}
JSON. Ops: rref / det / eigenvalues / inverse. Matrix cells go
through Fraction(Decimal(str(...))) for floats so 1, 1.0, "1.0"
all collapse to Rational(1, 1) — matching arithmetic@v1's
discipline. Without this fold, sp.sympify keeps floats as Float
(separate type) and downstream det/inverse return Float-shaped
bytes. Eigenvalues are sorted by srepr for determinism.
Output formats:
rref / inverse: rows/cols header + cells joined by | (rows by ||)
det: det:<num/den-or-srepr>
eigenvalues: eigenvalues:<value-1>x<mult-1>|...
#000030 Phase 7 — function-sampled@v1
======================================
Bridge to time-series-quantized@v1. SymPy expression + linspace
grid → quantized integer-vector signature in time-series's exact
output format (dt=...;dv=...;n=...;t0=0:v0|v1|...). Two functions
that render identically (within sample-grid tolerance) collapse
to the same canonical bytes. This is what plotting CAN become
in π* terms — the PNG render is a downstream view of the same
canonical evidence.
Math-only sampler (no numpy in the dep surface); Python's round()
is banker's-rounding so the bytes are interchangeable with
time-series-quantized@v1's output. Complex / non-finite samples
raise PiStarError rather than silently dropping imaginary parts.
tabular-pinned@v1 — last reserved stub graduates
=================================================
JSON-rows input ({schema, key_columns, rows}); declared
key_columns sort policy (stable sort by primary-key tuple);
type-fold per column (int/rational/bool through arithmetic@v1
discipline; str verbatim; bool normalized). Header case is
PINNED EXACT — Excel and PostgreSQL both care about case;
defaulting to lowercase-fold would break operator expectations.
Output: header (schema + key + n) + rows joined by \n + cells by |.
The π* registry has no remaining reserved stubs. Every modality
the substrate paper reserved is now real.
Test suite: 1568 passed (was 1467; +101). New closure-criterion
test (test_no_stub_pi_stars_remain) replaces the old reserved-stub
parametrize — adding a future stub re-opens this list.
110/110 fixtures pass across the 5 new bench-5s-* targets.
PHASE_1_CARRIERS gained calculus / linear-algebra / function-sampled
/ tabular.
Two new π* canonicalizers extend the math substrate above
arithmetic@v1 (closed-form rationals) and logic-kernel@v1
(propositional Boolean → CNF):
algebra-symbolic@v1 (Phase 1) — symbolic-algebra domain.
sp.expand → sp.srepr canonical bytes. Polynomial identity collapses
((x+1)**2 ≡ x**2 + 2*x + 1); exponential identity collapses
(exp(a+b) ≡ exp(a)*exp(b), inherited from sp.expand's default
behavior); trigonometric identity does NOT collapse
(sin²+cos² ≢ 1). The trig surface is reserved for a future
algebra-symbolic-simplified@v1 variant that wraps sp.simplify at
unbounded CPU cost. Rejects relationals (`x > 0`) and
BooleanFunction shapes (`x & y`) via `isinstance(expr, sp.Expr)` —
sp.Symbol confusingly inherits from Boolean so the right rejection
filter is "not Expr" rather than "Boolean".
calculus-derivative@v1 (Phase 2) — calculus domain. JSON-shaped
{f, x, n} input → sp.diff → sp.expand → srepr bytes. Output is
itself a valid algebra-symbolic@v1 input so the two compose
naturally under arborist.pi_star.compose. n defaults to 1; bools
explicitly rejected (Python isinstance(True, int) is True so we
filter that explicitly).
Optional dependency: sympy ships in the new [math] extra
(pyproject.toml). Folded into [dev] so make bootstrap pulls it
transitively. An explicit `bootstrap-math` Makefile target documents
the opt-in for minimal-install users. Both modules self-guard
via `try: import sympy as sp / except ImportError: sp = None` and
only register(...) when sympy is present, so a fresh checkout
without [math] still loads arborist.pi_star without raising.
Preflight algebra route lands in
arborist.qa.query._canonical_projection_preflight between the
arithmetic and logic routes. Charset regex (_CANONICAL_ALGEBRA_RE)
allows lowercase letters + math chars; requires at least one
letter (else arithmetic wins); rejects natural-language leading
verbs via _CANONICAL_ALGEBRA_NL_LEAD_RE (4-letter minimum so
single-/two-/three-char identifiers like x, xy, sin, cos, pi
survive while "simplify (...)", "factor x...", "expand (a+b)..."
fall through). PiStarError + KeyError both fall through cleanly
so a sympy-less install just routes everything past algebra.
Bench substrate:
- bench/batteries/base.py PHASE_1_CARRIERS gains "symbolic_algebra"
- bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl (10 fixtures)
- bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl (13 fixtures
including the documented trig non-collapse + exp collapse)
- Makefile bench-5s-algebra target → 100% pass
Tests: 18 algebra-symbolic + 38 calculus-derivative unit tests +
~10 new preflight-route tests in test_canonical_projection.py. All
gate on pytest.importorskip("sympy") so a sympy-less suite stays
green. Full suite: 1369 passed / 27 skipped.
Phases 3-7 (integral, limit, series, linear-algebra,
function-sampled) remain open as future work; each lands as its
own ticket when an actual consumer surfaces.
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.
## Lazy concept_relations loading
Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.
Refactored arborist/concepts/query.py:
- New _load_neighbors_for(shards_dir, tokens) — targeted
WHERE token IN (...) OR target IN (...) query. Returns just the
direct synonym neighborhood for the given tokens (~300 rows for
a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
expansion exceeds max_total (the cap). Most queries never reach
the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
loaders. Eager _load_indices / _get_indices kept for any
back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.
All 14 concept tests pass unchanged — the contract is preserved.
Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):
metric pre-fix post-Phase-1 post-lazy-concepts
wall_ms 14,500 13,400 9,300 (-36%)
search_ms 9,900 10,600 5,000 (-49%)
SQLite executes 10,623 3,687 3,708 ~same
synonym_expand 2,966 2,840 ~0 (lazy hit)
Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).
## Real-shard baseline artifact (Phase 2)
bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:
- bench/results/real-shard-baseline.json (durable; commit_sha,
shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)
Question set in bench/fixtures/real-shard-baseline-v1.jsonl:
virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
(rivalry path), and two canonical-projection cases (math + logic
preflight short-circuit).
First baseline run (commit e78814c plus this fan-out, BURN=1):
audit_mode n notes
STRICT 4 (virt-back, France, Mac OS X, Linux)
HYBRID 2 (Microsoft founder, AMD/Intel)
CANONICAL 2 (0.1+0.2, A IMPL B; <1 ms each)
wall median 4.1 s (range 0.6 ms – 7.2 s)
primary used 4 / 8
`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).
Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.
`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.
## Test status
Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
Closes the math half of fox's 2026-05-08 roadmap question ("did we
implement the math and logic stuff?"). Until now arborist had logic
KERNELS (Syllogism, Truthtables, Transitivity sub-batteries with
deterministic evaluators) but no math π*'s. SQD whitepaper §14
framed math as "operations over π*-canonical invariant objects:
integers, algebraic expressions, proof states, constraint graphs"
— that framing is now actually implementable.
arithmetic@v1 — exact rational arithmetic (SQD §14.1)
-----------------------------------------------------
Parse arithmetic expression → fractions.Fraction → canonical
"<num>/<den>" in lowest terms. Decimals via Decimal(str(...)) for
exact rational interpretation:
"0.1" → "1/10" (not 0.1±ε)
"0.1+0.2" → "3/10" (the SQD-canonical floating-point question)
"1+2" → "3/1"
"(1+2)*3" → "9/1"
"6/4" → "3/2" (lowest terms via Fraction invariant)
"2**3" → "8/1"
"-(1+2)" → "-3/1"
Rejects: identifiers, function calls, division by zero, non-integer
exponents, boolean literals. PiStarError with explanatory message.
logic-kernel@v1 — propositional CNF canonicalizer (SQD §14.3)
-------------------------------------------------------------
Parse Boolean expression (AND/OR/NOT/XOR/IMPL/IFF + named atoms +
parens) → eliminate IMPL/IFF/XOR → push NOT inward (NNF) →
distribute OR over AND (CNF) → dedupe + sort literals + sort
clauses + drop tautological clauses → serialize.
Equivalence classes preserved:
A AND B ≡ B AND A (commutativity)
(A AND B) AND C ≡ A AND (B AND C) (associativity)
A IMPL B ≡ NOT A OR B (IMPL rewrite)
A IMPL B ≡ NOT B IMPL NOT A (contrapositive)
NOT (A AND B) ≡ NOT A OR NOT B (De Morgan)
NOT NOT A ≡ A (double negation)
A OR (B AND C) ≡ (A OR B) AND (A OR C) (distribution)
A OR NOT A → TRUE (tautology)
A AND A ≡ A (idempotence)
Atom cap: N=8 (256 max clauses). Larger inputs raise PiStarError;
CNF blow-up is exponential in atom count, the cap keeps
canonicalization deterministic in bounded time.
Surface
-------
- arborist/pi_star/arithmetic.py — full implementation (new file)
- arborist/pi_star/logic.py — full implementation (graduates from
stub; preserves the LogicKernelV1 dataclass shape so the registry
key arithmetic@v1 / logic-kernel@v1 stays stable)
- arborist/pi_star/__init__.py — imports arithmetic to auto-register
- bench/batteries/base.py — PHASE_1_CARRIERS adds "arithmetic" + "logic"
- bench/fixtures/5s/syntax-arithmetic-v1.jsonl (12 fixtures)
- bench/fixtures/5s/semantics-arithmetic-v1.jsonl (15 fixtures)
- bench/fixtures/5s/syntax-logic-v1.jsonl (12 fixtures)
- bench/fixtures/5s/semantics-logic-v1.jsonl (18 fixtures)
- Makefile: bench-5s-arithmetic, bench-5s-logic-kernel,
bench-5s-math aggregate
- tests/test_pi_star.py: 30 new tests
- 13 for arithmetic@v1 (decimals, equivalence, rejects, idempotency,
integer exponents, negative results, the SQD 0.1+0.2 case)
- 17 for logic-kernel@v1 (commutativity, associativity, all rewrite
rules, De Morgan, double-neg, distribution, tautology, idempotence,
contrapositive, atom cap, syntax errors, idempotency)
Cross-modality discipline now spans:
text + claim_lattice + memory + code + arithmetic + logic
— six carrier domains, all with real π* implementations.
Two stubs remain (time-series-quantized@v1, tabular-pinned@v1);
neither is needed for math/logic coverage.
Full suite: 1256 passed, 36 skipped (+30 tests, +57 fixtures).
Completes the bridge: every 5F sub-battery now has an embedded
(Phase 1a) AND a live (Phase 1b.2) path. Per-task detail.source
distinguishes synthetic from live signal.
| Sub-battery | Live surface | Live fixtures |
|---|---|---|
| Formulate | qa.parse_claims.parse_pointer_claims | 15 (prior) |
| Feedback Loop | store.append_audit + memory.snapshot (temp shard) | 12 (prior) |
| Function | parse_pointer_claims + shape evaluators | 12 |
| Finetuning | selfmodel.store_snapshot + claims_for round-trip | 10 |
| Falsification | qa.verify.verify_quotes (real verifier) | 12 |
Function live: input_text routes through real parse_pointer_claims;
the same Phase-1a shape evaluators (shape_match / pointer_set_match
/ threshold_on_metric) run on the live-parsed output. Tests
parser→shape pipeline against actual organism behavior.
Finetuning live: gated via fixture's "live": true flag. Runner
writes parent + child SelfModel + capability_claim to a fresh temp
shard via real arborist.selfmodel.store_snapshot, reads back via
claims_for, runs the same improvement check. Tests the SelfModel
persistence + claim-attach surface, not just static fixture data.
Falsification live: answer_text + context routes through real
arborist.qa.verify.verify_quotes. Live signals translate the
verifier's audit_mode + verifier_method + unverified_quotes into
flat tags (UNGROUNDED, STRICT_<method>, HYBRID_<method>,
UNVERIFIED_QUOTE) the runner matches against expected_reason. One
fixture (5f-fal-live-003) deliberately documents a known soft-
signal gap — the entity strategy treats Insulin/Penicillin claims
as equivalent because both share "Alexander Fleming". Production
catches it via title-relevance + claim-lattice verifier; the
fixture pins the soft-path limit so future verifier changes
re-trigger review.
Surface:
- bench/batteries/b_5f.py — _live_function_produced,
_live_finetuning_measure, _live_falsification_violations
helpers + two-mode dispatch in run_function / run_finetuning /
run_falsification.
- bench/fixtures/5f/{function,finetuning,falsification}-live-v1.jsonl
(12 + 10 + 12 fixtures).
- Makefile: bench-5f-{function,finetuning,falsification}-live
targets; bench-5f-live aggregate now covers all five sub-batteries.
- Tests: 9 new in tests/test_bench_batteries.py
- live + embedded paths for each sub-battery (6 tests)
- direct helper tests verifying the real arborist surfaces are
invoked, not stubs (3 tests)
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at synthetic Phase-1a fixtures so `runner --all` behavior is
identical and deterministic; live mode invoked via explicit
--fixtures path or make targets.
Full suite: 1210 passed, 36 skipped.
5F battery is now the first to bridge synthetic→live across every
sub-battery. The pattern + live fixture format are reusable for
5R Phase 1b.2 (when SelfModel-backed workspace ops want live
verification).
Second live wire-up (after Formulate). run_feedback_loop now supports
both modes:
- Embedded (Phase 1a): chain of (operation, observation) string pairs;
runner aggregates observations and string-matches expected_delta.
- Live (Phase 1b.2): live_chain of typed ops applied to a fresh temp
arborist shard via real append_audit + memory.snapshot +
selfmodel.snapshot. expected_delta is a dict of predicates against
the resulting audit_events / memory_branch_summaries.
The live helper _live_feedback_chain creates a tempfile-backed shard,
runs the chain through real arborist surfaces, and queries the final
state. Every audit event chains via the production append_audit, so
the audit chain is re-verifiable after live execution (see new test
test_5f_live_feedback_chain_audit_chain_intact).
Three predicate types in expected_delta:
- audit_event_type_present: named event_type appears in audit chain
- memory_branch_present: named branch_id in memory_branch_summaries
- body_substring_present: substring appears in any audit body JSON
Surface:
- bench/batteries/b_5f.py — _live_feedback_chain helper +
_live_delta_satisfied predicate checker; two-mode dispatch in
run_feedback_loop
- bench/fixtures/5f/feedback-loop-live-v1.jsonl — 12 live fixtures
exercising providence_write, providence_repair, memory_snapshot,
selfmodel_snapshot ops. Includes 2 negative fixtures testing the
predicate checker (expected_delta absent → expected:fail).
- Makefile: bench-5f-feedback-loop-live + bench-5f-live aggregate
for all 5F Phase-1b.2 live wire-ups.
- Tests: 5 new in tests/test_bench_batteries.py
- live path runs all 12 fixtures
- embedded path still works (10 Phase-1a fixtures)
- helper directly tests audit-event write
- rejects unknown live op type
- audit chain re-verifies after live ops
Two of the five 5F sub-batteries now bridge synthetic → live
(Formulate + Feedback Loop). Function/Finetuning/Falsification
follow in subsequent commits.
Full suite: 1201 passed, 36 skipped.
First sub-battery to bridge from synthetic gold output to actual
organism behavior. run_formulate now supports two fixture modes
selected per-task:
- Embedded (Phase 1a): produced_lattice in the fixture. 10 seed
fixtures continue to pass via this path.
- Live (Phase 1b.2): only input_text in the fixture; runner calls
arborist.qa.parse_claims.parse_pointer_claims(input_text) and
matches the live output against expected_lattice.
Embedded takes precedence if both fields are present. Per-task
detail.source ("embedded" | "live") surfaces in bench output so
synthetic vs live signal is distinguishable.
Surface:
- bench/batteries/b_5f.py — _live_produced_lattice helper +
two-mode dispatch in run_formulate
- bench/fixtures/5f/formulate-live-v1.jsonl — 15 live-mode fixtures
with input_text + expected_lattice (no produced_lattice)
- Makefile: bench-5f-formulate-live target
- Tests: 4 new in tests/test_bench_batteries.py
- live path routes through real parser, all 15 pass
- embedded path still works (10 Phase-1a fixtures)
- _live_produced_lattice helper directly verifies parse output
- fixture missing both fields fails cleanly with explanatory reason
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at formulate-v1.jsonl so `runner --all` behavior is identical;
live-mode fixtures invoked via explicit --fixtures path.
Full suite: 1196 passed, 36 skipped.
Pattern set. Function/Finetuning/Falsification/Feedback Loop
follow in subsequent commits.
Phase 2 of #000021. React/Rearrange/Restore/Replicate/Resonate over
the workspace surface — selfmodel_records (#000014) + memory_records
(#000017), both landed earlier today. Closes the gap that gated 5R
since the substrate work shipped.
Sub-battery semantics (per SQD whitepaper §9.3 + ticket #000021 §4.2):
- React: incorporate new fact/constraint. Workspace = (snapshot_t0,
snapshot_t1, expected_delta). Pass = added_facts present + removed_facts
absent in t+1.
- Rearrange: restructure without semantic shift. Re-canonicalize
different surface forms through a named π*; pass = bytes match
expected_equivalent flag. Tests the order-invariance contracts in
SelfModel (capability_claim_hashes sorted) and Memory (branches
sorted by branch_id).
- Restore: retrieve prior fact. Workspace = (history[], current_facts[]).
Pass = fact in current OR any historical snapshot.
- Replicate: independent canonical encodings via π*. Same input run
N times must yield byte-equal output. Tests determinism contract.
- Resonate: variance across N runs. Deterministic π*'s yield
distinct=1; expected_max_distinct=1 enforces zero-variance contract.
Surface:
- bench/batteries/b_5r.py (5 deterministic runners; no LLM-as-judge)
- bench/fixtures/5r/{react,rearrange,restore,replicate,resonate}-v1.jsonl
(30 each = 150 new fixtures)
- runner.py registers 5r in _BATTERIES + _DEFAULT_FIXTURES
- Makefile: bench-5r + bench-suite (5S+5T+5F+5R aggregate)
Final tally:
5S syntax/semantics/syllogism/synthesis/semiotics 108
5T transfer/transfer-learning/triangulation/... 154
5F function/finetuning/falsification/... 50
5R react/rearrange/restore/replicate/resonate 150
TOTAL: 462 fixtures across 21 sub-batteries — 100% pass.
Tests: 6 new in tests/test_bench_batteries.py + adjustment to
test_session_integration.py for the 312→462 count + 5R sub-battery
presence assertion. Full suite: 1192 passed, 36 skipped.
Closes#000021. Phase 3 (external-corpus expansion) remains open
under the ticket but does not gate closure — the complete
Dav1DPrometheus surface is now executable infrastructure.
Per fox's "partial punt on larger ones" — ships the bench/ skeleton +
small seed fixture sets so future v8/v7-W/SelfModel work can cite a
real fitness target. Full Phase 1 (50-200 fixtures per sub-battery)
and Phases 2-3 stay open in the ticket.
Phase 1a delivers:
- bench/batteries/{base,b_5s,b_5t,runner}.py — Battery protocol,
BatteryResult, fixture-digest helpers, CLI runner.
- Seed fixtures:
- bench/fixtures/5s/syntax-v1.jsonl — 10 tasks against
wikitext-base@v1 and claim-lattice@v1
- bench/fixtures/5s/semantics-v1.jsonl — 8 equivalence tasks
- bench/fixtures/5t/transfer-v1.jsonl — 4 paraphrase-invariance
tasks
- Runners for 5S Syntax, 5S Semantics, 5T Transfer. Other 5S/5T
sub-batteries are stubs returning zero-task results.
- Makefile targets: bench-5s, bench-5t, bench-5s5t.
- runtime_digest field captures the active π* registry fingerprint
so a registry change surfaces in bench results.
Tests: tests/test_bench_batteries.py (17 cases). Full suite:
1076 passed, 36 skipped. `make bench-5s5t` runs end-to-end and
emits JSON results.
Ticket #000021 status: in progress · Phase 1a landed; Phase 1b/2/3
remain open.
Makefile cuts (64 → 58 documented targets):
- ingest-cur-parallel, ingest-old-parallel: parallel-shared mode
superseded by attached (no WAL contention)
- distill-shards: sequential never preferred over parallel variant
- bench-qa-quick: bench-qa-smoke covers same use case (~30s vs ~10s)
- ingest-grok, ingest-grok-media: single-DB grok rare; -attached is
canonical path
All cuts land in code that the underlying CLI still exposes — operators
who need the dropped variant call '.venv/bin/aborist ingest --shard ...'
directly. No behavior loss, just shortcut removal.
Docs improvements:
- New Concepts page (docs/_source/concepts.rst): orientation on what
aborist is, three layers (surface/core/providence), Merkle commitment,
8-dim cache key, audit chain, trichotomy + four-rung ladder, layered
verifier, falsification state, sidecars. Embeds module-graph and
verifier-ladder SVG diagrams.
- New Cookbook page (docs/_source/cookbook.rst): 8 recipes — recrawl,
falsify, ingest-self-providence, mixed-corpus query, LLM endpoint
override, integrity after bulk ops, bench, retrieval tuning.
- Quickstart embeds query-pipeline SVG diagram.
- docs/_source/diagrams symlinks to docs/diagrams so Sphinx can include
the SVGs (was orphaned, only README referenced them).
Better Makefile RTD page (docs/_source/_ext/makefile_targets.py):
- Group by workflow phase (Setup → Fetch → Ingest → Distill → Query →
Verify → Operations → Tests → Docs → Clean) instead of alphabetical
prefix. Tells a new operator the order they'd actually run things.
- Phase descriptions added; targets prefixed with 'make ' for copy-paste.
- Uncategorized leftover surfaces missing entries in PHASES list.
Implements Read the Docs infrastructure to generate API documentation
directly from code docstrings. Replaces static modules.md (1200+ lines).
New structure:
- docs/_source/conf.py — Sphinx configuration (furo theme)
- docs/_source/index.rst — Main TOC
- docs/_source/api/*.rst — Module groups (substrate, storage, retrieval,
qa, distill, mesh, cli)
- docs/_source/Makefile — Local build targets
- docs/_source/README.md — Documentation on building and extending
Makefile integration:
- make docs-api — generate HTML (output: docs/_source/_build/html/)
- make docs-api-clean — remove build artifacts
Build output (40 HTML files):
- API module reference with docstrings
- Source code links (:viewcode: extension)
- Full-text search index
- Module index (genindex, py-modindex)
Sphinx installed in venv as dev dependency. HTML is browseable at
docs/_source/_build/html/index.html (open in browser after build).
This justifies the deletion of modules.md: code docstrings + Sphinx
autodoc = automatically-generated, always-current API reference.
Closes#000008 with status `closed · landed in 4f2b5a6` per the
docs/TICKETS.md convention. The preflight guard mechanism + bench
cycle + default flip all shipped 2026-05-03; the design log stays
in place.
Opens #000009 — Quantifier preflight run-DAG node binding. Splits
the Phase 5 follow-up out of #000008 §11.11 into its own ticket.
Scope: bind the classifier output + policy decision into
`run_dag_root` so audit replay can distinguish guard-on vs guard-off,
cap-applied vs not, reminder-injected vs skipped. Currently those
appear on the result dict but are NOT in the run-DAG hash. Required
to close the audit-replay gap that blocks the §9.5 Merkle-AGI-DAG
framing from fully holding. Estimated 3-4h. Awaiting go/no-go.
Makefile shortcuts for the #000008 CLI flags (operator ergonomics):
BROAD=1 → --apply-quantifier-caps (flip cap apply-gate)
REJECT_BROAD=1 → --reject-broad (preflight rejection)
ALLOW_BROAD=1 → --allow-broad (emergent search)
Available on both `make query` and `make query-dry`. Default
behavior unchanged: ANSWER_MODE=claim_lattice (JSON), reminder ON
for lattice modes, cap operator-opt-in.
Smoke-tested:
$ make query-dry Q="winners of all major sports?" BROAD=1
→ cap applies on JSON; classifier reports ALL/unbounded
$ make query-dry Q="winners of all major sports?" REJECT_BROAD=1
→ preflight rejection, exit-1 (consistent with UNGROUNDED)
TICKETS.md index:
#000008 closed · landed in `4f2b5a6`
#000009 open · awaiting go/no-go (D3, D4)
Next ID bumped 000009 → 000010
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.
UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.
Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.
Touched:
aborist/qa/dag.py:_sha256_hex (the loud one)
aborist/qa/keys.py:_sha256
aborist/qa/evidence.py:_sha256_hex
aborist/store.py: chain_audit_events + append_audit
Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).
Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.
Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.
Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.
pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.
Bench-max scoreboard (predicted lift from this commit alone):
+1pp lattice modes (UTF fix)
+cycle-time enabler (smoke fixture, xdist)
no calibration cost — none of the verifier checks moved.
