Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification
controller) is doc-only and gates Phase 1 on §12 measured-pressure
triggers. This commit lands the empirical-evidence harness fox needs
for the go/no-go decision, plus the §16.2 test scaffolding so the
contract is discoverable from the test runner today.
bench/prometheus_sigma_trigger_probe.py
=======================================
Read-only walk of audit_events + capital_ledger across shards.
Reports each §12 trigger:
- Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a
fork_score branch-set table; reports "no data — single-validator
ForkScore Phase 1a" when absent. Trigger structurally cannot fire
until #000012 multi-branch persistence lands.
- Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10):
aggregates providence_canonical_witness audit-event bodies,
maps each agreement_label per #000028 §1.2 to a binary
LLM-divergence score, computes mean/stddev/ratio. Defensive on
the max(mean,ε) guard from §12 itself.
- Trigger 3 (witness cost share > 0.30): aggregates capital_ledger
rows; uses `material` (kWh proxy) as the canonical compute axis;
surfaces both `material` and `financial` so fox can pick a
different form if needed. Tags the report with the caveat that
the current ledger reflects ad-hoc activity, not a controlled
#000026 sweep.
- Trigger 4 (operator mission need): n/a — operator decision.
Pure measurement, no LLM, no schema change, no mutation. Output is
a markdown report at $(PROMETHEUS_PROBE_OUT) (default
bench/results/prometheus-sigma-triggers-<utc-date>.md).
Wired via `make prometheus-trigger-probe`.
bench/results/prometheus-sigma-triggers-2026-05-10.md
=====================================================
First captured baseline. Verdict on current shards:
Trigger 1: NO (no fork_score branch-set table; #000012 Phase 1a
is single-validator)
Trigger 2: NO (16 samples; N_min=30. But mean=0.625, σ=0.5,
ratio=0.8 — both ratio AND abs floors would fire
if N reaches 30. Signal is there; just needs more
samples.)
Trigger 3: NO (witness/total material = 0.005, well under 0.30
threshold. Caveated: ledger is ad-hoc, not a
controlled #000026 sweep.)
Trigger 4: n/a (operator-stated)
Empirical answer: no §12 measured-pressure trigger has fired yet.
Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4.
tests/test_prometheus_sigma.py
==============================
17 skip-stubs pinning the §16.2 acceptance contract. Each test:
- Collects today via `pytest --collect-only` so the test surface is
discoverable.
- Skips with reason "Phase 1 not landed; controller module
arborist/v9/prometheus.py absent" until that import succeeds.
- Carries a one-sentence intent line tying it back to a numbered
ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4).
When Phase 1 lands, the implementer adds the controller module
under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE
becomes True; each test gets its body filled in. The names, intents,
and skip-reason strings are the durable contract.
Hygiene
=======
- make test → 1623 passed, 45 skipped (was 28; +17 new skips).
- make chain-check-shards → 0 breaks across all 7 shards.
- No code touched outside the new probe + scaffolding files +
Makefile target wiring. fox's in-flight #000037 ticket
modifications left untouched.
Three orthogonal lifts under fox 2026-05-10 "all of the above"
directive:
(1) 7 medium-confidence citation aliases per #000041:
Mendelson -> Russell IMP + De Morgan FNL + Boole Laws of Thought
Enderton -> Russell IMP + De Morgan FNL
Jech -> Cantor 1915 Contributions
Landau -> Russell IMP
Each row carries partial-coverage caveat in decision_rationale.
(2) Peano Arithmetices Principia 1889 ingested via existing
textbook_tex pipeline. Source: mdnahas/Peano_Book GitHub
repo (CC-BY-SA-4.0, Verheyen+Nahas English+Latin LaTeX).
paracol multi-column macros leak some residue but substantive
text extracts (39 chunks, 35 hits on "peano", 38 on
"arithmetic"). Make target textbook-peano. Manifest entry
peano-arithmetices-principia-1889. Plus 3 substitute-citation
aliases:
Landau -> Peano (lifts 2 pillar III records)
Goedel -> Peano (2 variant strings)
(3) Title-from-author backfill on Cantor / Russell IMP / Bogart /
Judson / Peano shards. Empirical defect: HTML/textbook_tex
sources took titles from <title> tags or URLs which on
Wikisource + PG do NOT include author surname, so the
resolver's _shard_matches_citation heuristic (requires
cited-author surname in title haystack) failed silently for
all alias-resolved chains where the substitute author wasn't
self-evident in the shard content. Surgical SQL UPDATE on
documents.title fixed it. Single biggest lift in this turn:
+13 records.
Per-pillar:
I: 4/13 -> 7/13 (+3, Russell IMP / Boole / De Morgan)
II: 0/10 -> 5/10 (+5, Cantor)
III: 0/13 -> 7/13 (+7, Peano + Russell IMP)
IV: 18/18 stable
V: 0/5 stable (Kolmogorov URAA-blocked)
VI: 3/5 stable
VII: 9/14 stable
IX: 5/14 stable
Total: 54/92 (59%)
Title backfill is a one-off SQL; the proper long-term fix is in
the source ingest pipelines (HTML / textbook_tex should pull
author from manifest's `author` field at ingest time and prepend
to document.title). Tracked under #000031 Phase 1 follow-up.
Three high-confidence alias decisions per fox 2026-05-10 + 14
full-string aliases for compound pillar-VII citations + a deeper
Newton recrawl.
Lift breakdown:
pillar IV (Hilbert): 18/18 unchanged
pillar VI (Goldstein/Newton): 0/5 -> 3/5 (+3)
pillar VII (Stanley/Bogart): 0/14 -> 9/14 (+9)
pillar IX (Dummit-Foote/Judson):0/14 -> 5/14 (+5)
Citation aliases live in ~/.arborist/shards/000.db (audit table per
#000041). 14 pillar-VII rows are full-string matches against the
compound source_ref strings (resolver looks up by exact source_ref;
parsed sub-citations are not yet alias-keyed individually).
Newton shard rebuilt: original `crawl_en_wikisource_org_newton.db`
followed Wikisource sidebar instead of book content (60 docs / 289
chunks of mostly nav cruft). Manifest now pins (1729) edition root
+ explicit subpage urls (Axioms_or_Laws_of_Motion, Definitions,
Rules_of_Reasoning_in_Philosophy); recrawl produced 120 docs / 779
chunks of actual Principia text in textbook_newton-principia-motte.db.
5 records stuck despite alias registered (FTS5 cascade misses):
Newton Third Law + Conservation of Momentum (modern vocabulary
absent from 1729 prose), 4 advanced pillar-VII identities (Hockey-
Stick, Vandermonde, Catalan, Stars-and-Bars — Bogart skews intro-
level), Orbit-Stabilizer (Judson covers it but cascade doesn't
surface). Tractable one-by-one; deferred.
Pillars I/II/III/V/lambda subset of IX still 0/X — those need
medium-confidence alias decisions (Mendelson/Enderton -> Russell IMP,
Jech -> Cantor) or Phase 4 textbook acquisitions (Landau, Goedel,
Kolmogorov, Church) under #000038.
The bench-qa-smoke A/B against commit 416f956 surfaced a methodology
gap: the smoke fixture is structurally insensitive to retrieval-side
changes — every smoke query succeeds at full-AND chain 0 on every
shard, so cache_keys match byte-for-byte BEFORE/AFTER and any
STRICT-rate variance is pure LLM dice. Discovered when 5pp signal
floor at n=3 produced ±20pp swings with identical cache_keys.
This commit pins a fixture that ACTUALLY exercises the new code path:
- 5 PROGRESSIVE queries (full-AND fails on at least one shard;
_progressive_and_token_chains drops shortest-tokens-first). Includes
the canonical Gundremmingen case from commit 2b9d1f0.
- 2 DF_FILTER queries (every AND chain fails on at least one shard;
search lands in OR-mode where _filter_or_pool_by_df trims
high-DF tokens).
- 2 CONTROLs (full-AND succeeds on every shard; cache_keys must
remain identical BEFORE/AFTER for any retrieval-side change).
Which-chain-fires-on-which-shard was empirically verified at HEAD
against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header
documents the chain that wins per query; re-probe if shards drift.
Baseline A/B (commit 416f956 ON vs OFF, n=3, --burn between samples,
fixture run via make bench-qa-progressive-and):
category cache_key parity STRICT-rate delta median latency delta
PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s
DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s
CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s
The CONTROL +0.0s latency delta is the load-bearing diagnostic —
when retrieval is byte-identical the search-side change cannot
affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key
drift is the change actively redirecting retrieval, with the
expected latency win and no STRICT regression.
Future retrieval-side work (#000039 sqlite-vec backend, synonym
expansion changes, embedding rerank, hybrid scoring) should bench
against this fixture in addition to the full bench-qa scoreboard.
Hygiene:
- make test → 1623 passed, 28 skipped
- make chain-check-shards → 0 breaks across all 7 shards
Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of
Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy
(PG #41654, PD), and Judson Abstract Algebra Theory and Applications
(GFDL-1.3) to the textbook manifest, plus per-book make targets +
a `textbooks-base-knowledge` bulk target. Idempotent at DB layer.
These four primaries cover the proprietary citations across pillars
I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote)
that dominate the 74 unresolved claim-pack records. They land as
honest "no match" under the existing resolver `_shard_matches_citation`
heuristic (cited-author surname required) — full lift gates on
fox citation-alias decisions per #000041.
Total ingest delta: 43 docs / 295 chunks across four crawl shards.
Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
records per-book license rationale, projected lift table, and the
remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau,
Gödel, Kolmogorov, Church) for which no clean PD HTML edition was
located in this session.
Smoke-test alias rationale was published with #000042, but two of the
three trailing records ("non triviality", "side-angle-side") needed
their own per-concept aliases since Hilbert 1902's vocabulary differs
in more than one place. Map of registered aliases (geometry domain):
incidence -> connection (4 records lifted)
euclidean parallel postulate -> axiom of parallels (1)
non triviality -> space axiom (1)
side angle side -> included angle (1)
Hilbert pillar IV is now the first complete warrant-chain vertical:
18/18 records bind to a primary-source surface chunk via Merkle
inclusion proof. 11 direct + 7 via-alias derivations rows. Bench
journal under bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md
records reproducibility steps + per-alias signal data so a future
shift can extend the same pattern to Boole 1854, Newton 1729 Motte,
and Aristotle's traditional translations once #000038 lands those
surfaces.
Cluster-wide remains 18/92 (20%); the other 74 records cite textbooks
not yet ingested. Phase 4 content acquisition (#000038) gates next
lift.
Six-question Q&A sweep against the augmented shard cluster
post-#000031 Phase 3 commit (69e0a95). Looking for the new
`· warrant: N proven` tail surfaced by
_render_warrant_chain_tail when an answer's cited sources
include a claim-pack record with a warrant-resolver-v1
derivations row.
Result: 1 of 6 questions lit up the warrant tail.
Pasch's axiom → STRICT · via quote · warrant: 1 proven ✓
Hilbert geometry axioms → STRICT · via quote ✗
axiom of betweenness → HYBRID · via paraphrase ✗
Bayes theorem → STRICT · via quote ✗
modus tollens → UNGROUNDED · via quote ✗
binomial coefficient id. → HYBRID · via paraphrase ✗
1/6 matches the data substrate: 11 of 92 records have warrant
chains, all pillar-IV Hilbert axioms. The 5 unsigned cases are
honest:
- Hilbert / betweenness queries cited Wikipedia + crawled
textbook surfaces, NOT the claim-pack records — no claim-pack
record cited → no warrant chain to surface.
- Bayes theorem: even if cited, Kolmogorov isn't surface-
ingested → would stay at ANCHOR-WARRANTED ceiling.
- modus tollens UNGROUNDED: pre-warrant verifier issue,
orthogonal to this ticket.
- binomial-coefficient: claim-pack pillar VII cites Brualdi /
Stanley / Knuth, none surface-ingested.
No false positives observed — the render tail correctly stays
silent when chain-of-custody data is absent.
Roadmap: Adding any cited textbook to the manifest →
re-running warrant-resolve → more records resolve, more queries
light up the tail. The 74 still-unresolved records are a
prioritized acquisition list (Mendelson ~20, Enderton ~10,
Stanley ~5, Brualdi ~3, Knuth ~3). The substrate writes its
own roadmap.
Three streams. Two land in the repo; one lands in fox's Downloads
(existing g4 packs live there too).
Witness sweep automation
========================
`bench/scripts/witness_sweep_cron.sh` — schedulable harness that
runs `make bench-witness-sweep` + `make bench-witness-divergence`
back-to-back and (with --commit) auto-commits any new divergence
fixtures + sweep result. Fail-closed: pre-commit hook failure
aborts; non-main-branch refuses to commit.
$ bench/scripts/witness_sweep_cron.sh # dry run
$ bench/scripts/witness_sweep_cron.sh --commit # auto-commit
$ bench/scripts/witness_sweep_cron.sh --commit --push # full pipeline
Recommended cron (every 6 hours; LLM cost scales with frequency):
0 */6 * * * \
bench/scripts/witness_sweep_cron.sh --commit --branch main \
>> /var/log/arborist-witness.log 2>&1
Smoke this commit: dry-run produced 5 fresh divergences (10
cumulative across two extraction passes). The witness-sweep + 5F
extractor pipeline runs unattended now; calibration-data corpus
will grow over time.
#000018 — adversarial soft-hash covert-channel analysis
========================================================
`docs/soft-hash-channel-analysis.md` — formal threat model + channel-
capacity definition + cryptographic reduction + mitigation table
+ residual analysis. Doc-only deliverable per the ticket's hard
constraint.
Headline result: under threat models T1 (data poisoner) and T2
(gradient adversary), the channel is bounded by SHA-256 partial-
preimage hardness when φ (the anchor map) is independent of
parameter geometry. T3 (hyperparameter adversary) requires
M2 (per-checkpoint anchor nonce) for boundedness.
Recommended mitigation: M2 (per-checkpoint nonce) — 32 bytes added
to the snapshot manifest, no training-ergonomics regression,
closes T1+T2 channels under any φ choice and bounds T3 by
per-checkpoint window. Fall back to M1 (PRG-based φ) if M2-only
deployment surfaces structural concerns. M3 (drop anchor entirely)
stays in reserve as the strict-construction fallback.
Three open questions (§9): Hessian alignment under φ_linear,
PRG choice for φ_PRG, and explicit T3 per-window bound. Each is a
follow-up ticket.
Ticket #000018 status: closed · landed 2026-05-09 (analysis doc).
v7 § 9.10 spec amendment proposed in §7 of the analysis.
#000033 — pillar VII (combinatorics), Claude-authored
======================================================
NOT committed to the arborist repo (the existing g4-v2 packs live
in `/home/fox/Downloads/` too — that's the operator's bundle
location). Two new bundle files at:
/home/fox/Downloads/axiomsclaude-vii-v1.json (7 axioms)
/home/fox/Downloads/theoremsclaude-vii-v1.json (7 theorems)
Pillar VII covers combinatorial counting — the gap between Grok's
pillars VI and IX in the v2 packs:
axioms (7): addition principle · multiplication principle ·
pigeonhole principle · factorial definition ·
binomial coefficient definition · Pascal's rule ·
empty-set / boundary axiom
theorems (7): binomial theorem · inclusion-exclusion (counting
form) · hockey-stick identity · Vandermonde's
identity · Catalan number closed form · stars-and-
bars · strong pigeonhole
Each record in the dual-thread format the existing g4 packs use
(Δ symbolic LaTeX + ∇ verbose prose + ∇ concise + sigil + formal
language + role + status + source_reference + date + foundational
group + category + subfield). Per fox's directive: explicit
authorship metadata everywhere — `authored_by: Claude (Anthropic)
— model claude-opus-4-7`. NOT Grok-generated; no silent invention.
Each record carries `pi_star_ref: combinatorics@v1` so the kernel
binding is explicit. Theorems list `depends_on_axioms` arrays so
each theorem cites the foundation axioms it bottoms out on.
Smoke test (committed alongside):
$ arborist --db /tmp/test.db ingest --source claim_pack \\
--bundle /home/fox/Downloads/axiomsclaude-vii-v1.json \\
--bundle /home/fox/Downloads/theoremsclaude-vii-v1.json
→ 14 docs, 14 chunks, 0 cross-bundle edges
Source attributions: Stanley EC1, Brualdi Introductory
Combinatorics, Knuth TAOCP Vol 1, plus historical sources where
applicable (Pascal 1654, Vandermonde 1772, Dirichlet 1834, Catalan
1838, Feller 1950 for stars-and-bars).
Tests: 1636 passed, 37 skipped (no regressions; pillar VII
ingestion smoke covered above).
Apples-to-apples FTS5 search comparison on a single shard
(~/.arborist/shards/000.db, ~867 K Wikipedia surface docs)
before vs after ingesting the claim-pack source (#000029)
bundles.
Method: copy 000.db twice; ingest axiomsg4-v2.json +
theoremsg4-v2.json into one copy; run identical search queries
against both. FTS5-only path so retrieval-layer behavior is
isolated from QA pipeline reranking.
Results: 3/4 representative queries show claim-pack record in
top-3.
modus tollens → claim-pack at #3 (BM25 31.77)
law of excluded middle → claim-pack at #3 (related axiom
in same pillar)
associativity of addition → claim-pack at #1 — displaces
Wikipedia's general "Addition"
article entirely
Bayes theorem → no top-3 lift; Wikipedia "Bayes"
+ "Bayes rule" articles dominate
via short-doc BM25 bias
Storage tax: sub-MB. 78 chunks against 6.2M existing chunks is
below filesystem allocation granularity. Claim-pack rides
essentially for free on the existing Wikipedia substrate.
Honest verdict: claim-pack ingest earns its storage tax for
narrow technical queries where pre-distilled CORE-shape content
has title-token advantage. Doesn't help on queries Wikipedia
already covers with focused articles. The Bayes theorem case
likely shifts under the full QA pipeline's body-coverage sqrt
rerank (which counters BM25 short-doc bias) — that's a separate
larger bench worth running once the retrieval layer's other
stages are exercised.
Four-rung audit ladder still caps at ANCHOR-WARRANTED per
#000029's honest ceiling. Surface-ingest of cited textbooks
(#000031) is the path toward EVIDENCE-WARRANTED.
Three small streams:
#3 — close#000030 properly
============================
All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:
Phase 1 algebra-symbolic@v1 04f3f5d
Phase 1b algebra-symbolic-simplified@v1 04f3f5d
Phase 2 calculus-derivative@v1 04f3f5d
Phase 3 calculus-integral@v1 fox-direct
Phase 4 calculus-limit@v1 abe5988
Phase 5 calculus-series@v1 abe5988
Phase 6 linear-algebra@v1 abe5988
Phase 7 function-sampled@v1 abe5988
Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.
#5 — composition fixtures across new SymPy π*'s
================================================
12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:
- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
fingerprint is order-sensitive; composite domain == inner domain;
composite bytes == manual chain bytes.
Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().
#4 — end-to-end witness sweep against real shards + Hermes
===========================================================
New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.
`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.
First real sweep (this commit, against Hermes-3-8B):
agreement label count rate
KERNEL-LLM-DIVERGED 5 62.5%
KERNEL-LLM-AGREE 3 37.5%
───────────────────────────────────────────
divergence_count 5 62.5%
wall median / max 130 ms / 1.1 s
Hermes diverged on 5/8 of the canonical-shape questions:
- said `1/10` for `0.1 + 0.2` (kernel: `3/10`)
- said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already
expanded — but Hermes ALSO
emitted the unexpanded form
when given the expanded
form, vs the kernel's
deterministic expand)
- and 2 more.
These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.
Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).
Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.
## Lazy concept_relations loading
Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.
Refactored arborist/concepts/query.py:
- New _load_neighbors_for(shards_dir, tokens) — targeted
WHERE token IN (...) OR target IN (...) query. Returns just the
direct synonym neighborhood for the given tokens (~300 rows for
a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
expansion exceeds max_total (the cap). Most queries never reach
the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
loaders. Eager _load_indices / _get_indices kept for any
back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.
All 14 concept tests pass unchanged — the contract is preserved.
Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):
metric pre-fix post-Phase-1 post-lazy-concepts
wall_ms 14,500 13,400 9,300 (-36%)
search_ms 9,900 10,600 5,000 (-49%)
SQLite executes 10,623 3,687 3,708 ~same
synonym_expand 2,966 2,840 ~0 (lazy hit)
Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).
## Real-shard baseline artifact (Phase 2)
bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:
- bench/results/real-shard-baseline.json (durable; commit_sha,
shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)
Question set in bench/fixtures/real-shard-baseline-v1.jsonl:
virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
(rivalry path), and two canonical-projection cases (math + logic
preflight short-circuit).
First baseline run (commit e78814c plus this fan-out, BURN=1):
audit_mode n notes
STRICT 4 (virt-back, France, Mac OS X, Linux)
HYBRID 2 (Microsoft founder, AMD/Intel)
CANONICAL 2 (0.1+0.2, A IMPL B; <1 ms each)
wall median 4.1 s (range 0.6 ms – 7.2 s)
primary used 4 / 8
`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).
Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.
`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.
## Test status
Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
