Commit graph

9 commits

Author SHA1 Message Date
514e07d7c2
textbooks: TeX-source ingest closes pillars I + IV (Hilbert + Boole)
Two foundational PD textbooks ship from Project Gutenberg as
LaTeX source only — no clean HTML edition. Pandoc fails on PG's
custom preamble macros; a focused regex-based stripper is the
right amount of machinery for the well-known PG TeX format.

What landed
===========
- arborist/sources/textbook_tex.py — TextbookTexSource +
  strip_tex pipeline. Drops preamble + line comments + structural
  envs (tabular, figure, thebibliography, scshape, …); keeps the
  argument of structural-but-content-bearing single-arg commands
  (textbf, emph, section, chapter, paragraph, PG's custom \\rfa);
  drops zero-arg + brace-arg structural commands (noindent,
  thispagestyle, setcounter, label, index, …); substitutes
  symbol-level macros (\\to → →, \\neg → ¬, \\forall → ∀, \\S → §,
  Greek letters, etc.).

- arborist/cli.py — `--source textbook_tex` accepts --url,
  --bundle, or --urls-from. Reuses the existing fetch + chunker
  + Merkle commit + audit pipeline; idempotent at the database
  layer (same TeX → same prose → same document_root).

- bench/scripts/textbooks_manifest.py — gains `tex-targets`
  subcommand emitting tab-separated `<tex_url>\t<id>` rows for
  manifest entries with a `tex_url` field.

- Makefile targets:
    textbooks-tex         — ingest every entry with a tex_url
    textbook-hilbert      — convenience for PG #17384
    textbook-boole        — convenience for PG #15114
  Each writes to $(CRAWL_SHARDS_DIR)/textbook_<id>.db, idempotent
  on re-run.

- tests/test_textbook_tex.py — 20 unit tests covering preamble +
  postmatter stripping, line comments, env drops (tabular, figure),
  single-arg keepers (\\textbf, \\emph, \\section, \\rfa),
  symbol-level macro subs (10 paramerized cases), structural-cmd
  drops, whitespace cleanup, idempotence on already-stripped text.

End-to-end verification
=======================
Smoke test on PG #17384 + #15114:
  Hilbert Foundations of Geometry: 1 doc / 65 chunks (192K of
    plain prose). FTS5 finds "axiom of parallels" → real chapter
    content with axiom references intact (≡, §, math fragments).
  Boole Laws of Thought: 1 doc / 273 chunks (829K). FTS5 finds
    "law of contradiction" → "the principle of contradiction"
    passage from Chapter III of Boole's text.

Vital-books coverage now 6/7 pillars
====================================

  Pillar I   Logic        ✓ Levin + Aristotle Prior + Posterior + Boole
  Pillar II  Set Theory   ✓ Levin
  Pillar III Arithmetic   ✓ Levin
  Pillar IV  Geometry     ✓ Hilbert (PG TeX)
  Pillar V   Probability  ✗ Kolmogorov license analysis pending
  Pillar VI  Phys.        ✓ Newton Principia
  Pillar VII Combin.      ✓ Bogart + Keller-Trotter + Levin
  Pillar IX  λ-Calculus   ✗ Church + Turing 1936 papers pending

Test suite: 1574 passed / 28 skipped (was 1554 + 20 new TeX tests).

Out of scope: chunk-resolution + derivations.proof_blob warrant
promotion (#000031 follow-up; see also #000032).
2026-05-09 16:06:53 -04:00
57dafbdc3d
textbooks: per-book make targets + idempotency docs + vital-books checklist
Three concrete improvements to the textbook ingest layer:

Per-book make targets
=====================
- `make textbook ID=<id>` — generic, looks up one entry from the
  manifest and runs the right ingest path (BFS crawl if
  `crawl_url` is set, shallow URL ingest otherwise).
- `make textbook-list` — list ingestable textbook ids.
- Convenience targets per active manifest entry:
    textbook-bogart, textbook-keller-trotter, textbook-levin,
    textbook-aristotle-prior, textbook-aristotle-posterior,
    textbook-newton, textbook-morin

Each per-book target writes to a dedicated shard at
`$(CRAWL_SHARDS_DIR)/textbook_<id>.db` — one shard per textbook
id rather than one per host (which would conflate multiple
Wikisource books — Aristotle Prior, Posterior, Newton — into
one shard).

The manifest helper script gains `ids` and `lookup <id>`
subcommands so the Makefile can introspect manifest entries
without grep gymnastics.

Idempotency contract documented
================================
Every ingest path is content-addressed. Same content →
same `document_root` → no-op insert. Re-running `make textbook-*`
or `make crawl-textbooks` on the same network state is safe at
the database layer — no duplicates, no schema drift, no
audit-event proliferation. HTTP refetch happens every run; pair
with `make recrawl-check` for a HEAD-only freshness probe.

README rewritten to make this contract explicit, with shard-
layout table + crawl-delay caveats consolidated.

Vital-books checklist
=====================
The README now carries a g4-pillar coverage table:

  Pillar I   Logic        ✓ Levin + Aristotle Prior + Posterior
  Pillar II  Set Theory   ✓ Levin
  Pillar III Arithmetic   ✓ Levin
  Pillar IV  Geometry     ✗ Hilbert PG #17384 (PDF/TeX only)
  Pillar V   Probability  ✗ Kolmogorov license analysis pending
  Pillar VI  Phys. Newton ✓ Newton Principia Motte
  Pillar VII Combin.      ✓ Bogart + Keller-Trotter + Levin
  Pillar IX  λ-Calculus   ✗ Church + Turing 1936 papers pending

5 of 7 pillars now have surface coverage. The remaining gaps
are explicit license (V) or format (IV PDF/TeX, IX paper-level)
issues, each documented in the corresponding manifest entry's
notes field.

Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
2026-05-09 15:59:31 -04:00
f5c2c03d6a
textbooks: extend manifest for #000031 PD scope (Newton, Posterior Analytics, +placeholders)
Per the #000031 implementation track — surface-ingest cited
textbooks for warrant promotion of claim-pack records. Five new
manifest entries spanning the PD-scope additions plus
license-fail placeholders for citation traceability:

  newton-principia-motte (PD)
    Newton's Mathematical Principles of Natural Philosophy
    (Motte 1729 / Motte-Cajori 1846 English translations) via
    Wikisource. Crawled @ depth=2: 60 docs, 289 chunks, 10K
    edges. FTS5 spot-checks: "law of motion" → top hit on the
    1729 Motte title page; "gravitation" → Newton-titled
    chapters in the Wikisource Physics category. Cited by
    claim-pack pillar VI.

  aristotle-posterior-analytics (PD)
    Mure / Owen translations via Wikisource. Crawled @ depth=1:
    20 docs, 71 chunks. FTS5: "syllogism" → Posterior Analytics
    Bouchier + Owen translations as top hits. Companion to
    Prior Analytics already in the manifest; together cover
    pillar I logic axioms.

  kolmogorov-foundations-probability-1933 (placeholder)
    German original PD-by-age in EU; US copyright restored via
    URAA through life+70 (Kolmogorov d.1987 → US PD 2058).
    Morrison 1956 English translation: Chelsea Publishing
    copyright. Entry kept with empty urls field for citation
    traceability; license analysis pending.

  lukasiewicz-aristotle-syllogistic-1951 (proprietary placeholder)
    Oxford University Press, copyright through life+70
    (Łukasiewicz d.1956 → PD 2027 EU, longer US via URAA). NOT
    redistributable. Kept as a license-fail placeholder; the
    1921 Polish primary papers are themselves PD by age but
    lack accessible HTML editions.

License-validator drive-by
==========================
The textbooks_manifest.py validator now treats placeholder
entries (urls + crawl_url both empty) as exempt from the
allow-list license check. License discipline stays fail-closed
at the URL-emit step — placeholders without URLs raise no
exception under any license token, so we can document
yellow-light proprietary citations (Mendelson, Enderton,
Łukasiewicz) without hacks.

Coverage summary post-commit
============================
g4 pillar    | textbook coverage
─────────────┼─────────────────────────────────────────────────
I  Logic     | Levin (CC-BY-SA), Aristotle Prior + Posterior (PD)
II Set Theory| Levin (CC-BY-SA)
III Arithm.  | Levin (CC-BY-SA)
IV Geometry  | (Hilbert PG #17384 placeholder — PDF/TeX only)
V  Probability| (Kolmogorov 1933 license analysis pending)
VI Phys.     | Newton Principia Motte (PD)
VII Combin.  | Bogart (GFDL), Keller-Trotter (CC-BY-SA), Levin

Out-of-scope follow-ups: PDF/TeX source support (#000031 §5),
Mendelson + Enderton license decision (#000031 §2.1),
chunk-resolution + derivations.proof_blob warrant promotion
(separate ticket).
2026-05-09 15:47:42 -04:00
31235b6204
textbooks: deep ingest via existing crawler (one shard per host)
The first iteration of this layer landed scaffolding plus shallow
seed-URL ingest, which produced 300-1000 byte stubs (front matter
+ nav chrome only) instead of real book content. Fox flagged
that as the wrong outcome — we were supposed to track down the
actual texts.

This commit fixes it by adding the deep-ingest path: each
manifest entry now carries `crawl_url` + `crawl_depth` +
`crawl_max` fields, and `make crawl-textbooks` iterates them
through the existing BFS crawler (one shard per host).

Manifest changes
================
Three URL fixups uncovered by the first crawl attempt:

- PreTeXt textbooks (Bogart, Keller-Trotter, Levin) ship an
  `index.html` that's a 1-line meta-refresh redirect; the BFS
  crawler doesn't follow meta-refresh, so the seed URL has to
  point at the actual content entry (`frontmatter.html`,
  `app-comb-2.html`, `ctgd/index.html`).
- Open Data Structures uses trailing-slash directory paths
  (`/ods-cpp/`, `/ods-python/`, `/ods-java/`), not `/ods-cpp.html`.
- `appliedcombinatorics.org` declares `Crawl-Delay: 20` in
  robots.txt — full BFS takes ~25 minutes; documented in the
  manifest entry's notes so operators know to background it.

Hilbert + Boole stay flagged as `urls: []` placeholders because
PG eBooks #17384 + #15114 ship as PDF/TeX only (no HTML format).
Their `tex_url` + `pdf_url` fields are now preserved on the
entry for the future PDF/TeX source ticket.

Make targets
============
- crawl-textbooks         — BFS-crawl every entry with `crawl_url`
- crawl-textbooks-stats   — docs/chunks/size summary across crawl shards

Helper script: `bench/scripts/textbooks_manifest.py crawl-targets`
emits one tab-separated `<url>\t<depth>\t<max>\t<id>` row per
entry that declares a `crawl_url`.

Real-content verification
=========================
Crawl results across 4 of 5 textbooks (Keller-Trotter still
running in background due to crawl-delay; expected to complete
in ~25 min):

  crawl_bogart_openmathbooks_org.db       docs=44   chunks=161   971 KB
  crawl_discrete_openmathbooks_org.db     docs=51   chunks=376  1929 KB
  crawl_en_wikisource_org.db              docs=30   chunks=110  1651 KB
  crawl_opendatastructures_org.db         docs=64   chunks= 84  1110 KB

  TOTAL                                    189 docs, 731 chunks

FTS5 spot-checks confirm substantive content landed:
  - Bogart "binomial coefficient" → "CTGD Partitions and Stirling
    Numbers" chapter, top hit
  - Levin "modus ponens" → "Propositional Logic" chapter, top hit

That's the real lift signal we were after. Out-of-scope
follow-ups remain: PDF/TeX support for PG editions (Hilbert,
Boole) and full Aristotle Owen-translation depth (currently
~75% complete on Wikisource).
2026-05-09 15:36:53 -04:00
4a199c9dad
textbooks: PD/open-licensed math+logic+CS surface-ingest pipeline
A make-target scaffold for pulling public-domain and copyleft-
redistributable textbooks into a dedicated arborist shard via the
existing HtmlPageSource ingest pipeline. No new ingest machinery —
the existing path (robots.txt → noise-strip → 512-token chunk →
Merkle root → audit-event) IS the consistent process.

Manifest format
===============
bench/fixtures/textbooks/manifest-v1.jsonl — one JSON record per
textbook with explicit license tokens (PD / CC-BY / CC-BY-SA /
GFDL / OSI-approved). Fail-closed validation in
bench/scripts/textbooks_manifest.py refuses to emit URLs from
entries with missing or disallowed license tokens, so license
discipline surfaces at the URL-emit step rather than getting
silently buried in a fetch run.

Allow-list excludes CC-BY-NC (incompatible with arborist's
AGPLv3 distribution profile) and CC-BY-ND (no-derivatives
prevents chunking). Wilf's generatingfunctionology stays out
because its educational-use license forbids rehosting; citable
but not redistributable.

Initial entries (7 textbooks, 8 seed URLs)
==========================================
- Bogart, Combinatorics Through Guided Discovery — GFDL-1.3
- Keller & Trotter, Applied Combinatorics — CC-BY-SA-4.0
- Levin, Discrete Mathematics: An Open Introduction (3rd ed) —
  CC-BY-SA-4.0 (4th ed switched to NC; pin 3rd)
- Hilbert, Foundations of Geometry (Townsend 1902) — PD via PG
- Boole, An Investigation of the Laws of Thought (1854) — PD
- Aristotle, Prior Analytics (Jenkinson) — PD via Wikisource
- Morin, Open Data Structures — CC-BY-2.5

Covers pillars I (logic) / II (set theory) / III (arithmetic) /
IV (geometry) / VII (combinatorics) on the claim-pack
(#000029) layout, plus a CS anchor for downstream domain
expansion. Each entry's pillar_targets field lists which
claim-pack pillars its records are candidate citations for.

Make targets
============
- textbooks-summary  — license + URL counts per entry
- textbooks-urls     — flat URL list to stdout
- fetch-textbooks    — ingest all manifested URLs into
                       $(TEXTBOOK_DB) (default
                       ~/.arborist/textbooks.db)
- textbooks-stats    — documents / chunks / edges
- textbooks-verify   — sample Merkle proof verification

To grow coverage of one textbook: append more chapter URLs to
its `urls` array. For deep-BFS of a textbook home: use the
existing `make crawl-ingest URL=<base> DEPTH=N` instead.

Smoke-tested end-to-end against Wikisource Prior Analytics:
1 doc / 1 chunk / 12 outbound edges / Merkle proof passes.
Shows the pipeline works; populating each book to depth needs
either more URLs in the manifest or the crawler.

Out of scope
============
- PDF processing. The Internet Archive hosts PD textbooks
  (MacMahon's Combinatory Analysis 1915, Whitworth's Choice
  and Chance 1867, the Motte 1729 Principia) as scanned PDFs;
  a TextbookPdfSource with pdftotext / pypdf extraction is a
  separate ticket — current scope is HTML-shaped sources.
- Mendelson + Enderton (proprietary; await #000031 §2.1
  decision).
- Wilf generatingfunctionology (license forbids redistribution).

Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
2026-05-09 14:58:22 -04:00
7871e1fdfe
fan-out: witness automation · #000018 analysis · #000033 pillar VII (Claude)
Three streams. Two land in the repo; one lands in fox's Downloads
(existing g4 packs live there too).

Witness sweep automation
========================

`bench/scripts/witness_sweep_cron.sh` — schedulable harness that
runs `make bench-witness-sweep` + `make bench-witness-divergence`
back-to-back and (with --commit) auto-commits any new divergence
fixtures + sweep result. Fail-closed: pre-commit hook failure
aborts; non-main-branch refuses to commit.

  $ bench/scripts/witness_sweep_cron.sh                 # dry run
  $ bench/scripts/witness_sweep_cron.sh --commit        # auto-commit
  $ bench/scripts/witness_sweep_cron.sh --commit --push # full pipeline

Recommended cron (every 6 hours; LLM cost scales with frequency):

  0 */6 * * * \
      bench/scripts/witness_sweep_cron.sh --commit --branch main \
      >> /var/log/arborist-witness.log 2>&1

Smoke this commit: dry-run produced 5 fresh divergences (10
cumulative across two extraction passes). The witness-sweep + 5F
extractor pipeline runs unattended now; calibration-data corpus
will grow over time.

#000018 — adversarial soft-hash covert-channel analysis
========================================================

`docs/soft-hash-channel-analysis.md` — formal threat model + channel-
capacity definition + cryptographic reduction + mitigation table
+ residual analysis. Doc-only deliverable per the ticket's hard
constraint.

Headline result: under threat models T1 (data poisoner) and T2
(gradient adversary), the channel is bounded by SHA-256 partial-
preimage hardness when φ (the anchor map) is independent of
parameter geometry. T3 (hyperparameter adversary) requires
M2 (per-checkpoint anchor nonce) for boundedness.

Recommended mitigation: M2 (per-checkpoint nonce) — 32 bytes added
to the snapshot manifest, no training-ergonomics regression,
closes T1+T2 channels under any φ choice and bounds T3 by
per-checkpoint window. Fall back to M1 (PRG-based φ) if M2-only
deployment surfaces structural concerns. M3 (drop anchor entirely)
stays in reserve as the strict-construction fallback.

Three open questions (§9): Hessian alignment under φ_linear,
PRG choice for φ_PRG, and explicit T3 per-window bound. Each is a
follow-up ticket.

Ticket #000018 status: closed · landed 2026-05-09 (analysis doc).
v7 § 9.10 spec amendment proposed in §7 of the analysis.

#000033 — pillar VII (combinatorics), Claude-authored
======================================================

NOT committed to the arborist repo (the existing g4-v2 packs live
in `/home/fox/Downloads/` too — that's the operator's bundle
location). Two new bundle files at:

  /home/fox/Downloads/axiomsclaude-vii-v1.json    (7 axioms)
  /home/fox/Downloads/theoremsclaude-vii-v1.json  (7 theorems)

Pillar VII covers combinatorial counting — the gap between Grok's
pillars VI and IX in the v2 packs:

  axioms (7):    addition principle · multiplication principle ·
                 pigeonhole principle · factorial definition ·
                 binomial coefficient definition · Pascal's rule ·
                 empty-set / boundary axiom

  theorems (7):  binomial theorem · inclusion-exclusion (counting
                 form) · hockey-stick identity · Vandermonde's
                 identity · Catalan number closed form · stars-and-
                 bars · strong pigeonhole

Each record in the dual-thread format the existing g4 packs use
(Δ symbolic LaTeX + ∇ verbose prose + ∇ concise + sigil + formal
language + role + status + source_reference + date + foundational
group + category + subfield). Per fox's directive: explicit
authorship metadata everywhere — `authored_by: Claude (Anthropic)
— model claude-opus-4-7`. NOT Grok-generated; no silent invention.

Each record carries `pi_star_ref: combinatorics@v1` so the kernel
binding is explicit. Theorems list `depends_on_axioms` arrays so
each theorem cites the foundation axioms it bottoms out on.

Smoke test (committed alongside):

  $ arborist --db /tmp/test.db ingest --source claim_pack \\
        --bundle /home/fox/Downloads/axiomsclaude-vii-v1.json \\
        --bundle /home/fox/Downloads/theoremsclaude-vii-v1.json
  → 14 docs, 14 chunks, 0 cross-bundle edges

Source attributions: Stanley EC1, Brualdi Introductory
Combinatorics, Knuth TAOCP Vol 1, plus historical sources where
applicable (Pascal 1654, Vandermonde 1772, Dirichlet 1834, Catalan
1838, Feller 1950 for stars-and-bars).

Tests: 1636 passed, 37 skipped (no regressions; pillar VII
ingestion smoke covered above).
2026-05-09 14:36:39 -04:00
bc77f961f3
fan-out: close #000030 · composition fixtures · witness end-to-end
Three small streams:

#3 — close #000030 properly
============================

All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:

  Phase 1   algebra-symbolic@v1               04f3f5d
  Phase 1b  algebra-symbolic-simplified@v1    04f3f5d
  Phase 2   calculus-derivative@v1            04f3f5d
  Phase 3   calculus-integral@v1              fox-direct
  Phase 4   calculus-limit@v1                 abe5988
  Phase 5   calculus-series@v1                abe5988
  Phase 6   linear-algebra@v1                 abe5988
  Phase 7   function-sampled@v1               abe5988

Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.

#5 — composition fixtures across new SymPy π*'s
================================================

12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:

- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
  expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
  identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
  fingerprint is order-sensitive; composite domain == inner domain;
  composite bytes == manual chain bytes.

Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().

#4 — end-to-end witness sweep against real shards + Hermes
===========================================================

New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.

`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.

First real sweep (this commit, against Hermes-3-8B):

  agreement label             count    rate
  KERNEL-LLM-DIVERGED         5        62.5%
  KERNEL-LLM-AGREE            3        37.5%
  ───────────────────────────────────────────
  divergence_count            5        62.5%
  wall median / max           130 ms / 1.1 s

Hermes diverged on 5/8 of the canonical-shape questions:

- said `1/10` for `0.1 + 0.2`           (kernel: `3/10`)
- said `TRUE` for `A IMPL B`            (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1`  (kernel: `(x+1)**2` already
                                         expanded — but Hermes ALSO
                                         emitted the unexpanded form
                                         when given the expanded
                                         form, vs the kernel's
                                         deterministic expand)
- and 2 more.

These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.

Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).

Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
2026-05-09 13:29:59 -04:00
70ffc01ce4
fan-out: witness audit + 5F extractor + function-sampled demo + docs
Three small streams in one commit:

#000028 follow-up — witness divergence → 5F fixtures
=====================================================

Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.

New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).

Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.

Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).

Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.

#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================

`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.

  $ make demo-plot Q='sin(x)' PNG=/tmp/sin.png

Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.

Public docs polish (#7)
========================

- docs/_source/bench.rst: updated fixture-count narrative (~660 →
  662 default tasks + ~110 math π* fixtures); `make` quick-reference
  now lists all per-π* 5S targets (tabular, calculus-limit/series,
  linear-algebra, function-sampled) plus bench-real-shard,
  bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
  documentation + a Make-harness sub-section covering
  bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
  env-vars.

Tests
=====

- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
  write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
  determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
2026-05-09 13:19:30 -04:00
8b1de208c2
#000026 Phase 2: real-shard baseline + lazy concept loading
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.

## Lazy concept_relations loading

Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.

Refactored arborist/concepts/query.py:

- New _load_neighbors_for(shards_dir, tokens) — targeted
  WHERE token IN (...) OR target IN (...) query. Returns just the
  direct synonym neighborhood for the given tokens (~300 rows for
  a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
  ~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
  expansion exceeds max_total (the cap). Most queries never reach
  the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
  don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
  loaders. Eager _load_indices / _get_indices kept for any
  back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.

All 14 concept tests pass unchanged — the contract is preserved.

Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):

  metric           pre-fix   post-Phase-1   post-lazy-concepts
  wall_ms          14,500    13,400         9,300       (-36%)
  search_ms         9,900    10,600         5,000       (-49%)
  SQLite executes  10,623     3,687         3,708         ~same
  synonym_expand     2,966     2,840           ~0       (lazy hit)

Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).

## Real-shard baseline artifact (Phase 2)

bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:

- bench/results/real-shard-baseline.json (durable; commit_sha,
  shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)

Question set in bench/fixtures/real-shard-baseline-v1.jsonl:

  virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
  (rivalry path), and two canonical-projection cases (math + logic
  preflight short-circuit).

First baseline run (commit e78814c plus this fan-out, BURN=1):

  audit_mode      n   notes
  STRICT          4   (virt-back, France, Mac OS X, Linux)
  HYBRID          2   (Microsoft founder, AMD/Intel)
  CANONICAL       2   (0.1+0.2, A IMPL B; <1 ms each)

  wall median   4.1 s   (range 0.6 ms – 7.2 s)
  primary used  4 / 8

`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).

Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.

`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.

## Test status

Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
2026-05-08 13:10:58 -04:00