russell@unturf.com
38d9116c88
Implements the optional vec backend from the #000039 doc, with the "obvious" v1 tuning, and demonstrates it on a real corpus shard. arborist/search/vec.py (new): - VecBackend(SearchBackend) — ANN over chunk_vecs, UNGROUNDED hits (same as FTS5; vec changes recall, never warrant — embeddings are soft signal, never in the proof path). - chunk_vecs vec0 virtual table + vec_meta — sibling tables, additive, don't touch chunks/documents/the audit chain. - embed_documents() — batched ingest; delete-then-insert per chunk_id (vec0 doesn't honor INSERT-OR-REPLACE — re-inserting an existing PK is a hard UNIQUE error), so re-runs are idempotent and content- changed → re-embed works. Skips cold-evicted chunks (content NULL). - Pluggable Embedder callable; default = fastembed bge-small-en-v1.5 (~130 MB ONNX, downloads on first use). load_vec_extension(conn) toggles enable_load_extension + sqlite_vec.load. - v1 hyperparams (VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5- 384float32-cosine-flat): model bge-small-en-v1.5, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized, so cosine ranking ≡ L2 ranking), ANN flat (vec0 default), top_k 20. These five fold into governance_policy_hash in a later phase (§6). CLI (arborist/cli.py): - — populate chunk_vecs for --db; prints progress + timing. - — semantic ANN search (errors with an install/embed hint if [vec] missing or chunk_vecs empty). - Both surfaced only when sqlite_vec imports (mirrors the [html] / selectolax pattern). pyproject.toml: [vec] optional extra (sqlite-vec>=0.1.9, fastembed>=0.4); added to [dev]. Note: sentence-transformers is the heavier "official" embedder path §5 names; fastembed is the lightweight ONNX one. tests/test_search_vec.py (7 tests, skip-if-no-[vec]): deterministic stub embedder (hash → unit vector) so the suite exercises the sqlite-vec plumbing — ext load, schema, ingest, KNN, JOIN, Hit shape, limit, idempotent re-embed, --limit cap, empty/unpopulated — without the heavy fastembed model. Semantic quality is demonstrated on a shard, not unit-tested. Demonstrated on ~/.arborist/shards/crawl_appliedcombinatorics_org.db: 168 chunks embedded in ~37 s (mostly model load); semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations", "binomial coefficient counting" → "AC Introduction" (integer-solution counting) + "AC Combinatorial Proofs". None of the query tokens need stem-match the chunk — the semantic-allusion-gap closure the ticket promised. chain-check on that shard reports 0 after embedding (chunk_vecs is a sibling table). #000039 status flipped to "in progress · Phase 1 landed"; Phase 2 (RRF hybrid fusion in query.py) gated on a ≥5pp recall-lift measurement with no STRICT-rate regression (§8). (Unrelated: tests/test_weights.py::test_as_dict_returns_all_eleven_fields fails in the working tree — that's a parallel-clone in-flight change to arborist/substrate/weights.py + its test, not touched here.)
16 KiB
16 KiB
Arborist tickets — index
Arborist uses a flat ticket-NNNNNN-<slug>.md convention for actionable
proposals: defects with a known root cause, design proposals awaiting
go/no-go, or scoped enhancements queued for future implementation.
Convention
- Filename:
docs/tickets/ticket-NNNNNN-<slug>.md. The ID is a six-digit zero-padded integer; the slug is hyphen-lowercase and short enough to stay readable on a 100-column terminal. - Numbering: monotonic, never reused. Pick the next free integer
greater than every existing ticket. The current next ID is on the
Next IDline below; bump it when you open a ticket and commit the bump in the same commit as the new ticket file. - Header: every ticket starts with
# Ticket #NNNNNN — <short title> **Status:** open · awaiting go/no-go (or "in progress", "closed") **Opened:** YYYY-MM-DD **Scope:** <one-liner — what this ticket is for> **Audience:** <who reads this> **Hard constraint:** <invariants this ticket commits to NOT break> - Body: problem statement, design choices with trade-offs,
recommendation, implementation sketch, scope boundaries, status.
See
ticket-000001-retrieval-keywords-audit-gap.mdfor a worked template. - Closure: when a ticket lands, flip Status to
closed · landed in commit <sha>and leave the file in place. Do not delete tickets; they are the design log.
Distinction from other docs
docs/ also contains architecture references and bench journey notes
that are NOT tickets — they describe state of the world rather than
proposing change. Examples that stay un-numbered:
seven-point-program.md(north-star architectural directive distilled 2026-05-01; every new ticket / feature / prompt edit walks past this; bench harness reports per-mode directive coverage)cti-architecture.md(architecture reference for the Clause Tree Intelligence claim-lattice-pointer mode)mesh.md(mesh wire format + group-key state machine)benchmarks.md(bench harness orientation — fixtures, signal floor, make targets, bench-row schema, addenda index; read first when running a bench)bench-maxing.md(bench discipline — 5pp signal floor, n=3 default, avoid negation in prompts, honest verdicts over optimistic ones)qa-modes-bench.md(rolling bench-journal addenda with cross-references to per-ticket bench data)soft-hash-channel-analysis.md(analysis doc closing #000018 open questions §9.1 / §9.2 / §9.3; companion tosoft-hash-channel-t3-bound.md)soft-hash-channel-t3-bound.md(#000036 formal derivation of the per-window covert-channel budget; companion to the analysis doc, both feed the t3_bound_calculator)tool-action-dag-design.md(research path for an action- provenance layer downstream offinal_label. Captured pre-ticket per the 2026-05-07 arborist-vs-donto comparison; promotes to a ticket when an actual agent use case surfaces)warrant-substrate-cookbook.md(architecture reference for the 18 open-licensed textbook substrates backing every claim-pack record, written 2026-05-10 after the 18/92 → 92/92 push under#000031. Per-pillar substrate map, ingest patterns, alias discipline, cascade tuning, honest tier breakdown.)calculator-test-patterns.md(architecture reference for testing calculator / probe-style code. Twelve patterns split into two tiers: the original 9-item core (KAT regression, hand-computed formula assertions, monotonicity, closure / sum-of-parts invariants, parametrized invalid-input cones, CLI subprocess, doc parity, module-export shape) plus three domain-specific contract pins (projective-contract for one-way canonicalizers, dispatch-order pin for Python type-hierarchy gotchas, tie-breaking-rule pin for banker's rounding). Codified 2026-05-10 from the original three-module pattern bench acrosst3_bound_calculator/phi_alignment_probe/anchor_prg; extended same day with the four π* kernel exemplars (pi_star_arithmetic/_logic/_code/_time_series). Explicit "doesn't apply to verifier-style code" caveat.)
If a doc proposes change AND awaits a decision AND has scoped implementation cost, it's a ticket. Otherwise it's reference.
Index
Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|---|---|---|---|---|
| #000047 | ForkScore _delta_* aggregator (mean vs max vs sum) |
open · awaiting go/no-go · doc-only; parks until #000046 produces a below-ceiling baseline to bench against. Recommends: parameterize delta_aggregator (default mean), don't change the default without #000046 data. #000012-revision / #000025 §10.14 follow-up |
2026-05-11 | — |
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | in progress · Phase 1 landed 2026-05-11 — falsification-hard-v1.jsonl (12 near-misses, rate 4/12 at HEAD; verify_quotes over-grounds 8 via paraphrase/entity matching), make bench-5f-falsification-hard / bench-fork-baseline-hard, worked-example test (fork_score γ·Δ5f → positive on a lift to 1.0). Closure pending an actual verify_quotes tightening that lifts the rate; Phase 2 (extend to Formulate / a retrieval-backed sub) optional. #000025 §10.14 follow-up; gates #000047 |
2026-05-11 | — |
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across fc5ba50 / 03c0f6a / 6c6defb / f5dbfab / 3b30126 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) |
2026-05-10 | — |
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
| #000042 | Term-aliases table (vocabulary-mismatch bridge) | closed · 13 rows live across geometry + classical-physics + arithmetic domains by 2026-05-10 | 2026-05-09 | — |
| #000041 | Citation-aliases table (PD substitutes for proprietary cites) | closed · 74 rows live as of 2026-05-10 (count grew 40 → 54 → 74; Goldstein/Newton, Mendelson/Enderton/Jech/Landau/Gödel→{Russell IMP, Russell PoM, De Morgan, Boole, Cantor, Peano, Dedekind, SF-LF}, Stanley/Brualdi/Knuth → Bogart+Levin+Keller-Trotter, Dummit-Foote/Barendregt/Böhm-Jacopini → Judson/PLFA/SF, Kolmogorov → Grinstead-Snell+Laplace) | 2026-05-09 | — |
| #000040 | Phase 5 resolver fix — phrase + content-token cascade (Hilbert terminology mismatch surfaced) | closed · cascade landed 2026-05-09; lift blocked by 1902-vs-modern vocab; follow-up #000042 | 2026-05-09 | — |
| #000039 | Optional sqlite-vec retrieval backend (A/B vs FTS5, hybrid not replacement) |
in progress · Phase 0 doc + Phase 1 landed 2026-05-11 (arborist/search/vec.py — VecBackend + chunk_vecs vec0 + embed_documents + arborist embed / search --backend vec; [vec] extra = sqlite-vec + fastembed; v1 tuning: bge-small-en-v1.5 / 384 / float32 / cosine / flat / top_k 20; 7 tests; demonstrated on crawl_appliedcombinatorics_org.db — 168 chunks, semantic hits topically correct, chain-check 0). Phase 2 (RRF hybrid fusion) gated on ≥5pp recall lift |
2026-05-09 | — |
| #000038 | Phase 4 content acquisition — proprietary textbook license decisions for warrant coverage | closed · obviated 2026-05-10 by alias-substitution sprint under #000031 (74 rows in #000041 + 13 rows in #000042); 92/92 records now resolve. Residue (multilingual PD, Hilbert-Ackermann OCR, Knuth permission, personal-copy path B) preserved as design log §8 | 2026-05-09 | — |
| #000037 | Prometheus-Σ recursive falsification controller (bicameral substrate) | in progress · Phases 0 + 1 + 1.b + 1.c + 2 landed 2026-05-10; §12 Trigger 2 fired (divergence variance 0.575 / N=37); §22 Findings 2 + 3 RESOLVED (kernel/llm cost split + sweep_weights §15.4 + per-mode τ_qa); controller_events carries 4 event kinds (decision · difficulty · budget_allocation · falsification_proposal) feeding arborist controller-events inspector + live-harvest third bucket in bench/scripts/harvest_falsification_proposals.py; §12 Trigger 1 probe wired 2026-05-11 (trigger_1_branch_density reads fork_score_branches — measurable, not yet fired); Phase 3 sleep-sweep scheduler tracked under #000045 (gating ticket) |
2026-05-09 | — |
| #000036 | T3 per-window covert-channel budget bound | closed · 2026-05-11 · Phase 1 + dav1d review → Tier-1 + Tier-2 (Option B = b1_model=max_envelope default, in v1, no v2 fork) + KAT-regen tooling (scripts/generate_t3_bound_kat.py) all landed 2026-05-11; baseline 625.87 → 6183.02 (max_envelope), NOT_CERTIFIED_BY_BOUND at W=10000; 53 → 83 tests; 12-entry active KAT; both dav1d closure blockers cleared, all §5 acceptance criteria met. Continuation: empirical C_B* tightening under #000043 (parks on v7) |
2026-05-09 | — |
| #000035 | PRG choice for φ_PRG (HMAC-SHA-512 expansion) | in progress · Phase 1 landed 2026-05-10; §9.10 amendment text dav1d-reviewed final + endianness resolved 2026-05-11 (manifest field → anchor_prg_seed; v7 TLV confirmed little-endian → counter + uint32-word reads flipped big→little, PHI_PRG_VERSION → -le, 10 KATs regenerated; float-map prose corrected to "negligible mean −2^−32" not "unbiased"; dim_h ≤ 16·2^32 guard + bool-reject added; seed-independence + M1-policy-separation added; HMAC-SHA-512 / 32-byte / SHALL all LOCKED; 31 tests). Remaining: land §9.10 into v7 spec when plastic-training has a deployment target (exogenous) |
2026-05-09 | — |
| #000034 | Hessian alignment under φ_linear | in progress · Phase 1a landed 2026-05-10 (synthetic-ablation probe + KAT fixture); Phase 1b parks for v7 ramp-up | 2026-05-09 | — |
| #000033 | Claim-pack pillar VII (combinatorics) | closed · landed 2026-05-09 (live in shard 000.db; lift verified) | 2026-05-09 | — |
| #000032 | combinatorics@v1 π* (integer counting kernel) | closed · landed 2026-05-09 | 2026-05-09 | — |
| #000031 | Surface-ingest cited textbooks for claim-pack warrant promotion | closed · Phase 1 + 2 + 2.5 (92/92) + 3 (verifier wiring) + B-1 (via_citation_alias attribution: 18 direct + 74 +alias) + B-2 (source-side title-from-author backfill) all landed 2026-05-09 / 2026-05-10 | 2026-05-09 | — |
| #000030 | Math π* expansion: SymPy substrate (algebra · calculus · linalg) | closed · all 7 phases + 1b landed 2026-05-09 (abe5988) |
2026-05-09 | — |
| #000029 | Claim-pack source (axiom/theorem JSON bundles) | closed · landed 2026-05-09 | 2026-05-09 | — |
| #000028 | Multi-modality witness for canonical shapes | closed · landed 2026-05-09 + follow-ups (capital ledger · sample rate) | 2026-05-08 | — |
| #000027 | Canonical projections persist to providence_cache | closed · landed 2026-05-09 | 2026-05-08 | — |
| #000026 | Real-shard workload baseline + search latency | closed · Phase 1 + 2 + 3 landed 2026-05-09 (Phase 3 in 60b5748) |
2026-05-08 | — |
| #000025 | 5F battery (Function · Finetuning · Falsification · Formulate · Feedback Loop) | closed · 2026-05-11 — Phase 1a–1f landed 2026-05-09/10; Phase 1g (§10.13 feedback latency + persisted-footprint efficiency), 1h (§10.14 ForkScore threshold-calibration handoff → #000012 §8), 1i (§10.11 persistent SelfModel-chain lineage: bench-5f-selfmodel-snapshot grows the chain, run_finetuning shard-chain mode reads the 2 latest snapshots) all landed 2026-05-11. Every §10 closure criterion met |
2026-05-07 | — |
| #000024 | 5T Phase 1b + Dav1DPrometheus vocabulary alignment | closed · landed 2026-05-08 | 2026-05-07 | — |
| #000023 | 5S Phase 1b: Syllogism · Synthesis · Semiotics | closed · landed 2026-05-08 | 2026-05-07 | — |
| #000022 | Adapter LossReport (PRD I9 analogue) | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000021 | 5S/5T/5R benchmark fixtures + harness | closed · Phase 1a + 2 landed 2026-05-08 | 2026-05-07 | — |
| #000020 | Capital-cost ledger (8-capital queues) | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000019 | Specification methodology for π* and V | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000018 | Adversarial soft-hash covert-channel analysis | closed · landed 2026-05-09 (analysis doc) | 2026-05-07 | — |
| #000017 | Memory-root: lifelong learning audit chain | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000016 | ZK Phase-2 frontier proof (concretize) | parked · bench-plan + wire-protocol landed 2026-05-09 (sibling-repo measurement pending) | 2026-05-07 | — |
| #000015 | π* domain library + cross-domain composition | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000014 | SelfModel: schema, falsification, integration | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000013 | Spatial-temporal substrate (Merkle-AGI v7-W) | closed · landed 2026-05-09 (substrate paper + frontier catalog + namespace stub) | 2026-05-07 | — |
| #000012 | Selection & consensus protocol (Merkle-AGI v8) | in progress · Phase 1a (ForkScore) landed 2026-05-08; Phase 1b (consensus paper, docs/_source/merkle-agi-v8-consensus.rst 834 lines) landed 2026-05-10; Phase 1c (branch-set persistence — fork_score_branches sibling table, persist_branch_score + branch_set_density, 6 new CLI flags on arborist substrate score, default-off) landed 2026-05-10 — feeds #000037 §12 Trigger 1; Trigger 1 probe wired 2026-05-11 (trigger_1_branch_density reads fork_score_branches via branch_set_density() — measurable, not yet fired: no branch sets persisted) |
2026-05-07 | — |
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | closed · landed 2026-05-04 (zero-shot full impl) | 2026-05-04 | D1 (preserves) |
| #000010 | Meta-Cognition Preflight Guard (M0 / MCTL) | closed · landed 2026-05-03 (Phases 1–4); DAG binding shipped via #000009 | 2026-05-03 | D1, D3 |
| #000009 | Preflight run-DAG node binding (#000008+#000010) | closed · re-landed 2026-05-04 (§8 corrections: reject-path DAG, nested CTI clauses) | 2026-05-03 | D3, D4 |
| #000008 | Broad-quantifier preflight guard | closed · landed in 4f2b5a6; Phase 5 DAG binding split into #000009 |
2026-05-02 | — |
| #000007 | Query-layer hyphen folding | closed · 2026-05-02 | 2026-05-02 | — |
| #000006 | Bench-emergent findings (rolling research log) | open · rolling research log (verifier-ladder healthy at 134 cycles; emergent gems still being mined) | 2026-05-02 | — |
| #000005 | Label ladder migration (POINTER-LINKED → …) | closed · 2026-05-02 | 2026-05-01 | D7 |
| #000004 | Directive coverage in bench summary | closed · acd1f9c |
2026-05-01 | D8 |
| #000003 | Anchor-class warrant generalization (Module H+) | closed · 2026-05-02 | 2026-05-01 | D6 |
| #000002 | Reference-Frame Polarity Contract (Module L) | closed · 2026-05-02 | 2026-05-01 | D3 |
| #000001 | Retrieval-keywords audit gap | closed · 2026-05-02 | 2026-05-01 | D4 |
Next ID
000048