russell@unturf.com
38d9116c88
Implements the optional vec backend from the #000039 doc, with the "obvious" v1 tuning, and demonstrates it on a real corpus shard. arborist/search/vec.py (new): - VecBackend(SearchBackend) — ANN over chunk_vecs, UNGROUNDED hits (same as FTS5; vec changes recall, never warrant — embeddings are soft signal, never in the proof path). - chunk_vecs vec0 virtual table + vec_meta — sibling tables, additive, don't touch chunks/documents/the audit chain. - embed_documents() — batched ingest; delete-then-insert per chunk_id (vec0 doesn't honor INSERT-OR-REPLACE — re-inserting an existing PK is a hard UNIQUE error), so re-runs are idempotent and content- changed → re-embed works. Skips cold-evicted chunks (content NULL). - Pluggable Embedder callable; default = fastembed bge-small-en-v1.5 (~130 MB ONNX, downloads on first use). load_vec_extension(conn) toggles enable_load_extension + sqlite_vec.load. - v1 hyperparams (VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5- 384float32-cosine-flat): model bge-small-en-v1.5, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized, so cosine ranking ≡ L2 ranking), ANN flat (vec0 default), top_k 20. These five fold into governance_policy_hash in a later phase (§6). CLI (arborist/cli.py): - — populate chunk_vecs for --db; prints progress + timing. - — semantic ANN search (errors with an install/embed hint if [vec] missing or chunk_vecs empty). - Both surfaced only when sqlite_vec imports (mirrors the [html] / selectolax pattern). pyproject.toml: [vec] optional extra (sqlite-vec>=0.1.9, fastembed>=0.4); added to [dev]. Note: sentence-transformers is the heavier "official" embedder path §5 names; fastembed is the lightweight ONNX one. tests/test_search_vec.py (7 tests, skip-if-no-[vec]): deterministic stub embedder (hash → unit vector) so the suite exercises the sqlite-vec plumbing — ext load, schema, ingest, KNN, JOIN, Hit shape, limit, idempotent re-embed, --limit cap, empty/unpopulated — without the heavy fastembed model. Semantic quality is demonstrated on a shard, not unit-tested. Demonstrated on ~/.arborist/shards/crawl_appliedcombinatorics_org.db: 168 chunks embedded in ~37 s (mostly model load); semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations", "binomial coefficient counting" → "AC Introduction" (integer-solution counting) + "AC Combinatorial Proofs". None of the query tokens need stem-match the chunk — the semantic-allusion-gap closure the ticket promised. chain-check on that shard reports 0 after embedding (chunk_vecs is a sibling table). #000039 status flipped to "in progress · Phase 1 landed"; Phase 2 (RRF hybrid fusion in query.py) gated on a ≥5pp recall-lift measurement with no STRICT-rate regression (§8). (Unrelated: tests/test_weights.py::test_as_dict_returns_all_eleven_fields fails in the working tree — that's a parallel-clone in-flight change to arborist/substrate/weights.py + its test, not touched here.)
100 lines
2.9 KiB
TOML
100 lines
2.9 KiB
TOML
[build-system]
|
|
requires = ["setuptools>=68"]
|
|
build-backend = "setuptools.build_meta"
|
|
|
|
[project]
|
|
name = "arborist"
|
|
version = "0.0.1"
|
|
description = "An arborist for trees and forests of cross-linked information"
|
|
readme = "README.md"
|
|
license = { text = "AGPL-3.0-only" }
|
|
requires-python = ">=3.10"
|
|
authors = [
|
|
{ name = "Russell Ballestrini", email = "russell@unturf.com" },
|
|
{ name = "foxhop" },
|
|
{ name = "TimeHexOn" },
|
|
]
|
|
dependencies = [
|
|
"httpx>=0.27",
|
|
"zstandard>=0.22",
|
|
"cryptography>=42",
|
|
]
|
|
|
|
[project.optional-dependencies]
|
|
html = [
|
|
"selectolax>=0.3",
|
|
]
|
|
wikitext = [
|
|
"mwparserfromhell>=0.6",
|
|
]
|
|
mesh = [
|
|
# httpx is already in core deps; mesh wire only depends on stdlib +
|
|
# cryptography (also core). This extras block exists as the documented
|
|
# opt-in surface even though no extra packages are required today.
|
|
]
|
|
math = [
|
|
# Symbolic algebra/calculus π* substrate (ticket #000030). SymPy is
|
|
# ~30 MB installed; pulling it into core deps would inflate every
|
|
# fresh checkout. Tests skip via pytest.importorskip when absent.
|
|
"sympy>=1.13",
|
|
]
|
|
hessian = [
|
|
# Phi_alignment_probe (ticket #000034 Phase 1a). Lanczos top-k +
|
|
# bottom-k eigendecomposition for measuring whether v7's frozen
|
|
# linear projection W aligns with the loss Hessian's low-eigenvalue
|
|
# subspace. Numpy + scipy together ~80 MB; gated separately from
|
|
# core to keep the default install lightweight. Tests skip via
|
|
# pytest.importorskip when absent. Install with:
|
|
# pip install 'arborist[hessian]'
|
|
"numpy>=1.26",
|
|
"scipy>=1.11",
|
|
]
|
|
crawler = [
|
|
# Verbatim lift from agents.ai.unturf.com/core. Off by default — the
|
|
# default test suite never imports the crawler. Install with:
|
|
# pip install 'arborist[crawler]'
|
|
# then run `make test-crawler`.
|
|
"aiohttp>=3.8",
|
|
"beautifulsoup4>=4.11",
|
|
"lxml>=4.9",
|
|
"html5lib>=1.1",
|
|
"html2text>=2024.2.26",
|
|
"miniuri>=1.1",
|
|
"feedparser>=6.0",
|
|
"Pillow>=10.0",
|
|
"cairosvg>=2.7",
|
|
"pypdf>=4.0",
|
|
]
|
|
vec = [
|
|
# Optional sqlite-vec semantic retrieval backend (ticket #000039).
|
|
# sqlite-vec ships only the loadable SQLite extension (~1 MB);
|
|
# fastembed pulls onnxruntime + tokenizers + huggingface-hub
|
|
# (~150 MB) and downloads the bge-small-en-v1.5 ONNX model
|
|
# (~130 MB) on first use. Gated separately so a fresh checkout
|
|
# stays python3.12 + venv + sqlite3. CLI surfaces `arborist embed`
|
|
# / `--backend vec` only when `sqlite_vec` imports. Install with:
|
|
# pip install 'arborist[vec]'
|
|
# (sentence-transformers is the heavier "official" embedder path
|
|
# the ticket §5 names; fastembed is the lightweight ONNX one.)
|
|
"sqlite-vec>=0.1.9",
|
|
"fastembed>=0.4",
|
|
]
|
|
dev = [
|
|
"pytest>=8",
|
|
"pytest-asyncio>=0.23",
|
|
"pytest-xdist>=3.5",
|
|
"arborist[html]",
|
|
"arborist[wikitext]",
|
|
"arborist[mesh]",
|
|
"arborist[crawler]",
|
|
"arborist[math]",
|
|
"arborist[hessian]",
|
|
"arborist[vec]",
|
|
]
|
|
|
|
[project.scripts]
|
|
arborist = "arborist.cli:main"
|
|
|
|
[tool.setuptools.packages.find]
|
|
where = ["."]
|
|
include = ["arborist*"]
|