russell@unturf.com
d25c0fe66f
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with identical document/edge counts; Merkle proofs round-trip 30/30): 1. zstd-compressed chunks.content (level 3). Magic-byte detection on read means legacy plaintext rows pass through unchanged. Cores stay plaintext so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working. 2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root, dst_uri, anchor) covers every column, so a default rowid-based table near-doubles row data in the PK index. WITHOUT ROWID makes the table itself the B-tree. Drops idx_edges_dst_uri too — the only query that filters on dst_uri alone is gravity_top_inbound, a one-shot analytic. 3. contentless FTS5 (content='', contentless_delete=1) eliminates the 28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored. chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5 rowid maps back to chunks.chunk_id at search time. Snippets are built in Python (search/fts5.py:_build_snippet) since SQL snippet() returns empty in contentless mode. TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match let "intelligence", "intellectual", "intellivision" drown real hits like Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the distinct query tokens — multi-token coverage outranks single-token title boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty "insufficient sources" reply. Schema-level changes affect new DBs only; existing v9.8 DBs keep working at the old layout. Cross-shard UNION views explicitly list the intersection of columns so a mixed cluster (legacy + new schema shards in one --shards-dir) still unions cleanly.
36 lines
707 B
TOML
36 lines
707 B
TOML
[build-system]
|
|
requires = ["setuptools>=68"]
|
|
build-backend = "setuptools.build_meta"
|
|
|
|
[project]
|
|
name = "aborist"
|
|
version = "0.0.1"
|
|
description = "An arborist for trees and forests of cross-linked information"
|
|
readme = "README.md"
|
|
license = { text = "AGPL-3.0-only" }
|
|
requires-python = ">=3.10"
|
|
authors = [
|
|
{ name = "Russell Ballestrini", email = "russell@unturf.com" },
|
|
{ name = "foxhop" },
|
|
{ name = "TimeHexOn" },
|
|
]
|
|
dependencies = [
|
|
"httpx>=0.27",
|
|
"zstandard>=0.22",
|
|
]
|
|
|
|
[project.optional-dependencies]
|
|
html = [
|
|
"selectolax>=0.3",
|
|
]
|
|
dev = [
|
|
"pytest>=8",
|
|
"aborist[html]",
|
|
]
|
|
|
|
[project.scripts]
|
|
aborist = "aborist.cli:main"
|
|
|
|
[tool.setuptools.packages.find]
|
|
where = ["."]
|
|
include = ["aborist*"]
|