russell@unturf.com
d25c0fe66f
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with identical document/edge counts; Merkle proofs round-trip 30/30): 1. zstd-compressed chunks.content (level 3). Magic-byte detection on read means legacy plaintext rows pass through unchanged. Cores stay plaintext so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working. 2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root, dst_uri, anchor) covers every column, so a default rowid-based table near-doubles row data in the PK index. WITHOUT ROWID makes the table itself the B-tree. Drops idx_edges_dst_uri too — the only query that filters on dst_uri alone is gravity_top_inbound, a one-shot analytic. 3. contentless FTS5 (content='', contentless_delete=1) eliminates the 28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored. chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5 rowid maps back to chunks.chunk_id at search time. Snippets are built in Python (search/fts5.py:_build_snippet) since SQL snippet() returns empty in contentless mode. TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match let "intelligence", "intellectual", "intellivision" drown real hits like Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the distinct query tokens — multi-token coverage outranks single-token title boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty "insufficient sources" reply. Schema-level changes affect new DBs only; existing v9.8 DBs keep working at the old layout. Cross-shard UNION views explicitly list the intersection of columns so a mixed cluster (legacy + new schema shards in one --shards-dir) still unions cleanly.
89 lines
3.3 KiB
Python
89 lines
3.3 KiB
Python
"""Transparent zstd compression for chunk content.
|
|
|
|
Backward-compatible: reads detect the zstd magic byte sequence
|
|
(\\x28\\xb5\\x2f\\xfd) and decompress; plaintext rows from before this
|
|
feature was introduced pass through unchanged.
|
|
|
|
Why zstd over zlib: ~3-5x compression on natural language vs zlib's
|
|
~2-3x, and zstd decompresses ~5-10x faster than zlib. Level 3 is the
|
|
default — higher levels (5-9) gain only ~5-10% extra ratio at
|
|
significant compression-time CPU cost. Decompression speed is
|
|
level-independent (~250 MB/s on modern x86).
|
|
|
|
Storage in SQLite: declared TEXT columns happily hold BLOB cells thanks
|
|
to SQLite's dynamic typing. We don't change the schema; the column type
|
|
discipline is enforced at the application layer through these helpers.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import zstandard
|
|
|
|
# Module-level singletons. The ZstdCompressor / ZstdDecompressor objects are
|
|
# stateless across calls — safe to share across threads in this codebase
|
|
# (we don't run multi-threaded ingests inside one process).
|
|
_COMPRESSOR = zstandard.ZstdCompressor(level=3)
|
|
_DECOMPRESSOR = zstandard.ZstdDecompressor()
|
|
|
|
# zstd frame magic (4 bytes). RFC 8478 §3.1.1.
|
|
_ZSTD_MAGIC = b"\x28\xb5\x2f\xfd"
|
|
|
|
# Below this UTF-8 byte length, compression overhead dominates and the
|
|
# compressed form is larger than the source. Empty / tiny chunks pass
|
|
# through as plaintext.
|
|
_MIN_COMPRESS_BYTES = 64
|
|
|
|
|
|
def is_compressed(value: object) -> bool:
|
|
"""True if value looks like a zstd frame (magic bytes match)."""
|
|
return isinstance(value, (bytes, bytearray, memoryview)) and bytes(value[:4]) == _ZSTD_MAGIC
|
|
|
|
|
|
def pack_chunk(text: str) -> bytes | str:
|
|
"""Compress text for storage in `chunks.content`.
|
|
|
|
Returns:
|
|
bytes — zstd-compressed UTF-8 if compression saves space.
|
|
str — original text, unmodified, when too small to benefit.
|
|
|
|
SQLite stores either as the column's natural cell type (TEXT for str,
|
|
BLOB for bytes); both round-trip correctly through the Python sqlite3
|
|
binding when read back.
|
|
"""
|
|
if not text:
|
|
return text
|
|
raw = text.encode("utf-8")
|
|
if len(raw) < _MIN_COMPRESS_BYTES:
|
|
return text
|
|
compressed = _COMPRESSOR.compress(raw)
|
|
# Defensive: if the entropy is near-incompressible (already-compressed
|
|
# data, very short repeats), keep the smaller representation.
|
|
if len(compressed) >= len(raw):
|
|
return text
|
|
return compressed
|
|
|
|
|
|
def unpack_chunk(value: object) -> str | None:
|
|
"""Decompress a chunk content value if compressed; otherwise return as-is.
|
|
|
|
Inputs:
|
|
None -> None (cold-tier or null content)
|
|
str -> str (legacy plaintext rows)
|
|
bytes-like -> str (decoded UTF-8; decompressed first if zstd-framed)
|
|
|
|
Raises TypeError on unexpected input shapes so a bad row surfaces loudly
|
|
instead of silently returning garbage.
|
|
"""
|
|
if value is None:
|
|
return None
|
|
if isinstance(value, str):
|
|
return value
|
|
if isinstance(value, (bytes, bytearray, memoryview)):
|
|
b = bytes(value)
|
|
if is_compressed(b):
|
|
return _DECOMPRESSOR.decompress(b).decode("utf-8")
|
|
# Legacy or non-compressed BLOB cell — try UTF-8 decode.
|
|
return b.decode("utf-8")
|
|
raise TypeError(
|
|
f"unexpected chunk content type: {type(value).__name__}"
|
|
)
|