The 3.15 GB-per-shard index I added in bc7efe4 bloated every live
shard 33% to optimize a transient dump-time operation. Wrong trade —
the shards exist for arborist's query/retrieval/falsification path,
not for cold-pack's convenience. fox: "wtf you added an index to
fucking make the shard bigger?"
Refactor: instead of building a permanent on-disk index to avoid the
sort, configure the dump connection to spill the sort to a temp file:
conn.execute("PRAGMA temp_store = FILE")
conn.execute("PRAGMA cache_size = -10000") # 10 MB
SQLite's sort still happens, but it spills to /tmp instead of growing
the process heap. cache_size caps the page cache the sort works
inside. No permanent disk cost; transient temp disk only during the
dump phase.
DROP INDEX idx_edges_dst_uri_type_anchor + VACUUM ran across the 4
live shards before this commit (out-of-band ops):
000.db: 12.48 → 8.75 GB (~3 GB reclaimed)
001.db: 12.43 → 8.71 GB
002.db: 12.53 → 8.78 GB
003.db: 12.51 → 8.76 GB
total: ~12.6 GB reclaimed across the corpus
For a freshly-restored shard from a metadata pack, the index never
gets created (it was only in _dump_edges_fan_in, not in SCHEMA_SQL),
so consumers get the lean version from day one.
33/33 cold + evict tests pass.
russell@unturf.com
315e783fa1