#53: pre-size chunks rows at INSERT to skip phase-2 page splits

Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).

Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.

Implementation:

  arborist/cold_pack_metadata.py
    _dump_generic_table for `chunks`:
      Emit synthetic `_content_size` column = length(content) at the
      end of the columnar header. Underscore prefix avoids collision
      with any future schema column.

    _restore_routed_table:
      Detect `_content_size` in the JSONL header; if present (and
      table == chunks), build INSERT against [chunks_cols] + ['content']
      and substitute a bytes(_content_size) placeholder for the
      content position. Phase 2 UPDATE later replaces those bytes.

Forward compatibility:
  - Old packs (no _content_size): consumer uses today's NULL-content
    INSERT path. No behavior change.
  - New packs: consumer auto-detects, uses pre-sized path.

SPV-wallet trade-off:
  In just-enough mode the consumer pulls only the metadata pack so
  chunks land with the placeholder bytes (NOT NULL anymore). That's
  a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
  means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
  ever ships, it must distinguish placeholder bytes (where every
  byte is 0) from real content.

New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
  Hydrate just-enough → chunks rows have non-NULL bytes content of
  correct size. Catches the regression if a future change reverts
  the placeholder logic.

Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
This commit is contained in:
russell@unturf.com 2026-05-27 06:09:21 -04:00
parent db18172f9e
commit e2bc7a926d
No known key found for this signature in database
2 changed files with 125 additions and 6 deletions

View file

@ -266,6 +266,65 @@ class TestFtsPackRoutingRegression:
# lives in chunks_fts_docsize which is what we asserted above.
class TestPreSizedChunks:
"""#53: producer dumps a synthetic _content_size column for chunks;
consumer INSERTs with right-sized BLOB placeholders so phase 2
UPDATE replaces same-size content in place — no row growth, no
page splits, no ext4 metadata-journal events per UPDATE.
"""
def test_chunks_content_pre_sized_after_metadata_restore(
self, producer_shard: Path, tmp_path: Path
):
from arborist.evict import push_pack
backend = MemoryBackend()
with sqlite3.connect(str(producer_shard)) as src_conn:
src_conn.row_factory = sqlite3.Row
push_result = push_pack(
src_conn, backend,
document_root=None,
max_chunks=None,
max_pack_bytes=10 * 1024 ** 2,
allow_license_class="unknown",
include_fts=True,
)
metadata_pack_hash = next(
p["pack_hash"] for p in push_result["packs"]
if p.get("kind") == "metadata"
)
# Pull ONLY the metadata pack (just-enough hydration). Verify
# chunks rows land with right-sized BLOB content (NOT NULL).
target_dir = tmp_path / "tgt"
target_dir.mkdir()
M = 4
targets = _make_target_shards(target_dir, M)
try:
hydrate_from_metadata_pack_routed(
targets, backend, metadata_pack_hash, M=M, mode="just-enough",
)
finally:
for t in targets:
t.close()
# Find the target with chunks rows; verify content is non-NULL
# BLOB of correct size.
for i in range(M):
tdb = target_dir / f"{i:03d}.db"
c = sqlite3.connect(f"file:{tdb}?mode=ro", uri=True)
try:
rows = c.execute(
"SELECT chunk_id, length(content) AS sz, content IS NOT NULL AS has "
"FROM chunks LIMIT 3"
).fetchall()
finally:
c.close()
if not rows:
continue
for cid, sz, has in rows:
assert has, f"chunk {cid} on shard {i} has NULL content (#53 not applied)"
assert sz > 0, f"chunk {cid} on shard {i} has zero-size placeholder"
class TestRoutedHydrateValidation:
def test_M_mismatch_rejected(self, tmp_path: Path):
target_dir = tmp_path / "tgt"