#53: pre-size chunks rows at INSERT to skip phase-2 page splits
Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).
Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.
Implementation:
arborist/cold_pack_metadata.py
_dump_generic_table for `chunks`:
Emit synthetic `_content_size` column = length(content) at the
end of the columnar header. Underscore prefix avoids collision
with any future schema column.
_restore_routed_table:
Detect `_content_size` in the JSONL header; if present (and
table == chunks), build INSERT against [chunks_cols] + ['content']
and substitute a bytes(_content_size) placeholder for the
content position. Phase 2 UPDATE later replaces those bytes.
Forward compatibility:
- Old packs (no _content_size): consumer uses today's NULL-content
INSERT path. No behavior change.
- New packs: consumer auto-detects, uses pre-sized path.
SPV-wallet trade-off:
In just-enough mode the consumer pulls only the metadata pack so
chunks land with the placeholder bytes (NOT NULL anymore). That's
a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
ever ships, it must distinguish placeholder bytes (where every
byte is 0) from real content.
New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
Hydrate just-enough → chunks rows have non-NULL bytes content of
correct size. Catches the regression if a future change reverts
the placeholder logic.
Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
This commit is contained in:
parent
db18172f9e
commit
e2bc7a926d
2 changed files with 125 additions and 6 deletions
|
|
@ -266,6 +266,65 @@ class TestFtsPackRoutingRegression:
|
|||
# lives in chunks_fts_docsize which is what we asserted above.
|
||||
|
||||
|
||||
class TestPreSizedChunks:
|
||||
"""#53: producer dumps a synthetic _content_size column for chunks;
|
||||
consumer INSERTs with right-sized BLOB placeholders so phase 2
|
||||
UPDATE replaces same-size content in place — no row growth, no
|
||||
page splits, no ext4 metadata-journal events per UPDATE.
|
||||
"""
|
||||
|
||||
def test_chunks_content_pre_sized_after_metadata_restore(
|
||||
self, producer_shard: Path, tmp_path: Path
|
||||
):
|
||||
from arborist.evict import push_pack
|
||||
backend = MemoryBackend()
|
||||
with sqlite3.connect(str(producer_shard)) as src_conn:
|
||||
src_conn.row_factory = sqlite3.Row
|
||||
push_result = push_pack(
|
||||
src_conn, backend,
|
||||
document_root=None,
|
||||
max_chunks=None,
|
||||
max_pack_bytes=10 * 1024 ** 2,
|
||||
allow_license_class="unknown",
|
||||
include_fts=True,
|
||||
)
|
||||
metadata_pack_hash = next(
|
||||
p["pack_hash"] for p in push_result["packs"]
|
||||
if p.get("kind") == "metadata"
|
||||
)
|
||||
|
||||
# Pull ONLY the metadata pack (just-enough hydration). Verify
|
||||
# chunks rows land with right-sized BLOB content (NOT NULL).
|
||||
target_dir = tmp_path / "tgt"
|
||||
target_dir.mkdir()
|
||||
M = 4
|
||||
targets = _make_target_shards(target_dir, M)
|
||||
try:
|
||||
hydrate_from_metadata_pack_routed(
|
||||
targets, backend, metadata_pack_hash, M=M, mode="just-enough",
|
||||
)
|
||||
finally:
|
||||
for t in targets:
|
||||
t.close()
|
||||
# Find the target with chunks rows; verify content is non-NULL
|
||||
# BLOB of correct size.
|
||||
for i in range(M):
|
||||
tdb = target_dir / f"{i:03d}.db"
|
||||
c = sqlite3.connect(f"file:{tdb}?mode=ro", uri=True)
|
||||
try:
|
||||
rows = c.execute(
|
||||
"SELECT chunk_id, length(content) AS sz, content IS NOT NULL AS has "
|
||||
"FROM chunks LIMIT 3"
|
||||
).fetchall()
|
||||
finally:
|
||||
c.close()
|
||||
if not rows:
|
||||
continue
|
||||
for cid, sz, has in rows:
|
||||
assert has, f"chunk {cid} on shard {i} has NULL content (#53 not applied)"
|
||||
assert sz > 0, f"chunk {cid} on shard {i} has zero-size placeholder"
|
||||
|
||||
|
||||
class TestRoutedHydrateValidation:
|
||||
def test_M_mismatch_rejected(self, tmp_path: Path):
|
||||
target_dir = tmp_path / "tgt"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue