Per fox 2026-06-01: same treatment as #000071 — replace the
review-archaeology structure with what we SHOULD grow. Ticket goes
from 978 lines (original Anchor6 §§1-8 design log + dav1d-review §0
retrofit) to 498 lines of directive spec. **Joseph6 stays as the
first registered example grammar** per fox's note — concrete enough
that an implementer sees what a WorldDimensionGrammar looks like
end-to-end, not abstract enough to lose its load-bearing role.
What changed in shape:
Before: §0 dav1d verdict retrofit + §§1-8 archaeology of the
original Anchor6 spec being reviewed (validate seed source,
segmentation method, mapper choice — all decisions long
since made).
After: §1-13 forward spec. Goal · Axis split · Hard constraints
(all phases) · AnchorN primitive · WorldDimensionGrammar ·
Quantization mappers (with uint256-H₁ + no-SO(3) corrections
documented inline) · π*_w_object canonicalizer with
four-identity-hash record · **Joseph6 as worked example** ·
Phase 1 deliverables (9 items) · Pre-review empirical
bench preserved as §7 · Phase 2/3 deferred · Open questions
(3 remaining; 5 closed by bench, 4 by dav1d's review) ·
Cross-references · Five-step alignment · One-line review
history at the bottom.
What changed in content: nothing material. The corrected spec from
the prior §0 retrofit IS the body now. The original Anchor6 design
log is no longer inlined — git history preserves it at commit
`862662b` (pre-rewrite tip); readers who want the rejection-by-
rejection detail go to
docs/dav1d-reviews/000070-spatial-anchor-pi-w-object--2026-06-01.txt.
Critical technical corrections preserved inline (not as "what was
fixed", but as the directive answer):
- §3.1: uint256 for H₁ position (octree depth >8 entropy
preservation)
- §3.2: rename `map_rotation_so3` → `map_rotation_euler_ypr` (no
SO(3) overclaim — quantized Euler is not SO(3) coverage)
- §4: WorldObjectRecord carries all four identity hashes
(grammar_hash, axiom_pack_hash, manifest_hash, seed_hash) for
replayability
- §5: Joseph6 ships as one example grammar; future grammars
register through the same mechanism
TICKETS.md index row also rewritten in directive voice.
doc_counts tests still pass.
Per fox 2026-06-01: replace the descriptive review-archaeology
structure with what we SHOULD grow. Ticket goes from 805 lines
(original §§1-8 design log + my dav1d-review §0 retrofit) to
389 lines of clean directive spec.
What changed in shape:
Before: §0 "Dav1d review verdict" decision-table + §§1-8
archaeology of the original Joseph6 spec being rejected.
After: §1-12 forward spec. Goal at the top, hard constraints,
composite ChainRoot identity, five bridge outcomes with
typed witness shapes, privacy class vocabulary, Phase 1
deliverables, Phase 2/3/4 roadmap, retro-validation
appendix, cross-references, three remaining open
questions, five-step alignment, one-line review history
at the bottom pointing at the archive file.
What changed in content: nothing material. The corrected spec from
the prior §0 retrofit IS the body now. The original Joseph6 design
log is no longer inlined — git history preserves it at commit
fadc50a; readers who want the rejection-by-rejection detail go to
docs/dav1d-reviews/000071-world-bridge-grammar--2026-06-01.txt.
TICKETS.md index row also rewritten in directive voice. Shorter,
less "what was wrong" + more "what to build."
Net effect: an implementer picking this up reads a forward-looking
ticket they can act on, not an archaeology of which framing was
rejected. The dav1d review history is one line at the bottom, not
the structural frame.
doc_counts tests still pass.
The 1026-line de-novo response arrives after #000070 was corrected
from fixed Anchor6 to generic AnchorN/WorldDimensionGrammar. The
bridge concept is valid and necessary; the Joseph6-coupled framing
is now wrong. Same pattern as #000070's review: §§1-8 preserved as
the design log of what was reviewed; new §0 carries the corrected
spec.
Verdict matrix (12 rows in §0 decision table):
✅ GO: bridge concept (Agreement / Translation / Embassy), Phase 1
doc-only, bridge_seed@v1 deferred to Phase 2, no audit_mode,
no SQL, bridge atlas Phase 3.
❌ REWRITE: #000070-as-Joseph6-sibling → AnchorN sibling;
chain_id=governance_policy_hash → composite ChainRoot
with optional v7-W fields (null sentinels for language-
only QA chains); single event_type='bridge' → typed
schemas per kind; privacy as Phase 3+ footnote → Phase 1
vocabulary (4 classes); #000059 "already-shipped"
overclaim → proposed/structurally-aligned unless repo
confirms.
❌ REJECT: doc-only Phase 1 flipping #000013 to "kernel_in_progress"
— correct status is "bridge_grammar_specified"; kernel_
in_progress is for actual kernel landings.
Five bridge outcomes (was three):
Agreement — grammars match on invariant set
Translation — hash-pinned adapter proves the mapping
Embassy — foreign object hosted with limited rights
Quarantine — NEW: bridge attempted, invariant validation FAILED,
record the rejection so future attempts see what broke
No-bridge — NEW: grammars genuinely incompatible, explicit
declaration that no bridge will exist
Composite ChainRoot identity:
chain_id = SHA256(canonical({
chain_id_version, history_root, governance_policy_hash,
canonicalization_version, schema_version, chunking_version,
world_manifest_hash, world_dimension_grammar_hash,
axiom_pack_hash, optional_verifier_policy_hash
}))
Language-only chains: three v7-W fields are null sentinels →
chain_id collapses to existing identity. v7-W chains: all mandatory.
Phase 1 deliverable (corrected):
1. Substrate-paper extension §"World-bridge grammar" (5 outcomes
with typed witness shapes + composite ChainRoot + privacy
vocabulary + sovereignty rule + #000070 cross-ref + retro-
validation appendix with corrected shipped/proposed framing)
2. Optional namespace stub arborist/world/bridge/__init__.py
(STATUS = "namespace_reserved")
3. NO kernels / NO bridge_seed canonicalization / NO SQL / NO
cache-key dimension / NO new audit_mode / NO verifier change /
NO KATs
4. #000013 status bumps to "bridge_grammar_specified", NOT
"kernel_in_progress"
Sovereignty rule survives unchanged (chain law local, bridge law
treaty-only, no bridge overwrites native chain law — guard against
forced-unity failure mode). Hash-pinned translators survive
unchanged. Topic-named under arborist/world/bridge/ unchanged.
Full review archived at:
docs/dav1d-reviews/000071-world-bridge-grammar--2026-06-01.txt
Folds dav1d's 1904-line review verdict into the ticket as §0 (new),
preserves §§1-8 as the design log of the original Anchor6 proposal
that was reviewed. Status flips from "open · awaiting dav1d review"
to "open · dav1d GO with rewrite · spec revision pending before any
kernel ships."
Verdict matrix:
✅ GO: generic AnchorN substrate primitive (not fixed Anchor6)
✅ GO: Joseph6 as the first registered default grammar (not THE
ontology)
✅ GO: deterministic object-state canonicalization
✅ GO: axiom/physics-loaded WorldDimensionGrammar as the scalable
path
❌ NO-GO: hard-coding H₁..H₆ as final ontology
❌ NO-GO: runtime LLM-decided dimensionality (axioms MAY propose,
only deterministic validators may accept; grammars must
freeze via grammar_hash before proof-path use)
❌ NO-GO: framing this as a "semantic verifier warrant"
❌ NO-GO: relation/event/place/agent_trace in this ticket
❌ NO-GO: SQL persistence at Phase 1
Critical technical corrections:
- H₁ MUST use uint256, not uint64 (octree position entropy at
depth >8)
- Rotation mapper is map_rotation_euler_ypr(), NOT "SO(3)"
(review §17 — quantized Euler triple, no continuous SO(3))
- Canonical record carries grammar_hash + axiom_pack_hash +
manifest_hash + seed_hash alongside the per-dimension values
- Missing privacy.class = HARD reject (no PUBLIC fallback)
- spatial-anchor-object@v1 folds into canonicalization_version
only — no new audit_mode token, no verifier_policy_hash bump
Corrected package layout:
arborist/substrate/spatial_anchor.py (AnchorN, split_anchor_n)
arborist/world/grammar.py NEW — WorldDimensionGrammar
arborist/world/pi_star/object.py (derive_world_object_record
+ 5 mappers)
arborist/pi_star/spatial_anchor_object.py NEW — registry adapter
bench/fixtures/spatial-anchor-object/known-answer-tests.jsonl
tests/test_spatial_anchor.py
tests/test_world_dimension_grammar.py
tests/test_pi_star_spatial_anchor_object.py
Implementation NOT started — arborist/world/__init__.py STATUS still
"namespace_reserved"; only bench/spatial_anchor_validation.py
(pre-review empirical bench, commit 55b651f) on disk. Review arrives
at the right moment: no production code committed against the original
Anchor6 spec yet, so the AnchorN reframe lands in the spec before
the wrong primitive ships.
Full review archived at:
docs/dav1d-reviews/000070-spatial-anchor-pi-w-object--2026-06-01.txt
Five of ten original open questions resolved by the pre-review bench
(Q1/Q2/Q3/Q8/Q9); review answers Q4/Q5/Q6/Q7; Q10 (paper-amendment
wording) is the only remaining open question — substrate paper text
needs revision to introduce AnchorN + grammar layer instead of fixed
Anchor6.
Bench-v5 (5 themes × 3 questions × 2 paths, Hermes-3-8B) AFTER the
fold-stack lift + apply_title_boost wire-up to use _title_query_tokens
shows the SAME 12 regressions as the pre-wire-up baseline:
- 5 wrong-primary picks (Dr Who, Albert/Ahmed/Alaric the third/first,
Casa Batlló error)
- 7 STRICT→HYBRID demotes on correct primaries (Spider-Man, Ampère,
Dr Syn, Dr V64, Finnish Defence Forces, Hopewell Centre, labor
economics)
Diagnosis: apply_title_boost only reranks docs ALREADY in the
candidate set. body-only retrieval (default with multi_route=False)
never surfaces "Doctor (Doctor Who)" so no fold-aware rerank can
promote it. Legacy surfaces it via the title route. FTS5 porter
stemmer handles plurals but NOT Dr→Doctor or third→III, so the
SqliteShardCorpus.fts_title method also doesn't help here without
the folds applied at retrieval-token-gen time, not rank time.
Updated ticket with the honest assessment: 5 Path A stages across
v1/v2/v3 have now proven legacy query() doesn't decompose into a
library of helpers. Three forward paths offered for fox to decide:
A. leave default at providence, accept fold regressions (env
escape hatch ARBORIST_LEGACY_QUERY=1 already shipped)
B. flip default back to legacy, treat providence as
infrastructure for cloud-query/corpus-query only
C. keep both alive long-term — separate query2 command
Two coupled changes that close most of the bench-driven regression
in #000072 PLUS fix the user-facing output regression from the
CLI default flip (492d1a8).
PART 1 — Lift fold-variants stack to _text_norm.py
=================================================
Moves _TITLE_TOKEN_RE, _TITLE_STOPWORDS, _TITLE_TOKEN_POLICY,
_NUM_ORD_TO_ROMAN, _NUM_ROMAN_TO_ORD, _HONOR_FOLD, _BRIT_FOLD,
_ascii_fold, _hyphen_fold_variants, _numeral_fold_variants,
_accent_fold_variants, _honorific_fold_variants,
_brit_fold_variants, _title_query_tokens from query.py:115-325
into arborist/qa/_text_norm.py.
query.py re-exports the symbols under their original names so
existing call sites + tests (test_accent_fold, test_numeral_fold,
test_query, test_claim_lattice) keep working.
apply_title_boost (in arborist/qa/corpus.py) NOW USES
_title_query_tokens for both query and title sides, replacing the
weaker tokenize_text + numeral_expand pair. Without this rewire
the lift was a no-op — the helpers were available but nothing on
providence_query's body-only default path called them. Bench
v3 (commit pending: 23-14-10Z.jsonl) confirmed the no-op state;
the v4 wire-up is what produces actual fold behavior.
source_roles.py + retrieval_routes.py + corpus_query.py drop their
lazy `from arborist.qa.query import _title_query_tokens` dance and
import directly from _text_norm — kills the lazy-import warts and
removes a circular-import hazard.
PART 2 — Restore providence_query output parity
================================================
Fox surfaced 2026-05-31: "the new providence query output with and
without json is worse it doesn't give timings anymore and a bunch
of other stuff." Comparing field surfaces, legacy emitted 38 keys,
providence_query 19. Half the renderer's lookup table was missing.
New on providence_query result (miss + hit branches both):
- prompt_chars: DICT with messages_total / system_prompt /
evidence_or_context / user_question / grounding_reminder
(not the bare int the first cut emitted — the renderer reads
.get('messages_total') etc.)
- answer_chars: int
- context_root: Merkle root over source document_roots (matches
legacy's _context_root helper)
- unverified_quotes, partially_verified_quotes,
warrant_proven_claim_idxs, format_collapsed: surfaced from the
verdict that run_query already computed but wasn't propagating
- timings: now emits BOTH float-second keys (search, context,
llm, verify, total + retrieval/prompt_build aliases) for bench
scripts AND int-millisecond keys (search_ms, context_ms,
llm_ms, verify_ms, cache_lookup_ms, persist_ms, total_ms) for
the renderer's _render_query_human lookup table
Still absent (require pre/post gates that providence_query doesn't
port yet — separate work): answerability, claim_cap_applied,
frame_detection, lazy_anchor_ratio, pointer_id_distribution,
preflight_hash, quantifier_intensity, quantifier_matched_token,
quantifier_explicit_count, question_state, retrieval_purity,
scope_bound_hint, soft_preflight_hint, repair_changes,
pre_repair_audit_mode, failure_stage.
Human render now shows: 5-line audit summary · source list · capacity
line · timings line · cache_key tail — same density legacy emits.
JSON consumers get 26 keys (was 19).
283 tests pass.
Bench note: a v4 fold-themes bench would now show whether the
apply_title_boost wire-up cleared the 12 regressions; killed mid-
run when this fix landed. Re-run when convenient.
Re-bench legacy vs providence_query on fold themes (accent, hyphen,
honorific, brit, numeral) after the proxy memory fix.
Result (15 question-pairs through Hermes-3-8B): 12 regressions,
2 improvements, 1 tie. Net-negative on these themes, BUT all 12
regressions trace to a single root cause — the 5 fold-variants
helpers (_hyphen, _numeral, _accent, _honorific, _brit) live inside
_title_query_tokens at query.py:288-325 and providence_query
lazy-imports the WRAPPER without lifting the fold helpers.
Same gap manifests two ways:
- Wrong primary (5): Dr Who → pathology; Albert/Ahmed/Alaric the
third/first → wrong articles; Casa Batlló → error
- STRICT → HYBRID on correct primary (7): the verifier's Rule 8
title-overlap check calls the SAME _title_query_tokens —
without folds, "Andre-Marie" (claim) and "André-Marie" (title)
are distinct tokens, overlap fails, audit_mode demotes
Path A v3 surfaces: lift the fold-variants stack to _text_norm.py,
re-export from query.py, drop the lazy-imports in source_roles.py +
retrieval_routes.py. ~250 LOC moved + ~50 LOC import-rewrites,
half-day. Lower risk than v1 (pure code motion, helpers are
identical between paths).
Themes deliberately skipped this round (need their own gates ported
separately): quantifier_subset, metacog_subset, warrant_chain_probe,
es, fr. Re-bench AFTER v3 lands.
Also commits bench/legacy_vs_providence_bench.py + the result JSONL
so the regression set is reproducible.
bench/chunk_fetch_speed.py — measure per-chunk fetch latency for the
two cloud paths a future JUST_ENOUGH=1 blob-publish move would
compare against: (1) apsw HttpRangeVFS on the big shard .db (current
FtsSidecarShardClient fallback path), (2) direct HTTP GET on a
same-bucket object (proxy for per-chunk blob fetch).
Measured 2026-05-31 against clones/full-bench/000.db (12.5 GB) +
clones/sidecars-fts/000.idx.db:
apsw median: 704 ms / chunk (mean 773, first 1594, warmup ~500)
blob median: 91 ms / chunk (mean 92, flat — no warmup effect)
speedup: 7.7×
Real-world: ~2.6 s saved per fresh 4-chunk query. For cache-miss
flows that already pay 5-15 s on the LLM call this is real but not
transformative. The big win for blobs is cache HITS that don't go
to LLM (returns drop from ~100 ms via cached chunks to sub-50 ms
via blobs) and bulk bench runs (400 fetches = 4 min vs 30 s).
Interactive single-question flow with LLM in the loop is fine on
the apsw path; blobs stay as future optimization, not blocker.
Also archives bench/three_way_results/*.jsonl (3 runs across the
#000072 Phase 1 progression) + bench/slim_fts_parity_results/ so
the journey from "cloud diverges from local" through "cloud matches
local 5/5" is on disk for the design-log record.
#000072 Phase 2 step 3. The user-facing 'arborist query' command now
routes through arborist.qa.providence_query (cache-aware run_query
wrapper) instead of the legacy 2000-line arborist.qa.query.query().
Legacy function stays alive in the module — other importers
(test_query.py, internal calls) keep working — but the CLI defaults
to the new orchestrator.
Escape hatch:
arborist query "..." --legacy
ARBORIST_LEGACY_QUERY=1 arborist query "..."
Either re-routes through the legacy retrieval+gate pipeline. Useful
when a themed bench subset regresses on the new path and operators
need fleet-wide fallback while the gap gets ported.
Why now: the legacy "dinosaur → Edwina" primary-source bug fixes
itself on the default path — slim FTS5 cloud parity already proved
the run_query orchestrator picks the canonical 'Dinosaur' article
on Q5. Smoke confirms it:
arborist query "why did the dinosaurs go extinct?" --dry-run
→ primary: Dinosaur (default — new)
arborist query "..." --legacy --dry-run
→ primary: Edwina, the Dinosaur Who Didn't Know She Was Extinct
Known gaps providence_query DOESN'T port today (legacy still has):
- pre-retrieval: canonical_projection, crosslang sandwich,
quantifier preflight, metacog, soft_preflight, frame_detection
- post-retrieval: answerability, repair, witness, sandwich edge-out
- merkle_proof column is "[]" placeholder; equivalence_class
fallback lookup omitted
- args legacy accepts that providence ignores: retrieval_keywords,
extra_body, translator, fidelity, over_fetch
Forcing-function-style rollout per fox 2026-05-31 — the bench
regression-finding IS the next signal. Themed subsets in
bench/qa_questions.txt that exercise the missing gates may regress;
those are the targets for the next round of porting.
CLI changes:
- new --legacy flag (with ARBORIST_LEGACY_QUERY=1 env equivalent)
- _cmd_query branches on use_legacy → legacy query() vs builds
Corpus + calls providence_query
- providence_query result.status mapped fresh_persisted/burned →
cache_miss_then_written so the bottom-of-function exit-code
check stays consistent
providence_query.providence_query cache_hit branch now surfaces
raw_answer, verifier_method, n_quotes, n_verified, unverified_quotes,
violations so the render layer + journal emitter don't crash on
None when serving from cache.
283 tests pass in the broader gate.
Phase 2 step 2 of #000072. The skeleton landed in 20faae0 looked up
providence_cache + returned on hit, but the miss branch just
returned the fresh run_query result without writing anything back —
so every call paid the LLM cost. Now misses persist:
1. Build run_dag via arborist.qa.dag.build_run_dag (claim-lattice
mode, 9-stage variant — same shape legacy query() emits)
2. Append a providence_write event via arborist.store.append_audit
INSIDE a BEGIN IMMEDIATE transaction
3. INSERT INTO providence_cache with all 25 columns the schema
requires (cache_key, source_root, document_uri, question_*,
answer_text, merkle_proof placeholder, 4-dim policy hashes,
3-dim schema versions, audit_event_hash linking to the just-
appended event, run_dag_root + run_dag_blob, audit_mode, etc.)
4. COMMIT — atomic; audit chain + providence_cache stay
consistent on crash mid-way
Result dict on miss now carries audit_event_hash + run_dag_root so
the caller can reference the audit chain or replay the DAG.
status="fresh_persisted" (was "fresh" in the skeleton) names the
new behavior.
NOT done yet (deferred to subsequent steps):
- merkle_proof is "[]" placeholder. Schema requires NOT NULL.
Real per-chunk proofs are a follow-up; the cache row is
consistent without them but downstream wallet verification has
nothing to walk.
- burn_existing doesn't emit a providence_burn audit event
(legacy query() does — query.py:3084 region). Audit chain
still grows monotonically on the persist side, just doesn't
record what was burned.
- equivalence_class fallback lookup (legacy tries both dedup-mode
keys when fidelity allows; primary only here)
Tests (tests/test_providence_query.py, 4):
- first call persists row + audit event
- second call returns cache_hit ignoring different stub
- burn_existing forces re-run + re-persist; audit chain grows
- chain links correctly across two distinct cache_keys
283 tests pass in the broader query/corpus/sidecar/wallet/bucket/
claim_lattice/byte_identity/providence gate.
Records the 2026-05-31 attempt at Path A (port the 5 reranks,
re-bench) and the result: smoke score went DOWN from 3/5 to 1/5
with reranks wired in. Root cause: legacy's rerank multipliers are
tuned against legacy's candidate-set shape (over_fetch=32, per-shard
parallel routes, body-density baked in earlier), not against my
multi_route fan-out's shape (per_route_limit=top_k*4, post-merge
candidates, filter-then-rerank instead of filter-during-route).
Helpers stayed in tree as importable building blocks (commit
d099995). Wire-up was reverted before commit so user surface is
unchanged.
Four v2 directions surfaced and documented for the future
investigation:
1. Match legacy's oversample factor (top_k*8+ or over_fetch=32)
2. Apply body-density filter BEFORE rerank cascade, at source
3. Rerun reranks on per-shard route output before final merge
4. Synonym expansion at retrieval time, not just filter time
None are blockers individually but each is a focused investigation.
The honest takeaway: legacy query()'s rerank pipeline is not a
"library of multipliers you compose in order" — it's a tightly
coupled cascade where each stage's tuning depends on what the
previous stages emitted. Collapsing it requires understanding
those couplings, not just lifting the helpers.
#000072 Path A first attempt: ported the 5 downstream reranks from
legacy query() into arborist.qa.retrieval_routes:
- body_density_passes / filter_by_body_density (Corpus.doc_body)
- rerank_by_source_role (SOURCE_ROLE_RANK_WEIGHTS)
- rerank_by_title_purity ((1+overlap)*(1+purity), shards_dir-gated
synonym_expand_strict)
- rerank_by_ordered_token_match (LCS over title tokens)
- rerank_by_body_coverage (sqrt body coverage, Corpus.doc_body)
Also unfreezes ``arborist.qa.corpus.Hit`` so the reranks can mutate
.score in place (matches legacy _Hit convention). ChunkRow stays
frozen (it's content-addressable evidence). test_hit_is_frozen test
renamed and inverted.
NOT WIRED INTO run_query: smoke probe with all 5 wired in legacy
order (filter → body_density → body_coverage → source_role →
title_purity → ordered_token → apply_title_boost) made the
multi_route regression WORSE:
pre-reranks: 3/5 correct (Soviet Union ✓, Mt Kilimanjaro ✓,
Mona Lisa ✓, Mercury Seven ✗,
dinosaurs ✗)
post-reranks: 1/5 correct (Soviet Union ✗ → "national bandy team",
Mt Kilimanjaro ✓,
Mona Lisa ✗ → "Painting Mona Lisa",
Mercury Seven ✗ → "305th Air Mobility
Wing", dinosaurs ✗)
Root cause: legacy's reranks were tuned against legacy's
candidate-set shape (multi-shard parallel _search_corpus with
body-density baked in EARLIER, over_fetch larger than the per_route
limit I'm using, and a different rivalry-exclusion order). Applying
the same multipliers to my multi_route fan-out's candidate set
lands the cascade in a different basin — short noisy titles with
high stem-overlap get amplified into rank-1 territory.
The helpers stay in tree as importable building blocks for a future
Path A v2 attempt. Possible v2 directions: (a) match legacy's
oversample factor (32+ vs my 4×top_k); (b) apply body-density
filter BEFORE rerank cascade (legacy does this earlier in
_search_corpus); (c) rerun against per-shard route output instead
of post-merge candidates so per-shard discrimination survives.
policy=None / experimental multi_route=True paths unchanged in
behavior — multi_route is still strictly worse than body-only
(documented in #000072) but no longer worse than itself with
reranks; reranks aren't auto-applied.
264 tests pass.
Documents the work shipped this session (Phase 1 foundation, 10
commits 9ba6317..20faae0) and the blocker that stops Phase 2: the
multi_route pipeline regresses on 2/5 smoke fixture questions
(Mercury Seven → Sam T. Beddingfield; dinosaurs extinct →
Paul Austin Kelly) because Phase 1 hasn't ported the 5 downstream
rerank stages legacy query() uses to suppress noisy phrase-route
hits — body-density check, body-coverage sqrt rerank, source-role
rerank, title-purity rerank, ordered-token-match rerank.
Path B chosen 2026-05-31: stop here, leave multi_route off by
default (which IS default — policy=None preserves body-only
behavior). Phase 1 foundation stays in tree as future-ready
infrastructure; user surface unchanged.
Path A (port the 5 stages, multi-day effort) reserved for a
future focused session. Until then, legacy query() keeps
producing the same answers it always has, INCLUDING the wrong
"Edwina" pick on dinosaur Q5. The slim-FTS5 cloud path already
fixes that bug for `arborist cloud query` / `arborist
corpus-query` callers (proven 5/5 source parity in d9fb6a9).
Bumps Next ID 000072 → 000073.
The cache wrapper Phase 2 needs to start collapsing legacy query()
into a thin adapter. Minimal-viable shape:
providence_query(corpus, question, chat_client, *, qa_db, policy,
model_id, burn_existing, top_k, ...) → dict
- Computes 8-dim cache_key from question + policy + model + source_root
+ canonical messages (system + user with EVIDENCE/QUESTION/grounding
reminder — same shape as run_query, byte-identity gate from step 5
catches drift)
- Optional burn_existing: deletes the live cache row first
- Looks up providence_cache: hit returns cached row + bumps hit_count
- Miss: calls run_query(corpus, question, chat_client, policy=policy)
and returns the result with cache metadata
NOT done yet (deferred to subsequent Phase 2 sub-steps):
- Cache PERSIST on miss — run_query produces audit_mode + sources but
legacy providence_cache schema wants run_dag_root, context_root,
prompt_hash, verifier_method, n_quotes/n_verified, violations_json,
etc. that aren't on run_result yet. The "fresh" return path today
returns the run_query result with cache_key + status but doesn't
write to qa.db. Persist needs lifting from query.py:3746 lines.
- equivalence_class fallback lookup (legacy tries both dedup-mode
keys when fidelity allows; minimal path checks only primary)
- Pre-retrieval gates (canonical_projection, crosslang, quantifier,
metacog, soft_preflight, frame_detection) — they stay in legacy
query() for now
- Post-retrieval add-ons (answerability, repair, witness, sandwich
edge-out) — same
- retrieval_keywords, translator, extra_body — legacy-only knobs
Legacy arborist.qa.query.query() is UNTOUCHED — still the user-facing
entrypoint. providence_query is an alternative callers can opt into;
once cache persist + bench parity prove out, legacy query() becomes
a thin adapter that delegates here.
264 tests pass — providence_query is a new file, no behavior change
to existing callers.
Closes Phase 1 step 6 of #53. When the caller passes
policy={"base_version": "wikitext-base-v1"} (or any truthy value),
run_query now invokes arborist.wikitext.to_base() on each chunk
span BEFORE truncating to per_doc_budget, so a 24 kB raw-template
span gets compressed first and the LLM/verifier see the same
prose. Same shape legacy query() applies at three sites.
policy=None / policy without base_version: no-op, preserves the
byte-identity gate. mwparserfromhell missing (the optional dep):
import fails, no-op — already the legacy convention.
policy["base_version"] folds into governance_policy_hash via the
caller's policy dict — opting in DOES rotate cache_key for that
policy partition, which is the right semantics (different prose →
different cache).
264 tests still pass. Phase 1 substantively complete; Phase 2
(providence_query.py + legacy query() collapse) starts next.
When policy.multi_route fans across body+title+phrase+core_keyword
and merges by MIN bm25, the resulting set over-recalls on noisy
titles (e.g. "East Asia" outranks "Nineteen Eighty-Four" on the
Orwell phrase query because raw bm25 over the merged set doesn't
know about accept-path semantics). The 5-accept-path filter
arborist.qa.retrieval_routes.filter_by_title_relevance (landed
step 3) gates the merged set on: title-overlap / synonym / TFIDF-core
match / verbatim-phrase match / hyphen-fold anchor.
Wired so accept-paths 2 and 4 actually fire — core_match_roots
and phrase_match_roots are passed through from this turn's
core_hits / phrase_hits, so docs surfaced via those routes pass
the filter even when titles miss.
Smoke (multi-shard wiki corpus, "has oceania always been at war
with east asia"): Nineteen Eighty-Four now ranks #1 (was rank 3
before filter). Other phrase-route hits like "Nineteen Eighty-Four
in popular media" stay; the geographic "East Asia" / "Oceania"
title hits get filtered out.
264 tests pass (was 263 + 1 multi-route test from 6c that confirms
filter path doesn't blow up on the Anarchism fixture either).
When the caller passes policy={"multi_route": True}, run_query now
fans out across four retrieval routes in parallel and merges:
1. fts_body — body BM25 (the only route in pre-6c)
2. fts_title — title-only BM25 via documents_fts
3. fts_phrase — verbatim 4-gram phrase MATCH; closes the
allusion gap ("always been at war" → 1984)
4. core_keyword_match — TF-IDF core route for neologisms
Each route is fail-open: NotSupportedError → []. core_keyword
returns [] on SidecarBucketCorpus (no derivations in slim sidecar);
phrase / title / body all work cloud-side via the slim FTS5 sidecar.
Merge by MIN bm25 per document_root (FTS5 returns negative; lower
wins). core_keyword's positive match_count scores are kept only as
a tiebreaker when no FTS5 route surfaced that doc — handled
explicitly via score-sign discrimination since the scales are
incomparable.
Lifted helper: question_phrases(question, n=4) from query.py's
_question_phrases into arborist.qa.retrieval_routes — pure-stdlib
n-token window extractor, no stopword stripping (the diagnostic
signal IS the stopword).
policy=None / policy={} (no multi_route flag) keep the existing
body-only path — byte-identity gate from step 5 still green. 263
existing tests + 1 new multi-route test pass.
Still missing for full legacy parity: filter_by_title_relevance
integration in run_query (the 5-accept-path filter is available in
retrieval_routes.py since step 3 but isn't wired into the
orchestrator yet). That's the next sub-step — without it, the
multi-route merge over-recalls on noisy title overlaps.
New bench/teacher_judge.py reads judge_input.jsonl, cross-judges
each (question, model, answer) using a DIFFERENT model than the
answerer (default: hermes-answers → qwen judges, qwen-answers →
hermes judges). Writes verdicts.jsonl + markdown summary including
a false-STRICT highlight section.
Key implementation details:
- Reuses _judge_prompt from cross_model_selfplay so iterating on
the prompt template doesn't require re-running benches.
- Sends chat_template_kwargs:{enable_thinking:false} on every call
— required for qwen.ai.unturf.com (llama.cpp deepseek-reasoning
format) where reasoning eats max_tokens before producing content.
vLLM (hermes) silently ignores the unknown kwarg.
- Prompt template tightened: previously said "Reply in format
VERDICT: <rationale>" which qwen took literally, replying with
the word VERDICT instead of the verdict token. Now explicit:
"write the chosen verdict word itself".
Live cross-judge result on 151-row 76-question bench:
hermes (judged by qwen): 44 CORRECT, 23 WRONG, 9 PARTIAL → 58% accuracy
qwen (judged by hermes): 45 CORRECT, 6 WRONG, 11 PARTIAL → 73% accuracy
Qwen is materially more accurate despite costing 1.8× more — matches
fox's "qwen slightly outperforms" intuition with hard numbers.
8 false-STRICTs surfaced including Q12 (hermes conflated Roman
Empire with HRE), Q48 (hermes answered with Niger River info on a
Nile question), and Q72 (both models missed the Game of Thrones
reference in "winter is coming").
When policy is provided (any non-None dict), the orchestrator now:
1. Classifies each hit's source_role via
arborist.qa.source_roles.classify_source_role(title, qtokens_stem,
document_uri). Same heuristic legacy query() uses — noisy/sequel/
secondary markers fire first, then breadth-of-title-stem-coverage
decides primary vs background.
2. Splits max_context_chars by SOURCE_ROLE_BUDGET_WEIGHTS
proportionally (primary 2.0, noisy/sequel 0.5, others 1.0).
Primary answer source claims ~2× the slice — same shape as
legacy query()'s per-source cap.
policy=None preserves the pre-step-6b shape EXACTLY: rank-based
roles (rank 1 = primary, else background), flat
`max_context_chars / len(hits)` budget. Byte-identity gate from
step 5 remains green.
Tested: policy={} on the Anarchism single-token fixture classifies
the top hit as primary_answer_source (matches legacy). Existing
259 tests + new role-class test = 262 in the gate.
Phase 1 step 6a of #53. Smallest additive change to set up Phase 2's
cache wrapper. Existing callers (corpus-query, cloud query) pass no
policy and behave IDENTICALLY to before — byte-identity gate from
step 5 stays green.
When policy IS provided, twelve recognized verifier kwargs forward
through to verify_claim_lattice:
allowed_source_roles, max_pointers_per_claim, min_citation_coverage,
min_claim_content_tokens, lazy_anchor_demote_threshold,
lazy_anchor_demote_min_pairs, max_claims_per_answer,
subject_tokens_absent_threshold, warrant_check_enabled,
deflection_check_enabled, format_collapse_check_enabled,
warrant_chain_roots
UNKNOWN keys are silently ignored — a policy dict shared with legacy
query() may carry fields (base_version, retrieval_keywords, etc.) the
orchestrator doesn't yet honor; ignoring them keeps the call site
clean instead of requiring callers to filter.
Steps 6b/c/d extend the policy surface:
6b: role-classified + role-weighted context budget
6c: multi-route retrieval (title + phrase + core_keyword + body merge)
6d: wikitext-strip
Tests cover three contracts:
- policy=None is byte-identical to pre-6a (262 tests including the
step-5 byte-identity fixture pass)
- policy={"max_claims_per_answer": 0} actually trips TOO_MANY_CLAIMS
(proves the kwarg reaches the verifier, not just the function
signature)
- policy with unknown keys (e.g. base_version) doesn't blow up
Phase 1 step 5 of #53 — the load-bearing piece. Freezes the cache-
identity byte shape today so the run_query rewrite landing in Phase 2
can be checked against it.
What's pinned (tests/fixtures/byte_identity/claim_lattice.json):
- SHA-256 of CLAIM_LATTICE_SYSTEM_PROMPT (drift = every cache rotates)
- SHA-256 of CLAIM_LATTICE_GROUNDING_REMINDER (same)
- Per-question question_hash (strict + equivalence_class modes)
- Per-question conversation_hash on the synthetic 2-message array
[system + user(EVIDENCE+QUESTION+grounding_reminder)] — the EXACT
shape arborist/qa/corpus_query.py:run_query builds
- governance_policy_hash on three reference policy shapes
- model_profile_hash for hermes / qwen / stub
Plus a determinism sanity test that pins the algos themselves
(SHA-256, _canonical_json key-sorting, dedup-mode question canonical).
Risk class addressed (Plan §6 risks #1+#2): conversation_hash takes
the FULL OpenAI messages array. Any drift — message reorder, whitespace
shift, optional message gated on a different condition — rotates every
cache_key in the world and orphans every providence_cache record on
re-lookup. Same for governance_policy_hash on the policy dict (a new
field rotates everything). The fixture catches a drift the SECOND it
happens, with a diff-style failure naming the path that drifted.
Re-capture mode: `CAPTURE=1 pytest tests/test_run_query_byte_identity.py`
rewrites the fixture. Only do this on deliberate prompt-shape or
policy-shape changes that are treated as cache-invalidation events.
Phase 1 step 4 of #53. Two new protocol methods that the unified
run_query needs but the protocol didn't expose:
- core_keyword_match(qtokens, *, limit) → list[Hit]
TF-IDF core-keyword route. Finds source docs whose distilled
tfidf-core content contains any of qtokens. Closes the neologism
gap (e.g. "permacomputer" matching a Grok conversation about it
via its TF-IDF core, even though permacomputer never appears in
a title). Returns Hits whose .score is the integer match_count
(also in .extras["match_count"]) — UNIQUE among routes in being
higher-is-better, not bm25 lower-is-better.
- doc_body(document_root) → str | None
Concatenated chunk text for one document. Used by the body-
coverage rerank stage that needs the full body (not just top-K
chunks) to decide whether the doc actually discusses qtokens.
Implementations:
- SqliteShardCorpus: full SQL, lifted verbatim from query.py's
_docs_with_core_keyword_match (same word-boundary LIKE +
match_count tallying). Gracefully returns [] if derivations
table missing (older shard).
- MultiShardSqliteCorpus: per-shard fan-out, dedupe by doc_root
keeping MAX match_count across shards (HIGHER wins for this
route).
- SidecarBucketCorpus: core_keyword_match raises NotSupportedError
(the derivations table that maps tfidf-core → source isn't in
the slim FTS5 sidecar). doc_body concatenates chunks_for_doc
output (works on bucket via blob fallback).
Smoke: on the genesis wikipedia shards core_keyword_match returns
[] cleanly — those shards have 92 claim_pack derivations but 0
tfidf-core derivations, so the route correctly produces nothing.
Validation: 257 tests in the query/corpus/sidecar/wallet/bucket/
claim_lattice gate pass.
Phase 1 step 3 of #53. Lift the title-relevance filter to its own
module so run_query can call the same filter the legacy query() uses.
Why this one first: it's the load-bearing piece of the 9-stage
retrieval pipeline that closes the verifier-fabrication gap on
synonym / TFIDF-core / phrase-route / hyphen-fold hits. The other
six rerankers (_rerank_by_title, _rerank_by_source_role,
_rerank_by_title_purity, _rerank_by_ordered_token_match,
_rerank_by_body_coverage, _ordered_match_length) can move
independently when run_query needs them; they aren't blockers.
Adaptation from legacy _Hit to a duck-typed Hit: the function now
reads .title + .document_root via getattr, so both the legacy
arborist.qa.query._Hit dataclass AND the protocol Hit from
arborist.qa.corpus satisfy it without a type bridge.
Lazy imports for _title_query_tokens (full fold-variants stack still
in query.py) and the concepts module (synonym_expand /
rivalry_excluded / has_compare_phrasing) avoid an import cycle and
defer cold-start cost.
Validation: 257 query/corpus/sidecar/wallet/bucket/claim_lattice
tests pass. No behavior change — query.py re-exports under the
same underscore name (_filter_by_title_relevance) so existing
call sites are byte-identical.
Phase 1 step 2 of #53. Hoist the trailing-s stemmer to a single home
in arborist.qa._text_norm so query.py, corpus.py, and source_roles.py
all use the same implementation.
Before:
- query.py:_stem_token_for_match — `len>4 and endswith('s') and not
endswith('ss')` (apostrophes assumed pre-stripped by _TITLE_TOKEN_RE)
- corpus.py:apply_title_boost._stem (inline) — same length/suffix
check PLUS apostrophe strip ("'", "’")
Behaviorally compatible when inputs are pre-stripped, but the dual
implementations were a drift risk waiting to bite. The apostrophe-safe
version (corpus.py's) is the canonical now — handles raw title text
without an upstream sanitizer, no behavior change for the pre-stripped
call sites.
Re-exports in query.py + import-update in corpus.py + source_roles.py
keep every existing caller working. 257 query/corpus/sidecar/wallet/
bucket/claim_lattice tests pass.
Defers lifting _title_query_tokens (and its 5-fold variant helpers —
hyphen, numeral, accent, honorific, brit) for later: those carry
years of bench-tuned hot-loop optimization and a TITLE_TOKEN_POLICY
slug threaded into run-DAG provenance. source_roles still lazy-imports
_title_query_tokens from query.py; no change there.
Phase 1 step 1 of #53 (collapse legacy 2000-line query() into the
unified run_query orchestrator). Pure code motion — no behavior
change, no signature change. Lifts:
- SOURCE_ROLE_BUDGET_WEIGHTS / SOURCE_ROLE_RANK_WEIGHTS
- _NOISY/_SEQUEL/_SECONDARY title-marker tuples
- _classify_source_role function
from query.py:781-854,1579-1591 to a new arborist/qa/source_roles.py
module. query.py re-exports under the old names so existing imports
(and the 45-test test_query.py suite) keep working.
Goal: run_query needs the source-role classifier to do role-weighted
context-budget splits (Plan §5 Phase 1 deliverable 1). Until this
file exists, run_query has no import path to it that doesn't pull in
the full 4280-line query.py module. Lazy-imports `_title_query_tokens`
and `_stem_token_for_match` from query.py for now — those move to
_text_norm.py in a later Phase 1 step.
Validation: 45 tests in test_query.py pass; 257 tests in the broader
query/corpus/sidecar/wallet/bucket/claim_lattice gate pass.
STRICT audit ≠ factually correct. The verifier passes any answer whose
quotes match source text; a model can quote correctly and still draw a
wrong conclusion. Without a judge, the bench can't detect false-STRICTs.
This flag emits <ts>.judge_input.jsonl alongside the regular results —
one row per (question, model) with a model-agnostic teacher prompt.
Feed to any teacher (claude -p per row, remote API, Hermes self-judge)
to get CORRECT|WRONG|PARTIAL|UNCERTAIN verdicts. Skips rows that
errored or returned empty answers.
Grounded in observed reality from the 76-question 2010-wiki bench:
Hermes produced 2 confidently-wrong STRICTs (Q12 conflated Roman Empire
with Holy Roman Empire; Q48 answered the Niger River when asked about
the Nile). Qwen had 0 false STRICTs over the same fixture. Naming the
flag --teacher-model-judge (not --opus-judge) keeps the harness
provider-agnostic.
The three "92 claim_pack docs" tags were drifting against shard 000.db's
21 rows because the harness only counted one shard, but the doc prose
("#000031 closed at 92") meant the corpus total (21+16+38+17 across
genesis shards 000-003).
Two-line fix path: extend the harness to sum across all ???.db shards
via a `*:` prefix (e.g. `*:documents?source_type=claim_pack`), then
prefix the three drifted tags. Aligns the harness scope with the
semantic scope of the claim instead of forcing the claim to shrink to
one shard.
The `*:` glob:
- Matches `[0-9][0-9][0-9].db` basenames only (operator sidecars
qa.db / snapshots.db / selfmodel-chain.db skipped)
- Skips shards lacking the named table (schema-version tolerance)
- Returns _DB_MISSING when no genesis shard exists (CI / fresh-
checkout skip semantic preserved)
- Returns _TABLE_MISSING when no contributing shard has the table
Documented in ticket-000044 §3.4 + a third example showing the new
syntax. Diagnosis credit to a sub-agent investigation that confirmed
zero eviction/falsification audit events on claim_packs — the data is
intact; the harness was just single-shard.
Completes the Corpus protocol surface. Both routes already worked at the
storage layer (every shard has chunks_fts AND documents_fts virtual
tables; FTS5 supports MATCH '"phrase"' natively); they just weren't
plumbed through. Now SqliteShardCorpus, MultiShardSqliteCorpus,
BucketClient, FtsSidecarShardClient, MultiShardSidecarCorpus, and
SidecarBucketCorpus all expose fts_title + fts_phrase end-to-end.
Bonus fix in arborist/ingest.py: every shard's INSERT INTO documents
now also INSERTs into documents_fts in the same transaction. Without
this, fts_title returned empty on freshly-ingested shards — only
migrated genesis shards had documents_fts populated. The cost is one
FTS5 row per new doc, negligible vs the chunk inserts.
Also includes operational cleanup (E):
- Deleted s3://arborist/clones/manifest-sidecar.json
- Deleted s3://arborist/clones/full-bench-64k/00[0-3].sidecar.bin
(~3.1 GB reclaimed; the slim FTS5 manifest is now the only cloud
surface)
- Fixed MultiShardSidecarCorpus.stats() — was calling .bucket on
FtsSidecarShardClient (attribute went away when SidecarShardClient
was deleted); now calls .stats() directly on whichever client.
Validation
----------
- Local fts_title("dinosaur"): "Dinosaur" main article ranks #1
- Local fts_phrase(["always been at war"]): "Nineteen Eighty-Four" ranks
#1 (the canonical phrase-route test from CLAUDE.md)
- Cloud (slim FTS5 sidecar) fts_title / fts_phrase: IDENTICAL ranking to
local on the same query (bit-for-bit FTS5 parity preserved)
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed (the same two
pre-existing failures from main HEAD)
Sibling to #000070 on the second axis of Joseph (@TrudoJo)'s spec.
Joseph posted the bridge-framework extension 6h after the 6-dim
spatial-ontology: chain A has its own world rules, chain B has its
own world rules, a bridge exists only where a deterministic
agreement layer can be proven. Three kinds: Agreement, Translation,
Embassy.
Phase 1 is doc-only -- the primary contribution is a retroactive-
validation appendix naming three already-shipped arborist primitives
as Joseph-bridge-shape patterns one-for-one:
#000066 graft-receipt mode = Agreement bridge
#000056 Operation Sandwich = Translation bridge (opus-mt hash
IS the translator_id)
#000059 graveyard (proposed) = Embassy bridge (POINTER-LINKED,
banner-labelled, never promotes)
The framework is not a new requirement on arborist. It is a unifying
spec that names what we already do in 3+ places, plus surfaces a
fourth pattern (bridge atlas for multi-instance federation with
potentially-different governance_policy_hash peers) as genuinely new
work for arborist/mesh/.
Bundle:
- docs/tickets/ticket-000071-world-bridge-grammar.md (~440 lines)
- docs/TICKETS.md index row + Next ID 000071 -> 000072
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md §6 gains
one-line cross-reference to #000071 as adjacent axis
- Joseph memory extended to cover both frameworks
Phase 2/3/4 deferred to separate tickets. Hard constraints: A1-A3,
no audit_mode token, no SQL change at Phase 1, sovereignty rule.
Status: open, awaiting dav1d Phase 1 review + fox go/no-go.
Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690
LOC) with a slim SQLite file that just COPIES the source shard's FTS5
shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval
then runs SQLite FTS5 bm25() on the same bytes the local shard uses —
bit-for-bit parity by construction. 5/5 source + audit_mode agreement
on the smoke fixture between local corpus-query and cloud-query against
the new manifest-fts.json.
Why
---
Custom binary sidecar was per-document BM25; main encyclopedia articles
got length-normalized so hard that on "why did the dinosaurs go extinct?"
"Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur"
(measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so
each chunk is a moderate-length doc and the main article wins multiply.
Different granularity, not a tuning knob — fix is to use the same
indexer cloud-side.
What ships
----------
- arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard,
copy documents (root/uri/title only), copy chunks (id/root/idx/leaf
only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with
same DDL as source, bulk-copy the four shadow tables verbatim,
VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full
4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min.
- arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim
sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens
read-only sqlite3 (check_same_thread=False for parallel shard fan-
out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash>
with HTTP-range big-shard fallback when blobs aren't published.
MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct
(both are FTS5 backends; merge by raw bm25 MIN ascending).
- arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False
(FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on
fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost
imports tokenizer helpers from new arborist/qa/_text_norm.py.
- arborist/qa/_text_norm.py — fold_accents, numeral_expand,
tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so
apply_title_boost keeps its lexical shape.
- arborist/cli.py: `arborist sidecar build-fts` subcommand; old
`sidecar build`/`sidecar search` removed. cloud_query recognizes
fts_sidecar_url + sidecar_url alike.
- Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets;
`sidecar-build` + `sidecar-search` removed.
- scripts/upload_fts_sidecars.py — boto3 producer: uploads slim
sidecars to clones/sidecars-fts/<n>.idx.db, publishes
clones/manifest-fts.json (4 wikipedia shards inherit existing
shard_url for content fallback; ACL public-read; idempotent on
size match). Existing manifest-sidecar.json untouched.
- tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py
converted from build_sidecar → build_fts_sidecar; 6 fixtures pass.
- bench/slim_fts_parity_bench.py — local 3-way bench
(legacy/corpus/slim_fts) over the smoke fixture.
Validation
----------
- Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25
scores to the source shard for top-10 of "dinosaurs extinct".
- 3-way bench (legacy local / corpus local / slim-FTS5 over real
bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement
between corpus and slim_fts. Q5 legacy disagreement (Edwina vs
Dinosaur) is the pre-existing 2000-line query() retrieval quirk,
unrelated.
- End-to-end cloud query against published manifest-fts.json (cold-
start, ~149 s sidecar download once): STRICT · Dinosaur, every
quote verified (2/2).
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre-
existing failure (tests/test_doc_counts.py — claim_pack docs row-
count drift) and one pre-existing cold_object failure, both reproduce
on main HEAD.
Bucket state
------------
- s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new
- s3://arborist/clones/manifest-fts.json — new
- s3://arborist/clones/manifest-sidecar.json — kept live (deprecated
but still readable; downstream callers should switch to
manifest-fts.json)
Updates the spatial-anchor ticket so the pre-review empirical bench
(landed in 55b651f) reads as a self-contained dav1d review packet:
- Status line: notes bench shipped in 55b651f
- §7a Pre-review empirical evidence: expanded with reproducibility
notes (RNG seed 0xa8c90e551fd34427, wall ~2s, stdlib-only), the
five bench purposes mapped to open questions, full headline-results
table, and per-headline "what each one buys dav1d" prose for Q1
through Q9.
- §3 Land sequence: step 1 marked done in 55b651f with explicit
pre-review-bench scope; step 2 annotates that 5 of 10 open
questions now resolve by measurement.
- docs/TICKETS.md index row: leading status cell now opens with the
bench headline + per-section numbers so the index reads as the
dav1d handoff summary, not just "open · awaiting review".
No new code, no new commits to the bundle itself -- 55b651f remains
the bench's shipping commit. Pure documentation refresh.
New ticket for Joseph (@TrudoJo)'s 6-dim procedural spatial-anchor
framework as the first verifier kernel under the #000013 v7-W reserved
namespace. A single committed 32-byte SHA-256 hash deterministically
expands into six 32-byte regions H1..H6 via the HMAC-SHA-512 KDF already
shipped in arborist/substrate/anchor_prg.py (#000035); each region drives
one quantized object dimension under a fixed canonical mapper. Domain
separation from #000035 by dedicated spatial_anchor_seed published in
the v7-W manifest -- preserves #000035's KAT freeze + dav1d 2026-05-11
final review intact.
Bundle:
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md (718 lines):
full spec with 8 design-choice subsections, working Python sketch,
12-NOT scope boundaries, 8-row cross-references, five-step deletions,
10 open questions for dav1d.
- bench/spatial_anchor_validation.py: pure-stdlib pre-review evidence
(~2s, RNG-seed-pinned, reproducible). Five benches: avalanche,
cell-distribution uniformity, collision vs birthday-bound, cross-region
independence, domain separation.
- bench/spatial_anchor_validation_results.md: report from first run.
- Makefile: 'make bench-spatial-anchor [SPATIAL_N=N]' target + PHONY.
- docs/TICKETS.md: index row + Next ID 000070 -> 000071.
Bench headlines (N=10000):
- Avalanche mean 767.85 bits (PRF null 768, z=-0.49) -> PASS
- Cell-distribution chi^2 |z|<1 at L=2,3,4 -> PASS
- Birthday-bound ratio obs/exp 0.989/1.038 at populated L -> PASS
- Cross-region Pearson all 15 pairs < 2sigma -> PASS
- Domain separation Arm A 767.91 / Arm B exact 0 collision -> PASS
Five of dav1d's ten open questions (Q1 seed source, Q2 segmentation,
Q3 position mapper, Q8 endianness, Q9 KAT adversarial vectors) now
resolve with measurements rather than appeals to PRF authority.
Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions.
Status: open, awaiting dav1d review + fox go/no-go. No registry slot
booked, no substrate-paper amendment landed, no kernel module created.
The Hank-Scorpio-vs-Mr.-Burns problem: body BM25 alone outranks the
canonical primary-source article ("Homer Simpson") below sibling
articles that incidentally mention the same tokens. The sidecar
pipeline already had title-boost-with-extras-penalty; lift it into
a shared utility that the orchestrator applies to ANY adapter's
fts_body output.
arborist/qa/corpus.py:apply_title_boost(hits, query, *,
higher_is_better)
Stems both sides (possessive + plural collapse), numeral-expands
(7↔VII), accent-folds (é→e). Effective bonus per hit:
max(0, overlap - extras/2) * boost
where extras = title tokens NOT in query. Score direction honors
each adapter's convention (BM25 negative → subtract; sidecar BM25
positive → add).
arborist/qa/corpus_query.py:run_query
Now oversamples fts_body by 4× and reranks via apply_title_boost
before slicing to top_k. Without oversampling the right primary
can sit at rank 7-15 in the body BM25 output and get cut before
the rerank sees it.
bench/three_way_bench.py: NEW — drives same fixture through
- `arborist query` (legacy 2000-line pipeline)
- `arborist corpus-query` (new, local shards via Corpus + run_query)
- `arborist cloud query` (new, sidecar via Corpus + run_query)
prints 3-column table + flags primary-source disagreements.
Smoke fixture (5 questions, all paths):
4/5 all-paths agree on primary source (was 2/5 pre-fix)
1/5 disagrees: dinosaur-extinction
legacy: UNGROUNDED · Edwina (children's book) ← wrong
corpus: STRICT · Dinosaur (main article) ← right
cloud: HYBRID · Edwina (children's book) ← wrong
The new title-boost lifted corpus-query above the legacy here;
cloud still picks Edwina because SidecarReader.search applies
title-boost INTERNALLY (it predates the shared util), so the
extras-penalty stacks weirdly when run_query applies it again.
Next fix: disable internal boost in SidecarReader once shared
util is the single source of truth.
26 corpus/wallet/sidecar tests still green.
SqliteShardCorpus.fts_body uses `chunks_fts JOIN chunks ON rowid`
which doesn't bridge per-shard rowid namespaces under a single
connect_query(ATTACH+UNION) connection. The fix is per-shard
connections: one SqliteShardCorpus per shard, fts_body merged by
raw BM25 score (FTS5's score is comparable across same-tokenizer
shards).
MultiShardSqliteCorpus
- opens each shard with connect(path), wraps in SqliteShardCorpus
- fts_body: sequential per-shard, merge by ascending score
(FTS5 BM25 is 'lower is better'), top-K
- chunks_for_doc: walk shards until one returns rows
- snapshot_root: Merkle over union-deduped document_roots
- Hit.shard_id annotates which shard surfaced each hit so
future routing decisions have it
- Sequential (not threaded): sqlite3.Connection enforces
single-thread access by default and the silent threadpool
exception-swallow was returning [] — local SSD FTS5 is fast
enough (~10ms/shard) that serial is fine
CLI wiring (_cmd_corpus_query):
- Prefer shards_dir over single db when both are set (args.db has
a non-None default which was always winning)
- --shards-dir → MultiShardSqliteCorpus, --db → SqliteShardCorpus
- Skip system shards (qa.db, snapshots.db, selfmodel-chain.db)
Live demo:
make corpus-query Q="who is homer simpsons boss?" LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 2/2 3.97s
(search 0.82s across 5 shards + LLM 3.14s + verify 0.01s)
Tests added (2): per-shard merge, chunks_for_doc routing.
20 corpus tests + 16 wallet/bucket tests still green (36 total).
Retrieval-quality note: the answer still surfaces "Hank Scorpio"
(the You Only Move Twice episode title boost) over Mr. Burns
because MultiShardSqliteCorpus doesn't yet have the title-boost
extras-penalty that SidecarBucketCorpus gained earlier. That's the
next fix to migrate — when it lands in SqliteShardCorpus too, both
backends pick the Homer Simpson primary article.
Parallel to `arborist cloud query`: same corpus_query.run_query
orchestrator, only the Corpus adapter differs (SqliteShardCorpus
instead of SidecarBucketCorpus). Proves the protocol works against
both backends with identical pipeline code:
arborist query "Q" legacy 2000-line query() (untouched)
arborist cloud query "Q" corpus_query.run_query + SidecarBucketCorpus
arborist corpus-query "Q" corpus_query.run_query + SqliteShardCorpus ← NEW
Same render layer (_render_cloud_query_human), same audit_mode +
sources + capacity + timings shape across both new paths.
Caveat: single-DB mode only for now. SqliteShardCorpus.fts_body uses
`chunks_fts JOIN chunks ON rowid`, which doesn't bridge per-shard
rowid namespaces under connect_query() ATTACH-and-UNION. Multi-shard
search needs a per-shard query + merge — the next protocol method
(fts_body_per_shard) to add. Until then, --db <path> or auto-picks
the first non-system .db under --shards-dir.
Live demo (homer + virt-back queries):
make corpus-query Q="who developed virt-back?" LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 2.29s
(search 0.01s · llm 2.26s · verify 0.01s · total 2.29s)
Same E1 citation + render as `make cloud-query` on the same
question; only "0 HTTP requests" footer betrays local vs cloud.
Makefile: `make corpus-query Q="..." [LLM=qwen]`.
`arborist/qa/corpus_query.py:run_query(corpus, question, chat_client, ...)`
is now the single retrieval + evidence + LLM + verify + annotate
pipeline. Takes any Corpus adapter (SqliteShardCorpus | SidecarBucketCorpus
| future edge-proxy), returns the same result-dict shape today's
cloud-query emits (audit_mode, sources w/ used + pointer-ids, capacity,
timings, raw_answer, rendered answer).
`_cmd_cloud_query` reduced from ~310 lines of inline pipeline to ~80
lines of corpus construction + run_query call + progress emission +
render. Behavior identical: same audit_mode, same sources, same
capacity/timings tail.
Before: cli._cmd_cloud_query owned chunk-pull, evidence-build, prompt
construction, LLM call, verifier call, source annotation,
spotlight render — 310 lines of duplication with the local
query() pipeline.
After : cli._cmd_cloud_query owns ONLY corpus construction +
progress emission + render layer. The pipeline lives in
corpus_query.run_query and will be the single source of
truth once query.py refactor moves local onto the protocol.
Tests (corpus_query unit, 4 passing):
* verbatim quote → STRICT (n_verified ≥ 1, used annotation correct)
* capacity + timings dict populated with expected keys
* empty retrieval → UNGROUNDED + zero LLM calls (StubClient.calls == [])
* corpus.name leaks into result for bench attribution
Bench parity preserved (cloud_vs_local.py smoke, 5 questions):
before refactor: 1 regression (Mercury Seven STRICT → HYBRID)
after refactor : 1 regression (same — LLM-stochastic, primary source
identical both sides)
Behavior-preserving. Foundation laid for the local query() refactor:
when query.py learns to take a Corpus parameter, it'll delegate to
the same run_query() and every quality fix lands once for both.
32 prior wallet + corpus tests still green.
First step toward DRYing the local-vs-cloud retrieval diff. Defines a
minimal Corpus Protocol (Hit, ChunkRow, fts_body, fts_title, fts_phrase,
chunks_for_doc, snapshot_root) and ships two adapters:
SqliteShardCorpus wraps a connect_query() connection. Implements
fts_body (via _to_fts5 sanitizer + chunks_fts
MATCH) and chunks_for_doc + snapshot_root.
fts_title / fts_phrase raise NotSupportedError
(those routes will move out of query.py inline
SQL in the next refactor pass).
SidecarBucketCorpus wraps a MultiShardSidecarCorpus. fts_body
delegates to the sidecar BM25 + title-boost +
extras-penalty + title-relevance pipeline.
chunks_for_doc walks the shard's apsw conn via
the bucket VFS, reusing warm page cache.
Title / phrase routes raise NotSupportedError
until the sidecar format ships those indexes.
Together: query() future-refactor takes a Corpus parameter, runs the
routes the adapter supports, skips NotSupportedError, falls through.
Every quality fix (today: possessive stem, accent fold, numeral fold,
extras-penalty title-boost, title-relevance filter) lands once and
both backends consume it through the protocol.
Tests (20 passing, three layers):
tests/test_qa_corpus.py (unit) — Hit/ChunkRow invariants,
Protocol conformance (runtime_checkable isinstance), NotSupportedError
on unimplemented routes, fts_body sanitization (no FTS5 syntax leak),
stopword-only query → [], chunks_for_doc idx-ascending shape.
tests/test_qa_corpus_integration.py (integration) — real corpus →
both adapters → assert recall (target doc in top-K of both) and
byte-identical chunks_for_doc decoding. Catches tokenizer/stem/fold
drift between FTS5's unicode61 and the sidecar's NFKD+ASCII path.
tests/test_qa_corpus_functional.py (functional) — full retrieval +
StubClient LLM + verify_claim_lattice pipeline routed via the
protocol for both adapters; asserts shared audit_mode + primary
source agreement. Prototype of what the post-refactor query() does.
Side fix: MultiShardSidecarCorpus title-relevance filter now falls
open when the title tokenizes to zero content tokens (single-char or
all-stopword titles like "A" or "I" shouldn't be dropped just because
the filter side has nothing to match on).
Real-world: shared endpoints can queue (hermes was queued behind
opencompletion this morning, single calls pushed past 180s but
completed in 121s on retry). 600s tolerates ~10× the typical
non-queued latency before declaring timeout.
Renamed DEFAULT_PRICE_CENTS (per-call) → DEFAULT_DOLLARS_PER_1K_GROUNDED.
Per fox 2026-05-31: at owned-hardware scale, COGS is naturally
expressed per-1000-grounded-answers (hermes $0.09, qwen $0.16),
not per-call cents. The baseline already amortizes ungrounded calls.
Report now shows: per-model implied spend, effective COGS in
$/1k grounded, and cascade delta as % vs always-most-expensive.
20-question live bench (real run): cascade beats always-qwen by
-39% COGS AND grounds more answers (19/20 vs 17/20). Hermes and
qwen are equally reliable (17/20 each) but catch *different*
questions — additive coverage.
Adds bench/cross_model_selfplay.py + `make bench-cross-model` target.
For each question in a fixture, runs `arborist query` once per
configured model (default: Hermes + Qwen) and tabulates:
* audit_mode per model (EVIDENCE-WARRANTED → POINTER-LINKED → UNGROUNDED)
* agreement on primary source URI
* grounding rate per model
* estimated $/grounded-answer (per-call prices configurable)
* cheap-first cascade analysis (try cheapest, escalate on UNGROUNDED)
This is the "ask twice for two options" pattern from the agent
perspective — bakes it in as a benchmark so we can measure whether
the cascade beats always-using-the-stronger-model on $/grounded.
First live run (2 questions × 2 models, $0.41):
- Hermes: 1/1 grounded (1 timeout — operational issue)
- Qwen: 2/2 grounded STRICT
- Cascade: 2/2 grounded for $0.25 — beats always-Qwen ($0.32)
when Hermes succeeds on its first call.
Output: bench/cross_model_results/<utc-iso>.{jsonl,md} (gitignored).
Cloud-query was picking sibling articles (Mona Lisa's Revenge instead
of Mona Lisa, Republics of the Soviet Union instead of Soviet Union,
Mercury 13 instead of Mercury Seven) because:
1. Title-boost counted overlap but not extra title tokens. Both
"Mona Lisa" and "Mona Lisa's Revenge" overlapped query by 2 →
same boost → BM25 favored the shorter movie article.
2. RRF merge squashed per-shard rank-1 hits into a 1/(60+1) tie
across 4 sidecar shards. Tie-breaking was undefined; the right
article was as likely to lose as win.
Two fixes:
sidecar.search title-boost:
+ Filter title tokens through STOPWORDS + len-1 cutoff so 's', 'of',
'the' don't count as extras.
+ Penalty: extras = |title_tokens - query_tokens|; effective bonus
is `max(0, overlap - extras/2) * title_boost`. "Mona Lisa" gets
full bonus; "Mona Lisa's Revenge" gets half.
MultiShardSidecarCorpus.fts_search merge:
+ When all CONTRIBUTING shards have sidecars (their BM25 + boost
scores are directly comparable), merge by max raw score across
shards. RRF was masking score discrimination at the top of the
list.
+ Mixed (sidecar + bucket-direct FTS5) falls back to RRF since
those scales aren't comparable.
Bench (5-question smoke, cloud_vs_local.py):
before fix: 4 regressions / 5
after fix : 1 regression / 5 (and that one is the right source,
only the audit_mode dropped STRICT
→ HYBRID due to LLM-stochastic answer
phrasing)
A query like 'when was final fantasy 7 created?' was scoring every
'Final Fantasy ___' article identically because:
1. '7' got stripped by the length>1 filter, so the query reduced
to {final, fantasy}
2. Even surviving, '7' literally doesn't appear in the corpus when
it spells 'VII'
Two fixes:
* tokenize_text now keeps single-char numeric tokens (so '7' survives
sanitization). Stopword + len filter still applies to letters.
* numeral_expand() pairs 1↔I, 2↔II, ..., 20↔XX. Applied in two places:
- search() expands the query term list before dict lookup, so '7'
finds 'vii' postings (no rebuild needed — chunks tokenized to
'vii' at build time).
- title-boost computes overlap on the expanded set, so 'Final
Fantasy VII' titles win the +24 boost vs +16 for siblings.
Live demo (cloud-query 'when was final fantasy 7 created?' LLM=qwen):
before: UNGROUNDED 0/1 — top sources Final Fantasy X-2, Tactics,
Character design (FF VII article rank 5, never made top-4)
after : EVIDENCE-WARRANTED 2/2 — 'Final Fantasy VII was originally
released on January 31, 1997.' + 'Development began in 1994.'
cited E1 = Final Fantasy VII (#1 in merged top-4)
Single-flag switch to swap the upstream LLM endpoint+model. Default
(LLM unset) leaves --endpoint/--model unspecified so cloud-query
falls back to its built-in default (Hermes-3-8B on ai.unturf.com).
LLM=qwen pins Qwen3.6-27B on qwen.ai.unturf.com (uncloseai). Granular
override still available via LLM_ENDPOINT=… and LLM_MODEL=…
make-vars or the underlying --endpoint/--model CLI flags.
Sample (homer's boss):
make cloud-query Q='who is homer simpsons boss?' LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 2/2 19.4s
(Hermes on the same query landed EVIDENCE-WARRANTED-PARTIAL 1/2)
Two bugs surfaced by 'what are the 3 starter pokemon in pokemon red?'
that returned UNGROUNDED with Russell Ballestrini's blog cited above
every Pokémon Wikipedia article.
Bug 1: ASCII-only word regex split 'Pokémon' on 'é' into ['Pok',
'mon'], so the term 'pokemon' never landed in the sidecar dict and
the title-boost never matched 'Pokémon Red and Blue' against a
'pokemon' query token. Fix: NFKD-fold accents before tokenizing
(both sides — build + query — agree). Mirrored in _to_fts5 too so
the bucket-direct FTS5 path stays consistent (FTS5's unicode61
tokenizer already folds, so the sanitizer was the only place that
needed the fix).
Bug 2: Multi-shard RRF treated a rank-1 hit in a 223-doc personal
blog identically to a rank-1 hit in a 1M-doc shard. Russell
Ballestrini's blog incidentally contains 'red' or 'starter'
somewhere, so its FTS5 returned the root page at rank 1; RRF tied
with Pokémon Red and Blue (also rank 1 in genesis shard 0) and
won by insertion order. Fix: post-merge title-relevance filter
(mirrors local query.py's search.title_filter) — drop hits whose
titles share zero stemmed+folded tokens with the query before RRF
combines them. Russell Ballestrini blog title {russell, ballestrini}
overlaps neither {starter, pokemon, red} → dropped.
Live demo (cloud-query 'what are the 3 starter pokemon in pokemon red?'):
before: UNGROUNDED 0/1 — Russell Ballestrini #1, no useful evidence
after : EVIDENCE-WARRANTED-PARTIAL 3/6 — 'Bulbasaur, Charmander,
and Squirtle' cited to Pokémon Red and Blue article
Mirrors local 'arborist query' instrumentation so the cloud path is
no less observable. Uses the existing arborist.qa.progress.Progress
emitter (auto-on at TTY, override via ARBORIST_PROGRESS=0|1).
Stages emitted:
manifest.start/done URL + shard/sidecar counts
corpus.open.start/done sidecar download + dict parse cost
search.start/done per-shard FTS + RRF merge
context.start/done chunk-content pulls + assembled bytes
llm.start/done model + endpoint + ctx + answer chars
verify.start/done audit_mode + n_verified/n_quotes
Render tail adds (matches local query render):
capacity: prompt N chars (sys N + evidence N + question N) → answer N chars
timings: manifest Xs · corpus_open Xs · search Xs · context Xs ·
llm Xs · verify Xs · **total Xs**
bucket: N HTTP requests · KB · endpoint / model
JSON output (--json / JSON=1) gains 'timings' + 'capacity' fields
with the same shape, so bench harnesses can consume them directly.
Surfaces that sidecar.parse dominates a cold cloud-query (~22s/shard
on the genesis 6M-term sidecar). That's a real future-opt target
(ProcessPool instead of ThreadPool to escape the GIL) but the
mechanism is observable now, which is the prerequisite for tuning.
usage: make cloud-query Q="your question" [JSON=1] [BUCKET_URL=...] [TOP_K=4] [MAX_CONTEXT=24000] [CACHE_MB=64] / — consistent with the
local usage: make query Q="your question" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1] / entry points. No alias kept
(per repo convention against backwards-compat hacks for unused
names; this command landed in 65b6fe6 / cb8c1de during a fast-
iteration burst and has no external consumers yet).
CLI: arborist cloud query '...' (was: arborist cloud ask '...')
Make: make cloud-query Q='...' (was: make cloud-ask Q='...')
Pure-BM25 ranking penalizes long main articles ('Homer Simpson'
Wikipedia article) vs short episode articles with the same query
tokens — length-normalization is doing what BM25 was designed to do
but the result mis-ranks the primary source. Local query.py runs a
SEPARATE title-LIKE retrieval route that surfaces title-match docs
unconditionally; sidecar didn't, so the Homer Simpson article landed
rank 4 behind 'Bart vs. the Space Mutants' and similar incidentals.
Fix: at query time, after BM25 scoring, add a fixed bonus per query-
term overlap with the doc title (already stored in the sidecar doc
table — no rebuild needed). Both sides stemmed with the same
possessive+plural normalization as _claim_title_overlap so
'simpsons' (query) overlaps 'simpson' (title). Default
title_boost=8.0 is enough to lift the right primary source above
incidental short-doc matches without overwhelming BM25 elsewhere.
Live demo (cloud-ask 'who is homer simpsons boss?'):
before: 'Hank Scorpio is Homer Simpson's boss' (cited episode article
where Scorpio appears; primary 'Homer Simpson' article ranked
rank 4 in its shard, never reached the merged top-K)
after : 'Homer Simpson's boss is Charles Montgomery Burns.'
(cited E1=Homer Simpson + E3=Homer's Odyssey episode)
EVIDENCE-WARRANTED-PARTIAL · via claim_lattice 1/2 26.6s
_claim_title_overlap's _stem only stripped trailing 's' on words >4
chars. 'homer's' (mid-apostrophe possessive) became 'homer'' after
stem, which didn't match 'homer' from the title — both claims of a
Simpsons answer cited articles that contained the right content but
the title-overlap check failed, demoting audit_mode UNGROUNDED via
the existing 'all resolving claims TITLE_MISMATCH' rule.
Fix: normalize apostrophes (ASCII and Unicode right-single-quote)
to empty before length/suffix check. Localized to _claim_title_overlap's
inline _stem — no other callers of _content_tokens affected.
Repro / regression:
before: _claim_title_overlap("Homer's boss", "Dancin' Homer") → False
after : _claim_title_overlap("Homer's boss", "Dancin' Homer") → True
unrelated still False; plural collapse still True; 115 verifier tests pass.
Live demo (cloud-ask 'who is homer simpsons boss?'):
before: UNGROUNDED 2/2 (false rejection of two correct claims)
after : POINTER-LINKED-PARTIAL 2/2 (Scorpio claim still title-
mismatched against 'You Only Move Twice' — the episode-
naming convention legitimately ducks token overlap there;
that demote is honest)
Was serial — 4 × 745 MB = ~2 min cold for the 4-genesis-shard manifest.
Build a ThreadPoolExecutor with one worker per shard so the wall cost
is max(per-shard download), not sum. Idempotent: SidecarShardClient
re-reads cached files without re-downloading on subsequent runs.
