Two related fixes for fox-flagged "the convo gets narrowed":
1. Anaphora gate. The retrieval_keywords flow from ancestor cited
titles into FTS5 was firing on every turn — even unrelated topic
switches. Asking "when was batman comic created?" in a Sonic-heavy
session pulled 8 Sonic titles into FTS5 and drowned the Batman
article → UNGROUNDED. Now gated on the question containing a
pronoun ("it", "him", "her", "they", "this", "that", "one"). For
subject-bearing questions, retrieval stands on the question text
alone — the LLM-side conversation_history (added 832297f) gives
the model pronoun-resolution context, but retrieval doesn't need
keywords when the new question already has a subject.
2. Parent-only scoping. When the question IS anaphoric, use ONLY the
immediate parent's cited titles — not the union across all
ancestors. Pronouns almost always refer to the most recent
subject; accumulating across a long Sonic-then-Batman thread let
Sonic dominate keyword weight and the model resolved "it" to
Sonic instead of Batman. Parent-scoped keywords align with how
English antecedent resolution actually works.
Live verified (3-turn smoke):
Sonic created? → HYBRID, Sonic article cited
Batman comic created? → HYBRID, Batman article cited
(was UNGROUNDED with bare-conv polluter)
who created it? → HYBRID, Bob Kane + Bill Finger via
List of Batman creators
(was Sonic Naoto Ōshima — wrong topic)
24 session+providence+run-query tests pass.
Pure density ranking ("count distinct tokens within ±half-window")
loses on Sonic-shaped chunks: "Sonic" appears 20+ times, "1990"
appears once, so any Sonic-anchored slice scores 4 distinct tokens
while the one 1990-anchored slice scores 2. The 1990 slice IS the
load-bearing claim evidence; the Sonic slice is just where the
chunk happens to lead.
Fox flagged 2026-06-01: STRICT-verified claim "Sonic the Hedgehog was
created in 1990" anchored to chunk E5 (franchise timeline), but the
spotlit excerpt showed the AM8/Sega-Team paragraph, not the "April
1990" year entry where the date claim grounds.
Fix: multiply density by 1/freq(anchor_token) in the span. A token
appearing once gets full weight; a token appearing 20× gets 0.05.
Rare anchors (years, named entities, version numbers) now win even
with fewer neighbors in the window — the rare token marks the slice
the verifier actually verified.
Sample: synthetic Sonic-shaped chunk + claim "created in 1990" →
excerpt now centers on "In 1990, Sega began developing the Sonic
the Hedgehog character." (was the Sonic Team / AM8 paragraph).
Live: `make query Q="when was sonic the hedgehog created?"` →
spotlight now leads with "In April 1990, Sega commissioned its AM8
R&D department to create a character…"
Tie-break stays smallest-idx (deterministic). 175 claim_lattice +
verify + inspect tests pass.
The verifier was assuming single-turn — a session pronoun follow-up
("who created it?" after "what is permaculture?") hit the LLM with
no antecedent in the question, so the model couldn't ground.
Retrieval keywords already flowed ancestor titles → FTS5 (the
Permaculture article was in context), but the LLM saw just "who
created it?" with no subject. → UNGROUNDED.
Wire (verifier-blind for retrieval; cache_key-aware for chat):
- providence_query + run_query: new ``conversation_history`` param
(list of OpenAI-shape {role, content} dicts). When set, messages
become [system, ...history..., user(EVIDENCE+QUESTION)] instead
of [system, user]. Folded into the canonical-messages hash so
``conversation_hash`` reflects the chat path — each distinct
branch gets its own cache_key (fix verified: the same question
with vs without history now produces different cache_keys).
- CLI session REPL: builds history from path_to_root of
current_bates (which IS the parent at the moment of the new turn).
Oldest-first. Strips claim-lattice pointer-line excerpts from
prior answers via _session_strip_pointers — the [E#] bindings are
stale across turns (each turn mints a fresh evidence_map) and
the spotlight excerpts inflate the prompt without antecedent
value. The bare prose claims remain.
- Off-by-one fix: was stripping path[1:] under the assumption that
the current node was the just-added turn; in the REPL, current is
the PARENT before the new turn is minted, so path-to-root IS the
history. Drop the [1:].
Live test: "what is permaculture?" → STRICT → "who created it?"
→ STRICT (was UNGROUNDED). The model cites the Permaculture and
David Holmgren articles correctly for the pronoun question.
33 verify+session+providence tests pass. The byte-identity test
for run_query without history is preserved — history defaults to
None and skips the extension entirely.
Python sqlite3's default ("legacy" / DEFERRED) isolation mode opens an
implicit read transaction on the first SELECT and holds it until
commit() / rollback(). For a long-lived viz connection that's a read
snapshot pinned at startup — when the REPL writes a new node in
another process, the viz keeps returning yesterday's view until the
viz process restarts.
Setting isolation_level=None puts the connection in autocommit: each
SELECT runs in its own implicit transaction that ends as soon as the
cursor is consumed, so the next call picks up any WAL frames the
writer committed in between.
Surfaced by fox writing a turn in `make session` and seeing /sessions
api/find return only OLDER sessions — the new node was in nodes_fts
(confirmed via direct sqlite3 CLI query) but invisible through the
viz's cached SessionsView. Restarting the viz "fixed" it, which is the
classic shape of this pin.
Read-seam tests still pass (9/9). The cost of autocommit is one extra
syscall per SELECT to start/end the implicit transaction — negligible
for the viz's request volume.
Refactor from per-session sqlite files to one shared shard at
~/.arborist/sessions.db. Three things that didn't work before now do:
1. Queries are first-class members of the tree.
nodes_fts (FTS5 over question + answer_text + cited_titles) lets
/find <query> walk every prior turn across every session. Cached
answers and threads become findable, surface in the REPL as
`[<bates>] <audit> <question>` lines.
2. Forking from history works the same as forking from a sibling.
parent_bates can cross sids. After /find returns a hit from
last week's session, /cd <bates> + ask = your next question
lands as a child under that historical turn. The cross-session
parent's subtree_hash ripples up its session_root.
3. One global audit chain instead of per-file.
audit_events.event_hash = sha256(prev || canonical body), one
chain over every state change in the shard. `make
session-chain-check` is now a single pass; tampering anywhere
in the operator's history breaks the chain.
Wire:
- arborist/qa/session.py — drop file-per-session SessionStore class;
Session becomes a viewport on SessionStore. cited_titles_json +
n_cited_sources materialized at insert time so FTS5 doesn't need
a join into providence_cache.
- arborist/cli.py — `session` subcommand swaps --gc for --find;
REPL adds /find. Ancestor-keyword extraction now reads
nodes.cited_titles_json directly (no qa.db roundtrip).
- Makefile — `make session-find Q="..." [LIMIT=N JSON=1]`; `make
session-gc` retired (no per-session files to GC).
- docs/sessions.md — rewritten for the single-shard shape.
- tests/test_session.py — 17 tests: create, add, fork (incl.
cross-session), find (FTS5 + by_cache_key), path_to_root crossing
sessions, audit chain (intact + tampered), cited-title extraction,
subtree_hash ripple across sessions.
Migration: pre-existing per-session dbs at ~/.arborist/sessions/*.db
become orphaned. None lost data — test sessions only. Operator can
rm -rf ~/.arborist/sessions/ (or rename to sessions-old/) at leisure.
126 session+providence+verify+inspect tests pass.
Four sites in the session paths were truncating session_root and
cache_key to 12-16 hex chars + "…". These are cryptographic content
addresses — the WHOLE hash IS the identifier; truncation makes them
unusable for copy-paste and forensic reference.
Affected:
- session.py render_tree() header
- cli.py session --list output
- cli.py session REPL banner
- cli.py session REPL per-turn footer
Other hash truncations elsewhere in cli.py (search/inspect/burn paths)
are left alone — those are scan-and-skim views where ellipsis is the
intended affordance. Touch them only when they bite.
Three fixes from interactive feedback:
1. **readline in the REPL** — input() now has up-arrow line history
and emacs/vi editing. Persists to ~/.arborist/sessions/.history
across runs (2000-entry cap). Without this, every typo required
retyping the whole line.
2. **π* canonical projection preflight in providence_query** —
bare arithmetic ("33/66", "0.1+0.2") was tokenized by FTS5 as
["33", "66"] and matched the year-article titles "33" and "66"
(low-DF integer tokens dominate bm25); the LLM then "interpreted"
the bare fraction as years 33 AD / 66 AD. The legacy query()
pipeline has the same preflight at the top of its pipeline;
providence_query was missing it (drift from the #000015/#000030
π* registry work). Now firing the arithmetic@v1 / algebra@v1 /
logic-kernel@v1 kernel short-circuits retrieval + LLM entirely
and returns audit_mode=CANONICAL_PROJECTION with the exact
rational answer.
Live: `make query Q="33/66"` → "1/2" (was HYBRID citing year
articles).
3. **Qwen as default LLM for `make session`** — interactive multi-
turn use rewards a smarter model over Hermes-3-8B's fast-sweep
strengths. LLM=hermes still toggles back to 8B; SESSION_LLM_*
env vars override.
66 verify+session+providence tests pass.
`arborist session` is an interactive multi-turn Q&A REPL where every
turn (or fork) mints one node in a per-session SQLite-backed tree.
Each node carries a stable Bates id (`<sid>-<6-digit>`) and folds into
a Merkle subtree-hash chain; the root node's subtree_hash is the
session_root.
Tree shape lets:
- **Forks** happen implicitly: `/cd <bates>` to a prior node, ask
again → sibling under that parent. Branch points (≥2 children)
surfaced by `/branches`.
- **Page-refresh caching** stay cheap: a client tracking
(bates → subtree_hash, body) only refetches subtrees whose hash
changed. Sibling subtrees that didn't change are byte-identical
→ cache-equivalent. Same property git pack-protocol and IPFS MFS
use.
- **Audit-chain verification** be per-session and independent: each
session db has its own session_audit_events with event_hash =
sha256(prev_hash || canonical_body). `make session-chain-check`
walks all sessions; 0 breaks each = intact.
Wire:
- arborist/qa/session.py — Session class, Bates minting, Merkle
recompute on O(depth) insert, audit chain, helpers (list, render,
resolve <bates|seq|label>).
- arborist/cli.py — `session` subcommand: REPL + --list / --tree
/ --chain-check / --gc / --json flags. Ancestor-titles → retrieval
keywords (parsed from cited-pointer lines in answer_text) flow down
the branch via policy["retrieval_keywords"].
- arborist/qa/providence_query.py — honor policy["retrieval_keywords"]:
augment FTS5 retrieval query without touching cache_key (mirrors
legacy --retrieval-keywords discipline, #000001).
- Makefile — `make session [SID=...]`, `make session-list`,
`make session-tree SID=...`, `make session-chain-check`,
`make session-gc SESSION_KEEP=N`.
- docs/sessions.md — schema, Merkle conventions (portability for
non-Python consumers), REPL command reference.
- tests/test_session.py — 15 tests: create, resume, add_node, fork
via cd, branches, root determinism, audit chain (intact + tampered),
resolve, list, render, sibling-invariance of subtree_hash.
Storage: ~/.arborist/sessions/<sid>.db (self-contained — no FK into
main store). Answers live in providence_cache keyed by cache_key;
session only carries conversation shape. Cache hits stay live across
sessions. Bounded growth via --gc.
Phase 1 scope: tree + Merkle + Bates + retrieval-keyword flow.
NOT in Phase 1: LLM-side conversation_history (threading prior Q&A
into the LLM prompt + conversation_hash). A bare-pronoun follow-up
("who created him?") gets the right retrieval today but the LLM may
still UNGROUNDED because it sees only the new question as user
message. Folding conversation_history into the prompt + cache_key's
conversation_hash dimension is the natural Phase 2.
197 tests pass.
The bullet prefix was a CLI-render convenience but it bleeds into the
answer_text field of the JSON payload — wrong layer. Indented
evidence excerpts under each flush-left claim provide enough visual
structure in plain text; a markdown frontend can re-add bullets if
that view wants them.
71 claim_lattice tests pass.
Pure per-token title MATCH broke multi-token title surfacing. For
"who created spider man?", per-token "spider" alone floods top-K
with single-word titles ("Spider", "Wolf spider", "Spiders (album)")
because single-token titles have shorter doc length → higher bm25.
The "Spider-Man" article (multi-token title) never reached top-32 →
retrieval surfaced only Spider-Man universe filler (Rhino, Sinister
Six, Liz Allan, …) and the canonical primary was absent → answer was
"the evidence does not contain who created Spider-Man" (HYBRID).
Fix at both SqliteShardCorpus and MultiShardSqliteCorpus:
1. AND pass: ``"tok1 AND tok2 AND …"`` single MATCH. Rewards
multi-token title hits — Spider-Man (which contains both
"spider" and "man" as separate tokens after FTS5 hyphen-splits)
ranks where its co-occurrence earns.
2. Per-token pass: one MATCH per token, round-robin merge across
buckets. Preserves the "capitol of paris" Paris-survival fix —
each token guaranteed surface area.
Merge by best bm25 per doc_root; AND-mode hits emit first because
their multi-token grounding is the strongest title-relevance signal.
Live:
- "who is spider man?" → STRICT, 0 violations (was STRICT)
- "who created spider man?" → STRICT, cites Spider-Man article
with Stan Lee + Steve Ditko answer
(was HYBRID, Spider-Man article was
missing from sources entirely)
- "what is the capitol of paris" → Paris still at rank 4 in titles
(the prior fix's invariant holds)
110 tests pass.
For Wikipedia-shaped "who is X?" Q&A, three independent demoters fired
on a definitionally-perfect answer ("who is spider man?" → HYBRID with
TITLE_MISMATCH ×4 + DEFLECTION_DETECTED + LAZY_ANCHOR_DEMOTE). Each
came from a different instrument/design gap:
1. DEFLECTION hyphen blindness (arborist/qa/inspect.py):
_content_tokens_in_order kept "Spider-Man" as one token. Question
"who is spider man?" → subject_anchor="man"; answer "Spider-Man..."
tokenized to ["spider-man"] without "man" → false-positive
DEFLECTION. Fix: split hyphenated compounds into parts (additive).
2. Rule 8 (TITLE_MISMATCH) co-reference gap (arborist/qa/verify.py):
Per-claim title-overlap fails on pronoun continuations. "Spider-Man
is X. The character was created by Y. He has Z." — claims 1 and 3
have zero stem overlap with title "Spider-Man" even though they all
cite the Spider-Man article. Fix: collect evidence_ids title-anchored
by ≥1 resolving claim, excuse subsequent claims whose every cited
eid is already anchored. Applied to both pointer and JSON variants.
3. LAZY_ANCHOR_DEMOTE primary-source excuse (arborist/qa/verify.py):
Pointer concentration ratio = max(pointer_count)/total. Wikipedia
"who is X?" answers naturally pin every claim to one canonical
primary article — the rule's "almost never honest" prior is wrong
for that shape. Fix: skip demote when the dominant pointer's
evidence is source_role="primary_answer_source" (the role only
attaches when retrieval explicitly identified the source as the
question's primary answer, via classify_source_role / rank-1).
Live: who-is-spider-man? — was HYBRID with three demoters firing,
now STRICT 3.0s zero violations.
108 verify+inspect+providence tests pass.
Two coupled fixes for "capitol of paris" UNGROUNDED regression:
1. Per-token title MATCH (SqliteShardCorpus + MultiShardSqliteCorpus).
Single-MATCH `tok1 OR tok2` over documents_fts is DF-biased — the
rarer token's hits dominate the bucket. "capitol" (low DF) crowded
out "paris" (higher DF), so the Paris article never reached the
merge pool. Per-token MATCH gives each query token its own slot;
round-robin merge balances representation across tokens.
2. run_query precomputed_hits kwarg + providence_query threads its
computed hits through. Previously providence_query ran body+title
union → cache lookup → on miss called run_query, which re-did
body-only retrieval internally and discarded the title-aware
union. Now run_query uses the caller's hits when provided.
Also: drop providence_query's outer ThreadPool over body/title —
MultiShardSqliteCorpus.fts_body/fts_title already fan out across
shards internally, single-shard SqliteShardCorpus connections aren't
thread-safe by default, and the bare-except was hiding ProgrammingError.
Live verification: `make query Q="what is the capitol of paris"` —
audit_mode STRICT in 1.2s, Paris article surfaces at rank 3 (was
absent before; sources were noise like SNCF Class BB 9200, Hendrix
"Live in Paris & Ottawa 1968" etc).
test_providence_query.py + test_run_query_byte_identity.py pass (6/6).
Pre-existing test_cold_object schema failure unrelated.
Fox flagged 2026-06-01: `make query Q="when did sesame street first
air on television?" LLM=qwen` returned UNGROUNDED with citations to
"AC Graph Coloring" and "AC Sudoku Puzzles" textbook chunks. None of
the retrieved docs mentioned Sesame Street.
Root cause: providence_query was running body-only FTS5 retrieval.
Body progressive-AND on a multi-token natural-language query
collapses to whichever-token-survives (here "first" or "air"),
which matches the textbook crawler shard (`crawl_appliedcombinatorics_org.db`)
heavily. The article "Sesame Street" never enters the candidate set.
Legacy query() avoids this by running a TITLE FTS5 route in parallel
with body — title BM25 over the documents_fts shadow index ranks
"Sesame Street" article at rank 1 (bm25 -19) for this query.
Fix has two parts:
1) SqliteShardCorpus.fts_body: re-flip FTS5Backend's sign-flipped
score back to raw bm25 (more-negative = better). Without the
reflip, body returned +0.137 (high = good) and title returned
-19 (low = good); the merge couldn't apply a uniform min-bm25
rule. Cloud SidecarBucketCorpus already emits raw bm25; local now
matches that convention.
2) providence_query retrieval phase: fan body + title in parallel
via ThreadPoolExecutor (max_workers=2), merge by min(score) per
document_root, apply_title_boost on the union, take top_k. Body
route catches docs with strong body BM25 but no title overlap;
title route catches canonical articles whose name matches the
question subject. Either side is sufficient to surface the right
doc; both together is what legacy does.
Measured (live `make query LLM=qwen` on sesame fixture):
before: UNGROUNDED, citations to "AC Graph Coloring" / "AC Sudoku
Puzzles", primary picked wrong
after: STRICT, "Sesame Street" article at primary, Qwen answers
"Sesame Street premiered on November 10, 1969" with two
STRICT-verified citations to the Sesame Street article and
History of Sesame Street article. 8.3s wall time.
Top-8 retrieved titles now read:
Sesame Street · Sesame Street (Japan) · History of Sesame Street
· Sesame Street media · 5, Rue Sésame · Sesame Park · 1, rue Sesame · ...
Smoke "test" + Anarchism fixture from test_providence_query also
pass (4 tests). 22 corpus + providence tests pass.
Progress event surface also gains body= and title= counts on
search.done so operators can see which route surfaced what.
Fox flagged 2026-06-01: "search is taking 45 secs when it used to
search across all 4 shards in like 2-5 secs before." Confirmed
search.done=48653ms on "when did the muppet show first air on
television?" — 9-token natural-language query.
Root cause: SqliteShardCorpus.fts_body used a naive `MATCH 'tok1
OR tok2 OR ...'` over the FTS5 chunks index. For multi-token
queries where ANY token has high document-frequency, BM25 has to
score every chunk in the OR-union. "muppet show first air
television" pulls ~300k matches on a 1.5M-chunk shard; BM25 ranks
all of them; ~30s cold I/O per shard × 5 shards = 150s wall time
before the parallel fan-out, ~45-50s after. Legacy already solved
this in arborist.search.fts5.FTS5Backend.search — progressive-AND
mode (intersect posting lists, fast), retries with shortest-token
dropped on zero, and only falls back to OR-mode with a high-DF
filter when AND exhausts. The comment literally says "~27s cold
I/O" for the OR-mode pathology.
Fix: SqliteShardCorpus.fts_body now delegates to FTS5Backend
instead of building its own OR-mode SQL. Same shard, same FTS5
index — just the right query construction.
All-stopword short-circuit preserved (FTS5Backend has a sentinel
``""`` fallback that matches arbitrary docs; pre-check via
_to_fts5() guards against that).
Measured (live `make query LLM=qwen` on the same fixture):
search: 48.7 s → 0.7 s (70× speedup)
total miss: 50 s → 2.9 s
total hit: ~50 s → 0.95 s
This is on fox's actual workload via Qwen. All 79 corpus/providence
tests pass.
Cloud/Sidecar path unchanged — slim FTS5 sidecars don't have the
high-DF posting-list issue at scale (their corpora are smaller) and
the cloud apsw connection wouldn't benefit from progressive-AND the
same way. Future work: same FTS5Backend pattern for
SidecarBucketCorpus if a similar slowdown surfaces.
Restores the per-phase stderr streaming that legacy query() had —
operator watches `arborist query` and sees which phase is currently
running, in real time, before the JSON / human render lands on
stdout. Fox flagged this regression after the CLI flip:
"it used to stream what arborist was doing to stderr before
emitting the json or quote."
Pattern: providence_query gains a `progress` kwarg (defaults to a
no-op shim so library callers without a Progress object don't
suddenly start writing to stderr). `_cmd_query` already builds a
Progress via _progress_from_env(); now threads it through.
Stages emitted, mirroring legacy query()'s vocabulary:
search.start top_k=N
search.done hits=N ms=N
cache.lookup primary=<8-hex>
cache.burn removed=N (only when --burn)
cache.hit lookup_path=primary ms=N
-- OR --
cache.miss lookup_path=primary
llm.start model=<id> top_k=N
llm.done audit_mode=X n_verified=N n_quotes=N ms=N
persist.start
persist.done ms=N
Operator gets visibility into which phase is paying the time;
silent ARBORIST_PROGRESS=0 still no-ops as before.
4 providence_query tests still pass.
Two perf fixes after the snapshot_root() removal in 9f4152e
unblocked the real bottlenecks on multi-token natural-language
queries against the 5-shard genesis corpus.
1. MultiShardSqliteCorpus.fts_body + _fanout now parallel
============================================================
Previously sequential — 5 shards × ~6s per shard on a query like
"when did aliens film come out?" = 25s wall time per fts_body call.
ThreadPoolExecutor over the per-shard sqlite3 connections drops
that to roughly max(per_shard) instead of sum.
Connections are now opened with `check_same_thread=False`
(`arborist.store.connect()` gains a kwarg, default True preserves
the existing behavior; MultiShardSqliteCorpus passes False so
read-only fan-out works across threads). Read-only FTS5 queries
serialize under SQLite's internal locks; the per-Connection thread
check is what was blocking cross-thread use.
Measured (in-process, 5 shards, "when did aliens film come out?"):
before: 25.5 s sequential fan-out
after: 7.6 s parallel fan-out (warm)
12.3 s parallel fan-out (cold)
End-to-end `arborist query` on the same fixture:
before (post-snapshot_root fix): 65-110 s
after: 14-18 s (cache hit)
16-18 s (cache miss + persist)
legacy reference (--legacy): 10-13 s
Default path is now within 30-40% of legacy on this workload (down
from the 5-6× slowdown fox saw before). Headroom remains because
providence_query still re-runs retrieval inside run_query on miss
(redundant ~7 s); a future refactor can thread precomputed_hits
through.
2. Makefile: `make query LLM=qwen` actually routes to Qwen
============================================================
The `ifeq ($(LLM),qwen)` block setting LLM_ENDPOINT + LLM_MODEL
lived AT LINE 1633, AFTER the `query:` target at line 191.
Make evaluates top-down, so by the time `query:` ran the LLM
variables weren't set — and the recipe didn't reference them
anyway. Result: `make query LLM=qwen Q="..."` silently used
Hermes, the CLI default.
Fix: moved the ifeq block to line 192 (just above the `query:`
target so both `query:` AND the later `cloud-query:` see the
same definitions), and added
`$(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT) --model $(LLM_MODEL),)`
to the query recipe. cloud-query unchanged.
Smoke:
$ LLM=qwen Q="..." make query
# llm: https://qwen.ai.unturf.com/v1 / Qwen3.6-27B-UD-Q4_K_XL.gguf
.venv/bin/arborist ... query ... --endpoint https://qwen.ai.unturf.com/v1
--model Qwen3.6-27B-UD-Q4_K_XL.gguf ...
176 tests pass.
CATASTROPHIC perf bug surfaced by fox 2026-06-01: arborist query
was 50-110 s on local shards (vs 10 s legacy). Profiled to
corpus.snapshot_root() at 41-71 s per call. That helper walks
EVERY shard's documents table and Merkle-roots all ~4M
document_roots from the 5-shard wikipedia corpus. It's the right
answer to "what's the global corpus hash?" but the wrong answer
to "what's the source_root dimension of this query's cache_key?"
Fix: use _context_root([retrieved doc_roots]) — Merkle root over
the 4-8 docs that retrieval actually surfaced. Same helper legacy
query() uses for source_root in cache_key (query.py:3758 region).
Microseconds vs minutes.
Cache identity semantics unchanged: rows still key on which docs
surfaced + question + policy + model. The change is purely how
that root is computed.
Measured (StubClient smoke, 5 shards, top_k=4):
default fresh miss: 63s → 9.2s (~7× speedup)
default cache hit: 51s → 4.5s (~11× speedup; cache now saves
real work instead of running
the full pipeline twice)
legacy fresh (ref): 10s
Default `arborist query` is now faster than legacy on miss AND
dramatically faster on cache hit. Cache identity unchanged.
4 providence_query tests still pass.
Per fox 2026-06-01: same treatment as #000071 — replace the
review-archaeology structure with what we SHOULD grow. Ticket goes
from 978 lines (original Anchor6 §§1-8 design log + dav1d-review §0
retrofit) to 498 lines of directive spec. **Joseph6 stays as the
first registered example grammar** per fox's note — concrete enough
that an implementer sees what a WorldDimensionGrammar looks like
end-to-end, not abstract enough to lose its load-bearing role.
What changed in shape:
Before: §0 dav1d verdict retrofit + §§1-8 archaeology of the
original Anchor6 spec being reviewed (validate seed source,
segmentation method, mapper choice — all decisions long
since made).
After: §1-13 forward spec. Goal · Axis split · Hard constraints
(all phases) · AnchorN primitive · WorldDimensionGrammar ·
Quantization mappers (with uint256-H₁ + no-SO(3) corrections
documented inline) · π*_w_object canonicalizer with
four-identity-hash record · **Joseph6 as worked example** ·
Phase 1 deliverables (9 items) · Pre-review empirical
bench preserved as §7 · Phase 2/3 deferred · Open questions
(3 remaining; 5 closed by bench, 4 by dav1d's review) ·
Cross-references · Five-step alignment · One-line review
history at the bottom.
What changed in content: nothing material. The corrected spec from
the prior §0 retrofit IS the body now. The original Anchor6 design
log is no longer inlined — git history preserves it at commit
`862662b` (pre-rewrite tip); readers who want the rejection-by-
rejection detail go to
docs/dav1d-reviews/000070-spatial-anchor-pi-w-object--2026-06-01.txt.
Critical technical corrections preserved inline (not as "what was
fixed", but as the directive answer):
- §3.1: uint256 for H₁ position (octree depth >8 entropy
preservation)
- §3.2: rename `map_rotation_so3` → `map_rotation_euler_ypr` (no
SO(3) overclaim — quantized Euler is not SO(3) coverage)
- §4: WorldObjectRecord carries all four identity hashes
(grammar_hash, axiom_pack_hash, manifest_hash, seed_hash) for
replayability
- §5: Joseph6 ships as one example grammar; future grammars
register through the same mechanism
TICKETS.md index row also rewritten in directive voice.
doc_counts tests still pass.
Per fox 2026-06-01: replace the descriptive review-archaeology
structure with what we SHOULD grow. Ticket goes from 805 lines
(original §§1-8 design log + my dav1d-review §0 retrofit) to
389 lines of clean directive spec.
What changed in shape:
Before: §0 "Dav1d review verdict" decision-table + §§1-8
archaeology of the original Joseph6 spec being rejected.
After: §1-12 forward spec. Goal at the top, hard constraints,
composite ChainRoot identity, five bridge outcomes with
typed witness shapes, privacy class vocabulary, Phase 1
deliverables, Phase 2/3/4 roadmap, retro-validation
appendix, cross-references, three remaining open
questions, five-step alignment, one-line review history
at the bottom pointing at the archive file.
What changed in content: nothing material. The corrected spec from
the prior §0 retrofit IS the body now. The original Joseph6 design
log is no longer inlined — git history preserves it at commit
fadc50a; readers who want the rejection-by-rejection detail go to
docs/dav1d-reviews/000071-world-bridge-grammar--2026-06-01.txt.
TICKETS.md index row also rewritten in directive voice. Shorter,
less "what was wrong" + more "what to build."
Net effect: an implementer picking this up reads a forward-looking
ticket they can act on, not an archaeology of which framing was
rejected. The dav1d review history is one line at the bottom, not
the structural frame.
doc_counts tests still pass.
The 1026-line de-novo response arrives after #000070 was corrected
from fixed Anchor6 to generic AnchorN/WorldDimensionGrammar. The
bridge concept is valid and necessary; the Joseph6-coupled framing
is now wrong. Same pattern as #000070's review: §§1-8 preserved as
the design log of what was reviewed; new §0 carries the corrected
spec.
Verdict matrix (12 rows in §0 decision table):
✅ GO: bridge concept (Agreement / Translation / Embassy), Phase 1
doc-only, bridge_seed@v1 deferred to Phase 2, no audit_mode,
no SQL, bridge atlas Phase 3.
❌ REWRITE: #000070-as-Joseph6-sibling → AnchorN sibling;
chain_id=governance_policy_hash → composite ChainRoot
with optional v7-W fields (null sentinels for language-
only QA chains); single event_type='bridge' → typed
schemas per kind; privacy as Phase 3+ footnote → Phase 1
vocabulary (4 classes); #000059 "already-shipped"
overclaim → proposed/structurally-aligned unless repo
confirms.
❌ REJECT: doc-only Phase 1 flipping #000013 to "kernel_in_progress"
— correct status is "bridge_grammar_specified"; kernel_
in_progress is for actual kernel landings.
Five bridge outcomes (was three):
Agreement — grammars match on invariant set
Translation — hash-pinned adapter proves the mapping
Embassy — foreign object hosted with limited rights
Quarantine — NEW: bridge attempted, invariant validation FAILED,
record the rejection so future attempts see what broke
No-bridge — NEW: grammars genuinely incompatible, explicit
declaration that no bridge will exist
Composite ChainRoot identity:
chain_id = SHA256(canonical({
chain_id_version, history_root, governance_policy_hash,
canonicalization_version, schema_version, chunking_version,
world_manifest_hash, world_dimension_grammar_hash,
axiom_pack_hash, optional_verifier_policy_hash
}))
Language-only chains: three v7-W fields are null sentinels →
chain_id collapses to existing identity. v7-W chains: all mandatory.
Phase 1 deliverable (corrected):
1. Substrate-paper extension §"World-bridge grammar" (5 outcomes
with typed witness shapes + composite ChainRoot + privacy
vocabulary + sovereignty rule + #000070 cross-ref + retro-
validation appendix with corrected shipped/proposed framing)
2. Optional namespace stub arborist/world/bridge/__init__.py
(STATUS = "namespace_reserved")
3. NO kernels / NO bridge_seed canonicalization / NO SQL / NO
cache-key dimension / NO new audit_mode / NO verifier change /
NO KATs
4. #000013 status bumps to "bridge_grammar_specified", NOT
"kernel_in_progress"
Sovereignty rule survives unchanged (chain law local, bridge law
treaty-only, no bridge overwrites native chain law — guard against
forced-unity failure mode). Hash-pinned translators survive
unchanged. Topic-named under arborist/world/bridge/ unchanged.
Full review archived at:
docs/dav1d-reviews/000071-world-bridge-grammar--2026-06-01.txt
Folds dav1d's 1904-line review verdict into the ticket as §0 (new),
preserves §§1-8 as the design log of the original Anchor6 proposal
that was reviewed. Status flips from "open · awaiting dav1d review"
to "open · dav1d GO with rewrite · spec revision pending before any
kernel ships."
Verdict matrix:
✅ GO: generic AnchorN substrate primitive (not fixed Anchor6)
✅ GO: Joseph6 as the first registered default grammar (not THE
ontology)
✅ GO: deterministic object-state canonicalization
✅ GO: axiom/physics-loaded WorldDimensionGrammar as the scalable
path
❌ NO-GO: hard-coding H₁..H₆ as final ontology
❌ NO-GO: runtime LLM-decided dimensionality (axioms MAY propose,
only deterministic validators may accept; grammars must
freeze via grammar_hash before proof-path use)
❌ NO-GO: framing this as a "semantic verifier warrant"
❌ NO-GO: relation/event/place/agent_trace in this ticket
❌ NO-GO: SQL persistence at Phase 1
Critical technical corrections:
- H₁ MUST use uint256, not uint64 (octree position entropy at
depth >8)
- Rotation mapper is map_rotation_euler_ypr(), NOT "SO(3)"
(review §17 — quantized Euler triple, no continuous SO(3))
- Canonical record carries grammar_hash + axiom_pack_hash +
manifest_hash + seed_hash alongside the per-dimension values
- Missing privacy.class = HARD reject (no PUBLIC fallback)
- spatial-anchor-object@v1 folds into canonicalization_version
only — no new audit_mode token, no verifier_policy_hash bump
Corrected package layout:
arborist/substrate/spatial_anchor.py (AnchorN, split_anchor_n)
arborist/world/grammar.py NEW — WorldDimensionGrammar
arborist/world/pi_star/object.py (derive_world_object_record
+ 5 mappers)
arborist/pi_star/spatial_anchor_object.py NEW — registry adapter
bench/fixtures/spatial-anchor-object/known-answer-tests.jsonl
tests/test_spatial_anchor.py
tests/test_world_dimension_grammar.py
tests/test_pi_star_spatial_anchor_object.py
Implementation NOT started — arborist/world/__init__.py STATUS still
"namespace_reserved"; only bench/spatial_anchor_validation.py
(pre-review empirical bench, commit 55b651f) on disk. Review arrives
at the right moment: no production code committed against the original
Anchor6 spec yet, so the AnchorN reframe lands in the spec before
the wrong primitive ships.
Full review archived at:
docs/dav1d-reviews/000070-spatial-anchor-pi-w-object--2026-06-01.txt
Five of ten original open questions resolved by the pre-review bench
(Q1/Q2/Q3/Q8/Q9); review answers Q4/Q5/Q6/Q7; Q10 (paper-amendment
wording) is the only remaining open question — substrate paper text
needs revision to introduce AnchorN + grammar layer instead of fixed
Anchor6.
Bench-v5 (5 themes × 3 questions × 2 paths, Hermes-3-8B) AFTER the
fold-stack lift + apply_title_boost wire-up to use _title_query_tokens
shows the SAME 12 regressions as the pre-wire-up baseline:
- 5 wrong-primary picks (Dr Who, Albert/Ahmed/Alaric the third/first,
Casa Batlló error)
- 7 STRICT→HYBRID demotes on correct primaries (Spider-Man, Ampère,
Dr Syn, Dr V64, Finnish Defence Forces, Hopewell Centre, labor
economics)
Diagnosis: apply_title_boost only reranks docs ALREADY in the
candidate set. body-only retrieval (default with multi_route=False)
never surfaces "Doctor (Doctor Who)" so no fold-aware rerank can
promote it. Legacy surfaces it via the title route. FTS5 porter
stemmer handles plurals but NOT Dr→Doctor or third→III, so the
SqliteShardCorpus.fts_title method also doesn't help here without
the folds applied at retrieval-token-gen time, not rank time.
Updated ticket with the honest assessment: 5 Path A stages across
v1/v2/v3 have now proven legacy query() doesn't decompose into a
library of helpers. Three forward paths offered for fox to decide:
A. leave default at providence, accept fold regressions (env
escape hatch ARBORIST_LEGACY_QUERY=1 already shipped)
B. flip default back to legacy, treat providence as
infrastructure for cloud-query/corpus-query only
C. keep both alive long-term — separate query2 command
Two coupled changes that close most of the bench-driven regression
in #000072 PLUS fix the user-facing output regression from the
CLI default flip (492d1a8).
PART 1 — Lift fold-variants stack to _text_norm.py
=================================================
Moves _TITLE_TOKEN_RE, _TITLE_STOPWORDS, _TITLE_TOKEN_POLICY,
_NUM_ORD_TO_ROMAN, _NUM_ROMAN_TO_ORD, _HONOR_FOLD, _BRIT_FOLD,
_ascii_fold, _hyphen_fold_variants, _numeral_fold_variants,
_accent_fold_variants, _honorific_fold_variants,
_brit_fold_variants, _title_query_tokens from query.py:115-325
into arborist/qa/_text_norm.py.
query.py re-exports the symbols under their original names so
existing call sites + tests (test_accent_fold, test_numeral_fold,
test_query, test_claim_lattice) keep working.
apply_title_boost (in arborist/qa/corpus.py) NOW USES
_title_query_tokens for both query and title sides, replacing the
weaker tokenize_text + numeral_expand pair. Without this rewire
the lift was a no-op — the helpers were available but nothing on
providence_query's body-only default path called them. Bench
v3 (commit pending: 23-14-10Z.jsonl) confirmed the no-op state;
the v4 wire-up is what produces actual fold behavior.
source_roles.py + retrieval_routes.py + corpus_query.py drop their
lazy `from arborist.qa.query import _title_query_tokens` dance and
import directly from _text_norm — kills the lazy-import warts and
removes a circular-import hazard.
PART 2 — Restore providence_query output parity
================================================
Fox surfaced 2026-05-31: "the new providence query output with and
without json is worse it doesn't give timings anymore and a bunch
of other stuff." Comparing field surfaces, legacy emitted 38 keys,
providence_query 19. Half the renderer's lookup table was missing.
New on providence_query result (miss + hit branches both):
- prompt_chars: DICT with messages_total / system_prompt /
evidence_or_context / user_question / grounding_reminder
(not the bare int the first cut emitted — the renderer reads
.get('messages_total') etc.)
- answer_chars: int
- context_root: Merkle root over source document_roots (matches
legacy's _context_root helper)
- unverified_quotes, partially_verified_quotes,
warrant_proven_claim_idxs, format_collapsed: surfaced from the
verdict that run_query already computed but wasn't propagating
- timings: now emits BOTH float-second keys (search, context,
llm, verify, total + retrieval/prompt_build aliases) for bench
scripts AND int-millisecond keys (search_ms, context_ms,
llm_ms, verify_ms, cache_lookup_ms, persist_ms, total_ms) for
the renderer's _render_query_human lookup table
Still absent (require pre/post gates that providence_query doesn't
port yet — separate work): answerability, claim_cap_applied,
frame_detection, lazy_anchor_ratio, pointer_id_distribution,
preflight_hash, quantifier_intensity, quantifier_matched_token,
quantifier_explicit_count, question_state, retrieval_purity,
scope_bound_hint, soft_preflight_hint, repair_changes,
pre_repair_audit_mode, failure_stage.
Human render now shows: 5-line audit summary · source list · capacity
line · timings line · cache_key tail — same density legacy emits.
JSON consumers get 26 keys (was 19).
283 tests pass.
Bench note: a v4 fold-themes bench would now show whether the
apply_title_boost wire-up cleared the 12 regressions; killed mid-
run when this fix landed. Re-run when convenient.
Re-bench legacy vs providence_query on fold themes (accent, hyphen,
honorific, brit, numeral) after the proxy memory fix.
Result (15 question-pairs through Hermes-3-8B): 12 regressions,
2 improvements, 1 tie. Net-negative on these themes, BUT all 12
regressions trace to a single root cause — the 5 fold-variants
helpers (_hyphen, _numeral, _accent, _honorific, _brit) live inside
_title_query_tokens at query.py:288-325 and providence_query
lazy-imports the WRAPPER without lifting the fold helpers.
Same gap manifests two ways:
- Wrong primary (5): Dr Who → pathology; Albert/Ahmed/Alaric the
third/first → wrong articles; Casa Batlló → error
- STRICT → HYBRID on correct primary (7): the verifier's Rule 8
title-overlap check calls the SAME _title_query_tokens —
without folds, "Andre-Marie" (claim) and "André-Marie" (title)
are distinct tokens, overlap fails, audit_mode demotes
Path A v3 surfaces: lift the fold-variants stack to _text_norm.py,
re-export from query.py, drop the lazy-imports in source_roles.py +
retrieval_routes.py. ~250 LOC moved + ~50 LOC import-rewrites,
half-day. Lower risk than v1 (pure code motion, helpers are
identical between paths).
Themes deliberately skipped this round (need their own gates ported
separately): quantifier_subset, metacog_subset, warrant_chain_probe,
es, fr. Re-bench AFTER v3 lands.
Also commits bench/legacy_vs_providence_bench.py + the result JSONL
so the regression set is reproducible.
bench/chunk_fetch_speed.py — measure per-chunk fetch latency for the
two cloud paths a future JUST_ENOUGH=1 blob-publish move would
compare against: (1) apsw HttpRangeVFS on the big shard .db (current
FtsSidecarShardClient fallback path), (2) direct HTTP GET on a
same-bucket object (proxy for per-chunk blob fetch).
Measured 2026-05-31 against clones/full-bench/000.db (12.5 GB) +
clones/sidecars-fts/000.idx.db:
apsw median: 704 ms / chunk (mean 773, first 1594, warmup ~500)
blob median: 91 ms / chunk (mean 92, flat — no warmup effect)
speedup: 7.7×
Real-world: ~2.6 s saved per fresh 4-chunk query. For cache-miss
flows that already pay 5-15 s on the LLM call this is real but not
transformative. The big win for blobs is cache HITS that don't go
to LLM (returns drop from ~100 ms via cached chunks to sub-50 ms
via blobs) and bulk bench runs (400 fetches = 4 min vs 30 s).
Interactive single-question flow with LLM in the loop is fine on
the apsw path; blobs stay as future optimization, not blocker.
Also archives bench/three_way_results/*.jsonl (3 runs across the
#000072 Phase 1 progression) + bench/slim_fts_parity_results/ so
the journey from "cloud diverges from local" through "cloud matches
local 5/5" is on disk for the design-log record.
#000072 Phase 2 step 3. The user-facing 'arborist query' command now
routes through arborist.qa.providence_query (cache-aware run_query
wrapper) instead of the legacy 2000-line arborist.qa.query.query().
Legacy function stays alive in the module — other importers
(test_query.py, internal calls) keep working — but the CLI defaults
to the new orchestrator.
Escape hatch:
arborist query "..." --legacy
ARBORIST_LEGACY_QUERY=1 arborist query "..."
Either re-routes through the legacy retrieval+gate pipeline. Useful
when a themed bench subset regresses on the new path and operators
need fleet-wide fallback while the gap gets ported.
Why now: the legacy "dinosaur → Edwina" primary-source bug fixes
itself on the default path — slim FTS5 cloud parity already proved
the run_query orchestrator picks the canonical 'Dinosaur' article
on Q5. Smoke confirms it:
arborist query "why did the dinosaurs go extinct?" --dry-run
→ primary: Dinosaur (default — new)
arborist query "..." --legacy --dry-run
→ primary: Edwina, the Dinosaur Who Didn't Know She Was Extinct
Known gaps providence_query DOESN'T port today (legacy still has):
- pre-retrieval: canonical_projection, crosslang sandwich,
quantifier preflight, metacog, soft_preflight, frame_detection
- post-retrieval: answerability, repair, witness, sandwich edge-out
- merkle_proof column is "[]" placeholder; equivalence_class
fallback lookup omitted
- args legacy accepts that providence ignores: retrieval_keywords,
extra_body, translator, fidelity, over_fetch
Forcing-function-style rollout per fox 2026-05-31 — the bench
regression-finding IS the next signal. Themed subsets in
bench/qa_questions.txt that exercise the missing gates may regress;
those are the targets for the next round of porting.
CLI changes:
- new --legacy flag (with ARBORIST_LEGACY_QUERY=1 env equivalent)
- _cmd_query branches on use_legacy → legacy query() vs builds
Corpus + calls providence_query
- providence_query result.status mapped fresh_persisted/burned →
cache_miss_then_written so the bottom-of-function exit-code
check stays consistent
providence_query.providence_query cache_hit branch now surfaces
raw_answer, verifier_method, n_quotes, n_verified, unverified_quotes,
violations so the render layer + journal emitter don't crash on
None when serving from cache.
283 tests pass in the broader gate.
Phase 2 step 2 of #000072. The skeleton landed in 20faae0 looked up
providence_cache + returned on hit, but the miss branch just
returned the fresh run_query result without writing anything back —
so every call paid the LLM cost. Now misses persist:
1. Build run_dag via arborist.qa.dag.build_run_dag (claim-lattice
mode, 9-stage variant — same shape legacy query() emits)
2. Append a providence_write event via arborist.store.append_audit
INSIDE a BEGIN IMMEDIATE transaction
3. INSERT INTO providence_cache with all 25 columns the schema
requires (cache_key, source_root, document_uri, question_*,
answer_text, merkle_proof placeholder, 4-dim policy hashes,
3-dim schema versions, audit_event_hash linking to the just-
appended event, run_dag_root + run_dag_blob, audit_mode, etc.)
4. COMMIT — atomic; audit chain + providence_cache stay
consistent on crash mid-way
Result dict on miss now carries audit_event_hash + run_dag_root so
the caller can reference the audit chain or replay the DAG.
status="fresh_persisted" (was "fresh" in the skeleton) names the
new behavior.
NOT done yet (deferred to subsequent steps):
- merkle_proof is "[]" placeholder. Schema requires NOT NULL.
Real per-chunk proofs are a follow-up; the cache row is
consistent without them but downstream wallet verification has
nothing to walk.
- burn_existing doesn't emit a providence_burn audit event
(legacy query() does — query.py:3084 region). Audit chain
still grows monotonically on the persist side, just doesn't
record what was burned.
- equivalence_class fallback lookup (legacy tries both dedup-mode
keys when fidelity allows; primary only here)
Tests (tests/test_providence_query.py, 4):
- first call persists row + audit event
- second call returns cache_hit ignoring different stub
- burn_existing forces re-run + re-persist; audit chain grows
- chain links correctly across two distinct cache_keys
283 tests pass in the broader query/corpus/sidecar/wallet/bucket/
claim_lattice/byte_identity/providence gate.
Records the 2026-05-31 attempt at Path A (port the 5 reranks,
re-bench) and the result: smoke score went DOWN from 3/5 to 1/5
with reranks wired in. Root cause: legacy's rerank multipliers are
tuned against legacy's candidate-set shape (over_fetch=32, per-shard
parallel routes, body-density baked in earlier), not against my
multi_route fan-out's shape (per_route_limit=top_k*4, post-merge
candidates, filter-then-rerank instead of filter-during-route).
Helpers stayed in tree as importable building blocks (commit
d099995). Wire-up was reverted before commit so user surface is
unchanged.
Four v2 directions surfaced and documented for the future
investigation:
1. Match legacy's oversample factor (top_k*8+ or over_fetch=32)
2. Apply body-density filter BEFORE rerank cascade, at source
3. Rerun reranks on per-shard route output before final merge
4. Synonym expansion at retrieval time, not just filter time
None are blockers individually but each is a focused investigation.
The honest takeaway: legacy query()'s rerank pipeline is not a
"library of multipliers you compose in order" — it's a tightly
coupled cascade where each stage's tuning depends on what the
previous stages emitted. Collapsing it requires understanding
those couplings, not just lifting the helpers.
#000072 Path A first attempt: ported the 5 downstream reranks from
legacy query() into arborist.qa.retrieval_routes:
- body_density_passes / filter_by_body_density (Corpus.doc_body)
- rerank_by_source_role (SOURCE_ROLE_RANK_WEIGHTS)
- rerank_by_title_purity ((1+overlap)*(1+purity), shards_dir-gated
synonym_expand_strict)
- rerank_by_ordered_token_match (LCS over title tokens)
- rerank_by_body_coverage (sqrt body coverage, Corpus.doc_body)
Also unfreezes ``arborist.qa.corpus.Hit`` so the reranks can mutate
.score in place (matches legacy _Hit convention). ChunkRow stays
frozen (it's content-addressable evidence). test_hit_is_frozen test
renamed and inverted.
NOT WIRED INTO run_query: smoke probe with all 5 wired in legacy
order (filter → body_density → body_coverage → source_role →
title_purity → ordered_token → apply_title_boost) made the
multi_route regression WORSE:
pre-reranks: 3/5 correct (Soviet Union ✓, Mt Kilimanjaro ✓,
Mona Lisa ✓, Mercury Seven ✗,
dinosaurs ✗)
post-reranks: 1/5 correct (Soviet Union ✗ → "national bandy team",
Mt Kilimanjaro ✓,
Mona Lisa ✗ → "Painting Mona Lisa",
Mercury Seven ✗ → "305th Air Mobility
Wing", dinosaurs ✗)
Root cause: legacy's reranks were tuned against legacy's
candidate-set shape (multi-shard parallel _search_corpus with
body-density baked in EARLIER, over_fetch larger than the per_route
limit I'm using, and a different rivalry-exclusion order). Applying
the same multipliers to my multi_route fan-out's candidate set
lands the cascade in a different basin — short noisy titles with
high stem-overlap get amplified into rank-1 territory.
The helpers stay in tree as importable building blocks for a future
Path A v2 attempt. Possible v2 directions: (a) match legacy's
oversample factor (32+ vs my 4×top_k); (b) apply body-density
filter BEFORE rerank cascade (legacy does this earlier in
_search_corpus); (c) rerun against per-shard route output instead
of post-merge candidates so per-shard discrimination survives.
policy=None / experimental multi_route=True paths unchanged in
behavior — multi_route is still strictly worse than body-only
(documented in #000072) but no longer worse than itself with
reranks; reranks aren't auto-applied.
264 tests pass.
Documents the work shipped this session (Phase 1 foundation, 10
commits 9ba6317..20faae0) and the blocker that stops Phase 2: the
multi_route pipeline regresses on 2/5 smoke fixture questions
(Mercury Seven → Sam T. Beddingfield; dinosaurs extinct →
Paul Austin Kelly) because Phase 1 hasn't ported the 5 downstream
rerank stages legacy query() uses to suppress noisy phrase-route
hits — body-density check, body-coverage sqrt rerank, source-role
rerank, title-purity rerank, ordered-token-match rerank.
Path B chosen 2026-05-31: stop here, leave multi_route off by
default (which IS default — policy=None preserves body-only
behavior). Phase 1 foundation stays in tree as future-ready
infrastructure; user surface unchanged.
Path A (port the 5 stages, multi-day effort) reserved for a
future focused session. Until then, legacy query() keeps
producing the same answers it always has, INCLUDING the wrong
"Edwina" pick on dinosaur Q5. The slim-FTS5 cloud path already
fixes that bug for `arborist cloud query` / `arborist
corpus-query` callers (proven 5/5 source parity in d9fb6a9).
Bumps Next ID 000072 → 000073.
The cache wrapper Phase 2 needs to start collapsing legacy query()
into a thin adapter. Minimal-viable shape:
providence_query(corpus, question, chat_client, *, qa_db, policy,
model_id, burn_existing, top_k, ...) → dict
- Computes 8-dim cache_key from question + policy + model + source_root
+ canonical messages (system + user with EVIDENCE/QUESTION/grounding
reminder — same shape as run_query, byte-identity gate from step 5
catches drift)
- Optional burn_existing: deletes the live cache row first
- Looks up providence_cache: hit returns cached row + bumps hit_count
- Miss: calls run_query(corpus, question, chat_client, policy=policy)
and returns the result with cache metadata
NOT done yet (deferred to subsequent Phase 2 sub-steps):
- Cache PERSIST on miss — run_query produces audit_mode + sources but
legacy providence_cache schema wants run_dag_root, context_root,
prompt_hash, verifier_method, n_quotes/n_verified, violations_json,
etc. that aren't on run_result yet. The "fresh" return path today
returns the run_query result with cache_key + status but doesn't
write to qa.db. Persist needs lifting from query.py:3746 lines.
- equivalence_class fallback lookup (legacy tries both dedup-mode
keys when fidelity allows; minimal path checks only primary)
- Pre-retrieval gates (canonical_projection, crosslang, quantifier,
metacog, soft_preflight, frame_detection) — they stay in legacy
query() for now
- Post-retrieval add-ons (answerability, repair, witness, sandwich
edge-out) — same
- retrieval_keywords, translator, extra_body — legacy-only knobs
Legacy arborist.qa.query.query() is UNTOUCHED — still the user-facing
entrypoint. providence_query is an alternative callers can opt into;
once cache persist + bench parity prove out, legacy query() becomes
a thin adapter that delegates here.
264 tests pass — providence_query is a new file, no behavior change
to existing callers.
Closes Phase 1 step 6 of #53. When the caller passes
policy={"base_version": "wikitext-base-v1"} (or any truthy value),
run_query now invokes arborist.wikitext.to_base() on each chunk
span BEFORE truncating to per_doc_budget, so a 24 kB raw-template
span gets compressed first and the LLM/verifier see the same
prose. Same shape legacy query() applies at three sites.
policy=None / policy without base_version: no-op, preserves the
byte-identity gate. mwparserfromhell missing (the optional dep):
import fails, no-op — already the legacy convention.
policy["base_version"] folds into governance_policy_hash via the
caller's policy dict — opting in DOES rotate cache_key for that
policy partition, which is the right semantics (different prose →
different cache).
264 tests still pass. Phase 1 substantively complete; Phase 2
(providence_query.py + legacy query() collapse) starts next.
When policy.multi_route fans across body+title+phrase+core_keyword
and merges by MIN bm25, the resulting set over-recalls on noisy
titles (e.g. "East Asia" outranks "Nineteen Eighty-Four" on the
Orwell phrase query because raw bm25 over the merged set doesn't
know about accept-path semantics). The 5-accept-path filter
arborist.qa.retrieval_routes.filter_by_title_relevance (landed
step 3) gates the merged set on: title-overlap / synonym / TFIDF-core
match / verbatim-phrase match / hyphen-fold anchor.
Wired so accept-paths 2 and 4 actually fire — core_match_roots
and phrase_match_roots are passed through from this turn's
core_hits / phrase_hits, so docs surfaced via those routes pass
the filter even when titles miss.
Smoke (multi-shard wiki corpus, "has oceania always been at war
with east asia"): Nineteen Eighty-Four now ranks #1 (was rank 3
before filter). Other phrase-route hits like "Nineteen Eighty-Four
in popular media" stay; the geographic "East Asia" / "Oceania"
title hits get filtered out.
264 tests pass (was 263 + 1 multi-route test from 6c that confirms
filter path doesn't blow up on the Anarchism fixture either).
When the caller passes policy={"multi_route": True}, run_query now
fans out across four retrieval routes in parallel and merges:
1. fts_body — body BM25 (the only route in pre-6c)
2. fts_title — title-only BM25 via documents_fts
3. fts_phrase — verbatim 4-gram phrase MATCH; closes the
allusion gap ("always been at war" → 1984)
4. core_keyword_match — TF-IDF core route for neologisms
Each route is fail-open: NotSupportedError → []. core_keyword
returns [] on SidecarBucketCorpus (no derivations in slim sidecar);
phrase / title / body all work cloud-side via the slim FTS5 sidecar.
Merge by MIN bm25 per document_root (FTS5 returns negative; lower
wins). core_keyword's positive match_count scores are kept only as
a tiebreaker when no FTS5 route surfaced that doc — handled
explicitly via score-sign discrimination since the scales are
incomparable.
Lifted helper: question_phrases(question, n=4) from query.py's
_question_phrases into arborist.qa.retrieval_routes — pure-stdlib
n-token window extractor, no stopword stripping (the diagnostic
signal IS the stopword).
policy=None / policy={} (no multi_route flag) keep the existing
body-only path — byte-identity gate from step 5 still green. 263
existing tests + 1 new multi-route test pass.
Still missing for full legacy parity: filter_by_title_relevance
integration in run_query (the 5-accept-path filter is available in
retrieval_routes.py since step 3 but isn't wired into the
orchestrator yet). That's the next sub-step — without it, the
multi-route merge over-recalls on noisy title overlaps.
New bench/teacher_judge.py reads judge_input.jsonl, cross-judges
each (question, model, answer) using a DIFFERENT model than the
answerer (default: hermes-answers → qwen judges, qwen-answers →
hermes judges). Writes verdicts.jsonl + markdown summary including
a false-STRICT highlight section.
Key implementation details:
- Reuses _judge_prompt from cross_model_selfplay so iterating on
the prompt template doesn't require re-running benches.
- Sends chat_template_kwargs:{enable_thinking:false} on every call
— required for qwen.ai.unturf.com (llama.cpp deepseek-reasoning
format) where reasoning eats max_tokens before producing content.
vLLM (hermes) silently ignores the unknown kwarg.
- Prompt template tightened: previously said "Reply in format
VERDICT: <rationale>" which qwen took literally, replying with
the word VERDICT instead of the verdict token. Now explicit:
"write the chosen verdict word itself".
Live cross-judge result on 151-row 76-question bench:
hermes (judged by qwen): 44 CORRECT, 23 WRONG, 9 PARTIAL → 58% accuracy
qwen (judged by hermes): 45 CORRECT, 6 WRONG, 11 PARTIAL → 73% accuracy
Qwen is materially more accurate despite costing 1.8× more — matches
fox's "qwen slightly outperforms" intuition with hard numbers.
8 false-STRICTs surfaced including Q12 (hermes conflated Roman
Empire with HRE), Q48 (hermes answered with Niger River info on a
Nile question), and Q72 (both models missed the Game of Thrones
reference in "winter is coming").
When policy is provided (any non-None dict), the orchestrator now:
1. Classifies each hit's source_role via
arborist.qa.source_roles.classify_source_role(title, qtokens_stem,
document_uri). Same heuristic legacy query() uses — noisy/sequel/
secondary markers fire first, then breadth-of-title-stem-coverage
decides primary vs background.
2. Splits max_context_chars by SOURCE_ROLE_BUDGET_WEIGHTS
proportionally (primary 2.0, noisy/sequel 0.5, others 1.0).
Primary answer source claims ~2× the slice — same shape as
legacy query()'s per-source cap.
policy=None preserves the pre-step-6b shape EXACTLY: rank-based
roles (rank 1 = primary, else background), flat
`max_context_chars / len(hits)` budget. Byte-identity gate from
step 5 remains green.
Tested: policy={} on the Anarchism single-token fixture classifies
the top hit as primary_answer_source (matches legacy). Existing
259 tests + new role-class test = 262 in the gate.
Phase 1 step 6a of #53. Smallest additive change to set up Phase 2's
cache wrapper. Existing callers (corpus-query, cloud query) pass no
policy and behave IDENTICALLY to before — byte-identity gate from
step 5 stays green.
When policy IS provided, twelve recognized verifier kwargs forward
through to verify_claim_lattice:
allowed_source_roles, max_pointers_per_claim, min_citation_coverage,
min_claim_content_tokens, lazy_anchor_demote_threshold,
lazy_anchor_demote_min_pairs, max_claims_per_answer,
subject_tokens_absent_threshold, warrant_check_enabled,
deflection_check_enabled, format_collapse_check_enabled,
warrant_chain_roots
UNKNOWN keys are silently ignored — a policy dict shared with legacy
query() may carry fields (base_version, retrieval_keywords, etc.) the
orchestrator doesn't yet honor; ignoring them keeps the call site
clean instead of requiring callers to filter.
Steps 6b/c/d extend the policy surface:
6b: role-classified + role-weighted context budget
6c: multi-route retrieval (title + phrase + core_keyword + body merge)
6d: wikitext-strip
Tests cover three contracts:
- policy=None is byte-identical to pre-6a (262 tests including the
step-5 byte-identity fixture pass)
- policy={"max_claims_per_answer": 0} actually trips TOO_MANY_CLAIMS
(proves the kwarg reaches the verifier, not just the function
signature)
- policy with unknown keys (e.g. base_version) doesn't blow up
Phase 1 step 5 of #53 — the load-bearing piece. Freezes the cache-
identity byte shape today so the run_query rewrite landing in Phase 2
can be checked against it.
What's pinned (tests/fixtures/byte_identity/claim_lattice.json):
- SHA-256 of CLAIM_LATTICE_SYSTEM_PROMPT (drift = every cache rotates)
- SHA-256 of CLAIM_LATTICE_GROUNDING_REMINDER (same)
- Per-question question_hash (strict + equivalence_class modes)
- Per-question conversation_hash on the synthetic 2-message array
[system + user(EVIDENCE+QUESTION+grounding_reminder)] — the EXACT
shape arborist/qa/corpus_query.py:run_query builds
- governance_policy_hash on three reference policy shapes
- model_profile_hash for hermes / qwen / stub
Plus a determinism sanity test that pins the algos themselves
(SHA-256, _canonical_json key-sorting, dedup-mode question canonical).
Risk class addressed (Plan §6 risks #1+#2): conversation_hash takes
the FULL OpenAI messages array. Any drift — message reorder, whitespace
shift, optional message gated on a different condition — rotates every
cache_key in the world and orphans every providence_cache record on
re-lookup. Same for governance_policy_hash on the policy dict (a new
field rotates everything). The fixture catches a drift the SECOND it
happens, with a diff-style failure naming the path that drifted.
Re-capture mode: `CAPTURE=1 pytest tests/test_run_query_byte_identity.py`
rewrites the fixture. Only do this on deliberate prompt-shape or
policy-shape changes that are treated as cache-invalidation events.
Phase 1 step 4 of #53. Two new protocol methods that the unified
run_query needs but the protocol didn't expose:
- core_keyword_match(qtokens, *, limit) → list[Hit]
TF-IDF core-keyword route. Finds source docs whose distilled
tfidf-core content contains any of qtokens. Closes the neologism
gap (e.g. "permacomputer" matching a Grok conversation about it
via its TF-IDF core, even though permacomputer never appears in
a title). Returns Hits whose .score is the integer match_count
(also in .extras["match_count"]) — UNIQUE among routes in being
higher-is-better, not bm25 lower-is-better.
- doc_body(document_root) → str | None
Concatenated chunk text for one document. Used by the body-
coverage rerank stage that needs the full body (not just top-K
chunks) to decide whether the doc actually discusses qtokens.
Implementations:
- SqliteShardCorpus: full SQL, lifted verbatim from query.py's
_docs_with_core_keyword_match (same word-boundary LIKE +
match_count tallying). Gracefully returns [] if derivations
table missing (older shard).
- MultiShardSqliteCorpus: per-shard fan-out, dedupe by doc_root
keeping MAX match_count across shards (HIGHER wins for this
route).
- SidecarBucketCorpus: core_keyword_match raises NotSupportedError
(the derivations table that maps tfidf-core → source isn't in
the slim FTS5 sidecar). doc_body concatenates chunks_for_doc
output (works on bucket via blob fallback).
Smoke: on the genesis wikipedia shards core_keyword_match returns
[] cleanly — those shards have 92 claim_pack derivations but 0
tfidf-core derivations, so the route correctly produces nothing.
Validation: 257 tests in the query/corpus/sidecar/wallet/bucket/
claim_lattice gate pass.
Phase 1 step 3 of #53. Lift the title-relevance filter to its own
module so run_query can call the same filter the legacy query() uses.
Why this one first: it's the load-bearing piece of the 9-stage
retrieval pipeline that closes the verifier-fabrication gap on
synonym / TFIDF-core / phrase-route / hyphen-fold hits. The other
six rerankers (_rerank_by_title, _rerank_by_source_role,
_rerank_by_title_purity, _rerank_by_ordered_token_match,
_rerank_by_body_coverage, _ordered_match_length) can move
independently when run_query needs them; they aren't blockers.
Adaptation from legacy _Hit to a duck-typed Hit: the function now
reads .title + .document_root via getattr, so both the legacy
arborist.qa.query._Hit dataclass AND the protocol Hit from
arborist.qa.corpus satisfy it without a type bridge.
Lazy imports for _title_query_tokens (full fold-variants stack still
in query.py) and the concepts module (synonym_expand /
rivalry_excluded / has_compare_phrasing) avoid an import cycle and
defer cold-start cost.
Validation: 257 query/corpus/sidecar/wallet/bucket/claim_lattice
tests pass. No behavior change — query.py re-exports under the
same underscore name (_filter_by_title_relevance) so existing
call sites are byte-identical.
Phase 1 step 2 of #53. Hoist the trailing-s stemmer to a single home
in arborist.qa._text_norm so query.py, corpus.py, and source_roles.py
all use the same implementation.
Before:
- query.py:_stem_token_for_match — `len>4 and endswith('s') and not
endswith('ss')` (apostrophes assumed pre-stripped by _TITLE_TOKEN_RE)
- corpus.py:apply_title_boost._stem (inline) — same length/suffix
check PLUS apostrophe strip ("'", "’")
Behaviorally compatible when inputs are pre-stripped, but the dual
implementations were a drift risk waiting to bite. The apostrophe-safe
version (corpus.py's) is the canonical now — handles raw title text
without an upstream sanitizer, no behavior change for the pre-stripped
call sites.
Re-exports in query.py + import-update in corpus.py + source_roles.py
keep every existing caller working. 257 query/corpus/sidecar/wallet/
bucket/claim_lattice tests pass.
Defers lifting _title_query_tokens (and its 5-fold variant helpers —
hyphen, numeral, accent, honorific, brit) for later: those carry
years of bench-tuned hot-loop optimization and a TITLE_TOKEN_POLICY
slug threaded into run-DAG provenance. source_roles still lazy-imports
_title_query_tokens from query.py; no change there.
Phase 1 step 1 of #53 (collapse legacy 2000-line query() into the
unified run_query orchestrator). Pure code motion — no behavior
change, no signature change. Lifts:
- SOURCE_ROLE_BUDGET_WEIGHTS / SOURCE_ROLE_RANK_WEIGHTS
- _NOISY/_SEQUEL/_SECONDARY title-marker tuples
- _classify_source_role function
from query.py:781-854,1579-1591 to a new arborist/qa/source_roles.py
module. query.py re-exports under the old names so existing imports
(and the 45-test test_query.py suite) keep working.
Goal: run_query needs the source-role classifier to do role-weighted
context-budget splits (Plan §5 Phase 1 deliverable 1). Until this
file exists, run_query has no import path to it that doesn't pull in
the full 4280-line query.py module. Lazy-imports `_title_query_tokens`
and `_stem_token_for_match` from query.py for now — those move to
_text_norm.py in a later Phase 1 step.
Validation: 45 tests in test_query.py pass; 257 tests in the broader
query/corpus/sidecar/wallet/bucket/claim_lattice gate pass.
STRICT audit ≠ factually correct. The verifier passes any answer whose
quotes match source text; a model can quote correctly and still draw a
wrong conclusion. Without a judge, the bench can't detect false-STRICTs.
This flag emits <ts>.judge_input.jsonl alongside the regular results —
one row per (question, model) with a model-agnostic teacher prompt.
Feed to any teacher (claude -p per row, remote API, Hermes self-judge)
to get CORRECT|WRONG|PARTIAL|UNCERTAIN verdicts. Skips rows that
errored or returned empty answers.
Grounded in observed reality from the 76-question 2010-wiki bench:
Hermes produced 2 confidently-wrong STRICTs (Q12 conflated Roman Empire
with Holy Roman Empire; Q48 answered the Niger River when asked about
the Nile). Qwen had 0 false STRICTs over the same fixture. Naming the
flag --teacher-model-judge (not --opus-judge) keeps the harness
provider-agnostic.
The three "92 claim_pack docs" tags were drifting against shard 000.db's
21 rows because the harness only counted one shard, but the doc prose
("#000031 closed at 92") meant the corpus total (21+16+38+17 across
genesis shards 000-003).
Two-line fix path: extend the harness to sum across all ???.db shards
via a `*:` prefix (e.g. `*:documents?source_type=claim_pack`), then
prefix the three drifted tags. Aligns the harness scope with the
semantic scope of the claim instead of forcing the claim to shrink to
one shard.
The `*:` glob:
- Matches `[0-9][0-9][0-9].db` basenames only (operator sidecars
qa.db / snapshots.db / selfmodel-chain.db skipped)
- Skips shards lacking the named table (schema-version tolerance)
- Returns _DB_MISSING when no genesis shard exists (CI / fresh-
checkout skip semantic preserved)
- Returns _TABLE_MISSING when no contributing shard has the table
Documented in ticket-000044 §3.4 + a third example showing the new
syntax. Diagnosis credit to a sub-agent investigation that confirmed
zero eviction/falsification audit events on claim_packs — the data is
intact; the harness was just single-shard.
Completes the Corpus protocol surface. Both routes already worked at the
storage layer (every shard has chunks_fts AND documents_fts virtual
tables; FTS5 supports MATCH '"phrase"' natively); they just weren't
plumbed through. Now SqliteShardCorpus, MultiShardSqliteCorpus,
BucketClient, FtsSidecarShardClient, MultiShardSidecarCorpus, and
SidecarBucketCorpus all expose fts_title + fts_phrase end-to-end.
Bonus fix in arborist/ingest.py: every shard's INSERT INTO documents
now also INSERTs into documents_fts in the same transaction. Without
this, fts_title returned empty on freshly-ingested shards — only
migrated genesis shards had documents_fts populated. The cost is one
FTS5 row per new doc, negligible vs the chunk inserts.
Also includes operational cleanup (E):
- Deleted s3://arborist/clones/manifest-sidecar.json
- Deleted s3://arborist/clones/full-bench-64k/00[0-3].sidecar.bin
(~3.1 GB reclaimed; the slim FTS5 manifest is now the only cloud
surface)
- Fixed MultiShardSidecarCorpus.stats() — was calling .bucket on
FtsSidecarShardClient (attribute went away when SidecarShardClient
was deleted); now calls .stats() directly on whichever client.
Validation
----------
- Local fts_title("dinosaur"): "Dinosaur" main article ranks #1
- Local fts_phrase(["always been at war"]): "Nineteen Eighty-Four" ranks
#1 (the canonical phrase-route test from CLAUDE.md)
- Cloud (slim FTS5 sidecar) fts_title / fts_phrase: IDENTICAL ranking to
local on the same query (bit-for-bit FTS5 parity preserved)
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed (the same two
pre-existing failures from main HEAD)
