#000072: open ticket for legacy query() collapse + multi_route regression

Documents the work shipped this session (Phase 1 foundation, 10
commits 9ba6317..20faae0) and the blocker that stops Phase 2: the
multi_route pipeline regresses on 2/5 smoke fixture questions
(Mercury Seven → Sam T. Beddingfield; dinosaurs extinct →
Paul Austin Kelly) because Phase 1 hasn't ported the 5 downstream
rerank stages legacy query() uses to suppress noisy phrase-route
hits — body-density check, body-coverage sqrt rerank, source-role
rerank, title-purity rerank, ordered-token-match rerank.

Path B chosen 2026-05-31: stop here, leave multi_route off by
default (which IS default — policy=None preserves body-only
behavior). Phase 1 foundation stays in tree as future-ready
infrastructure; user surface unchanged.

Path A (port the 5 stages, multi-day effort) reserved for a
future focused session. Until then, legacy query() keeps
producing the same answers it always has, INCLUDING the wrong
"Edwina" pick on dinosaur Q5. The slim-FTS5 cloud path already
fixes that bug for `arborist cloud query` / `arborist
corpus-query` callers (proven 5/5 source parity in d9fb6a9).

Bumps Next ID 000072 → 000073.

docs/TICKETS.md

@@ -111,6 +111,7 @@ Newest first. Update on every open/close.
 
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
+| #000072  | Collapse legacy `query()` into the unified `run_query` orchestrator | **open · Phase 1 foundation shipped 2026-05-31 (10 commits) · Phase 2 BLOCKED on 5 missing rerank stages** — multi_route smoke probe regressed on 2/5 questions (Mercury Seven → Sam T. Beddingfield; dinosaurs → Paul Austin Kelly), so Phase 1 is structurally complete but not production-quality. Phase 1 shipped: `source_roles.py` extraction (9ba6317), canonical `stem_for_match` in `_text_norm.py` (a83e47b), `retrieval_routes.py` with `filter_by_title_relevance` (056d785), Corpus protocol gains `core_keyword_match` + `doc_body` (e322bbd), byte-identity test fixture as Phase 2 safety gate (1822741), `run_query` gains `policy=` kwarg threading 12 verifier kwargs (72d1117), role-classified + role-weighted budget (03f248c), multi-route retrieval body+title+phrase+core_keyword merge (5fdd573), filter wired into multi_route path (6c2ec1b), wikitext-strip when `policy.base_version` set (b8bd9d6), `providence_query.py` cache-wrapper skeleton with cache persist deferred (20faae0). Every commit ships with `policy=None` preserving pre-step byte-identity behavior — existing `arborist corpus-query` and `arborist cloud query` callers are unaffected; the slim-FTS5 cloud path already wins on the dinosaur Q5 fixture (picks "Dinosaur" not legacy's "Edwina"). Phase 2 blocked because Phase 1's `multi_route` pipeline lacks the 5 downstream rerank stages legacy uses to suppress noisy phrase-route hits: body-density check (`_body_density_passes`), body-coverage sqrt rerank (`_rerank_by_body_coverage`), source-role rerank (`_rerank_by_source_role` with RANK_WEIGHTS), title-purity rerank ((1+overlap)*(1+purity)), ordered-token-match rerank (longest-common-subsequence). **Path A** = port the 5 stages → re-bench → resume Phase 2 cache persist + legacy-query() collapse to 50-line adapter. **Path B chosen 2026-05-31** = stop here, defer Phase 2, leave multi_route off by default (which IS the default — `policy=None` preserves body-only retrieval). Until Path A: Phase 1 foundation stays in tree as future-ready infrastructure; user surface unchanged (legacy `query()` still produces same answers, INCLUDING the wrong "Edwina" pick on dinosaur Q5). Full spec + commit log + bench data + the missing-rerank-stages inventory in `docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md`. | 2026-05-31 | — |
 | #000071  | World-bridge grammar (Joseph 3-bridge inter-chain framework) | **open · awaiting dav1d review + fox go/no-go · doc-only Phase 1** (2026-05-31; sibling to #000070, opened same day after Joseph @TrudoJo posted his bridge-framework extension to the 6-dim spatial-ontology — "chain A has its own world rules, chain B has its own world rules, a bridge exists only where a deterministic agreement layer can be proven"). Three bridge kinds: **Agreement** (rules match, direct passage), **Translation** (rules differ but a hash-pinned adapter exists), **Embassy** (foreign-object representation with limited rights). Bridge identity: `bridge_seed = hash(src_chain_id ‖ src_node ‖ src_rule_v ‖ tgt_chain_id ‖ tgt_rule_v ‖ translator_id ‖ invariant_set)`. Joseph's sovereignty rule (hard): chain law remains local, bridge law governs translation only, **no bridge may overwrite native chain law** — guard against forced-unity failure mode. **Primary contribution: retroactive-validation appendix** naming three already-shipped arborist primitives as Joseph-bridge-shape patterns one-for-one — **#000066 graft-receipt mode = Agreement bridge**, **#000056 Operation Sandwich = Translation bridge** (opus-mt hash IS the translator_id), **#000059 bounded-ingestion graveyard = Embassy bridge** (POINTER-LINKED only, never promotes, banner-labelled). The framework is not a new requirement; it's a unifying spec that names what arborist already does in 3+ places, plus surfaces a fourth pattern (bridge atlas for multi-instance federation with potentially-different `governance_policy_hash` peers) as genuinely new work for `arborist/mesh/`. **Phase 1 is doc-only** — paper extension §7 in `docs/_source/merkle-agi-v7w-spatial-temporal.rst` + `arborist/world/bridge/__init__.py` namespace stub mirroring `arborist/world/__init__.py` discipline + Joseph-memory extension. No kernel code, no SQL change, no `cache_key` extension, no new `audit_mode` token, no bench (doc-only review surface). Phase 2 = `pi_star/bridge-seed@v1` canonicalization deliberately deferred to avoid premature locking before three implementation tickets opine. Phase 3 = per-bridge-type kernels + bridge atlas + `event_type='bridge'` writer + mesh integration. Phase 4 = cross-domain bridges (language ↔ spatial via #000015 composition theorem). **Ten open questions for dav1d** in §8: doc-only vs registry-slot-at-Phase-1 (Q1), chain_id = governance_policy_hash vs manifest hash (Q2), three-bridge taxonomy completeness (Q3), translator integrity hashing depth (Q4), bridge witness via `event_type='bridge'` (Q5), `audit_mode` discipline for translated objects (Q6), bridge privacy class default (Q7), endianness reconfirmation (Q8), Phase 2 KAT count + adversarial vectors (Q9), paper-amendment wording (Q10). Hard constraints: A1–A3, no continuous tensors, no `audit_mode` token, no SQL change at Phase 1, topic-named under `arborist/world/bridge/`. Full spec in `docs/tickets/ticket-000071-world-bridge-grammar.md`. | 2026-05-31 | — |
 | #000070  | Spatial-anchor π*_w_object (Joseph 6-dim determination kernel) | **open · awaiting dav1d review + fox go/no-go · pre-review bench shipped `55b651f`** — pure-stdlib empirical-validation script (`bench/spatial_anchor_validation.py`, `make bench-spatial-anchor`, ~2 s, RNG-seed-pinned) measures: §1 avalanche mean 767.85 bits vs PRF null 768 (z = -0.49) · §2 octree chi² uniformity \|z\| < 1 at L=2,3,4 · §3 collision birthday-bound ratio 0.989 / 1.038 at L=4,6 · §4 cross-region Pearson r in [-0.018, +0.012] across all 15 pairs of H₁..H₆ (none flag at 4σ) · §5 domain separation Arm A 767.91 bits independent / Arm B **0.00 bits exact collision** confirming the dedicated-`spatial_anchor_seed` discipline. Five of ten open questions (Q1 seed source · Q2 segmentation · Q3 octree mapper · Q8 endianness · Q9 KAT adversarial vectors) now resolve by measurement; Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions for dav1d. Full empirical detail in ticket §7a. Originally surfaced 2026-05-31 when fox brought in Joseph @TrudoJo's procedural-spatial framework — "hashes do not encode the world, hashes determine the world" — and asked whether it fits arborist's substrate model. It does, cleanly. Implements the first verifier kernel under the #000013 v7-W reserved namespace (`arborist/world/__init__.py` `STATUS = "namespace_reserved"` → `kernel_in_progress` on land): a single committed 32-byte SHA-256 hash deterministically expands into six named 32-byte regions `H₁..H₆` via the HMAC-SHA-512 KDF already shipped in `arborist/substrate/anchor_prg.py` (#000035), and each region drives one quantized object dimension under a fixed canonical mapper — `H₁→octree position`, `H₂→scale level`, `H₃→quantized SO(3) rotation`, `H₄→material palette`, `H₅→raw links (reserved for π*_w_relation sibling ticket)`, `H₆→behavior code`. Domain separation from #000035 by dedicated `spatial_anchor_seed` (manifest-published) — keeps #000035's KAT freeze + dav1d 2026-05-11 final review intact. Hard constraints: stays inside A1–A3, no continuous tensors in proof path, no new `audit_mode` token (substrate commitments are not warrants — the four-rung ladder is unchanged), no SQL schema change at Phase 1, `canonicalization_version` absorbs the new `spatial-anchor-object@v1` registry slot. Phase 1 deliverable ~150 LoC + 10 KATs + tests in one PR: `arborist/substrate/spatial_anchor.py` (segmentation), `arborist/world/pi_star/object.py` (six mappers), registry entry, KAT vectors at `bench/fixtures/spatial-anchor/known-answer-tests.jsonl` matching #000035 KAT discipline, `tests/test_spatial_anchor.py` + `tests/test_world_pi_star_object.py`, substrate-paper amendment citing Joseph (@TrudoJo) in `docs/_source/merkle-agi-v7w-spatial-temporal.rst`. Position mapper at Phase 1 = octree (substrate paper §2.1 + §A worked example); H3/S2/Hilbert/Morton siblings deferred — Morton's value is purely as the relation-kernel `pair(A,B)` cheap bit-interleave and surfaces in the sibling ticket. **Ten open questions for dav1d** in §8: seed-source choice, segmentation method, position mapper, scope split (object alone vs object+relation per CLAUDE.md memory `feedback_ticket_proliferation`), privacy-class fail-closed default, `audit_mode` discipline confirmation, manifest-validator timing, endianness reconfirmation, KAT count + adversarial vectors, paper-amendment wording. Five-step §7 deletions captured: dropped the prior `/tmp/arborist-spatial-ontology-plan.md`'s geographic-search backend (encoding-route confusion), `Document.extra` lat/lon hooks (different ticket if at all), `audit_mode=HYBRID` for spatial hits (convention-illegal), geohash (subsumed by Morton), quadtree as separate (octree at z-level-0), Hilbert at Phase 1 (deferred), tier-1/2/3 hash-suite framing (conflated hard vs soft hashes), and `arborist/spatial/` namespace (violates topic-naming rule). Full spec in `docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md`. | 2026-05-31 | — |
 | #000069  | Arborist VIZ / Merkle Command Center (Pyramid + six.js + SSE browser dashboard) | **open · awaiting go/no-go · doc-only scaffold** (2026-05-27; filed from `/home/fox/Downloads/TICKET_0000VIZ_*`, stack corrected same day per fox). Configurable browser dashboard for inspecting arborist's content-addressed state: Merkle root explorer, proof verifier, claim warrant + graveyard, audit timeline, run-DAG replay, cache-key explainer, root diff, 3D Merkle lattice, optional circuit/activation traces. Read-only consumer; arborist proper stays source-of-truth, dashboard projects state. **Stack pinned to unturf-native** (fox 2026-05-27, supersedes proposal §3): **Pyramid + Jinja2 + SQLAlchemy** (matches `remarkbox` / `make_post_sell` / `unhomeschool.com` idiom), **SSE** (`text/event-stream` via Pyramid streaming response) for live audit/claim/falsifier patches, **vanilla JS + six.js** (fox's patched three.js fork at `git.unturf.com/gumyum/six.js` — three.js r175 + CWE-407 patches incl. ObjectBVH O(N)→O(log N); bundles vendored from `~/git/cupPCB/cdn/six/`; third-instance MOAD-0001 dogfood alongside `java-topology` + gumyum-engine) for 3D widgets and large-graph rendering, SQLite for dashboard metadata (no PostgreSQL/ClickHouse/Redis/NATS by default — promote on measured need), no React / no Next.js / no Node build step. Server-rendered SVG (or Graphviz `.dot` per existing `docs/diagrams/*.dot` pattern) replaces React Flow for run-DAG widgets. Browser-side proof verification dropped from v1 (server-side Pyramid view returns PASS/FAIL + receipt; reinstate phase-N only if third-party-verification use case surfaces). **Three filing-note gates before phase 0** (in ticket body): **F-1** sibling-repo home — implementation lives in a new `~/git/arborist-viz` (Pyramid Python, matches existing unturf apps), not in-tree; arborist's contribution is the read-API spec + view package + arborist library import via `arborist.embed`. **F-2** scope split — proposal carries 8 phases (§17 phases 0–8); recommended cut keeps phases 0–3 (schema + shell + proof/root widgets + claim/audit/run widgets) inside #000069, and spawns sibling tickets for SSE streaming (4), 3D six.js (5), massive-graph (6, only if measured need surfaces), circuit-tracing (7, gated on #000062), embeddable widgets (8) — Dav1d-audience rule. **F-3** upstream prereqs — phase 7 (circuit/activation) consumes **#000062 Mechanistic Witness**'s `MechanisticWitnessRoot`; phase 3's claim-graveyard widget projects **#000059**'s bounded-ingestion graveyard. Hard constraints: arborist soft-vs-hard discipline applies verbatim (attribution weights renderable but never `audit_mode`, never causal without intervention/ablation evidence); private-leaf default-deny (commitments + hashes + redacted maps only without explicit auth); every widget exposes its data query + source roots. Reserved scope: NOT a replacement for `arborist controller-events` / `arborist analyze` / `arborist inspect` CLI — those stay canonical inspector surfaces; VIZ is the projection layer. | 2026-05-27 | — |
@@ -185,4 +186,4 @@ Newest first. Update on every open/close.
 
 ## Next ID
 
-`000072`
+`000073`

docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md

@@ -0,0 +1,163 @@
+# Ticket #000072 — Collapse legacy `query()` into the unified `run_query`
+
+Status: **open · Phase 1 foundation shipped 2026-05-31 · Phase 2 BLOCKED on
+the 5 missing rerank stages.** A multi_route smoke probe regressed on
+2/5 questions — Phase 1 infrastructure is structurally complete but
+not production-quality. Promoting `policy.multi_route=True` to default
+today would degrade the user surface.
+
+Opened: 2026-05-31  Owner: fox/blackops  Directive: cleanup
+
+## Why this exists
+
+`arborist/qa/query.py` is 4280 LOC; the `query()` function alone runs
+lines 1975-4196 (2200 LOC, single function). It is the local-CLI
+entry point for `arborist query` — every retrieval, rerank, cache
+lookup, audit-chain append, run-DAG build, and verifier kwarg threads
+through it. It also picks the WRONG primary source on the dinosaur-
+extinction smoke fixture: `Edwina, the Dinosaur Who Didn't Know She
+Was Extinct` (a children's book) instead of `Dinosaur` (the encyclopedia
+article). The cloud-via-slim-FTS5 path landed in d9fb6a9 already picks
+`Dinosaur` correctly through the unified `run_query` orchestrator.
+Collapsing legacy `query()` to delegate to the same orchestrator
+would fix that retrieval bug as a byproduct and delete ~1500 LOC of
+duplicated retrieval/rerank logic.
+
+## What shipped — Phase 1 foundation (2026-05-31)
+
+Ten incremental commits, each behavior-additive, each with the
+byte-identity gate (step 5) staying green:
+
+| commit | step | what landed |
+|---|---|---|
+| 9ba6317 | 1 | `arborist/qa/source_roles.py` — `classify_source_role`, weights, title-marker tuples extracted from query.py |
+| a83e47b | 2 | `arborist/qa/_text_norm.py` gains canonical `stem_for_match`; query.py + corpus.py both import |
+| 056d785 | 3 | `arborist/qa/retrieval_routes.py` with `filter_by_title_relevance` (5-accept-path filter); query.py re-exports |
+| e322bbd | 4 | `Corpus` protocol gains `core_keyword_match` + `doc_body`; SqliteShard implements; SidecarBucket raises `NotSupportedError` |
+| 1822741 | 5 | `tests/test_run_query_byte_identity.py` + `tests/fixtures/byte_identity/claim_lattice.json` — pins hash inputs as Phase 2 safety gate |
+| 72d1117 | 6a | `run_query` gains `policy=` kwarg threading 12 verifier kwargs |
+| 03f248c | 6b | role-classified + role-weighted budget when policy provided |
+| 5fdd573 | 6c | multi-route retrieval (body+title+phrase+core_keyword merge) when `policy.multi_route=True` |
+| 6c2ec1b | 6c-fil | wire `filter_by_title_relevance` into multi_route path |
+| b8bd9d6 | 6d | wikitext-strip when `policy.base_version` is set |
+| 20faae0 | P2/1 | `arborist/qa/providence_query.py` — cache-aware wrapper around `run_query` (skeleton; cache persist not yet done) |
+
+Every commit ships with `policy=None` preserving pre-step byte-identity
+behavior — existing `arborist corpus-query` and `arborist cloud query`
+callers are unaffected. 264 tests in the
+query/corpus/sidecar/wallet/bucket/claim_lattice/byte_identity gate
+pass at HEAD.
+
+## What's broken — the multi_route smoke regression (2026-05-31)
+
+Retrieval-only smoke against the 4-shard wikipedia corpus (no LLM,
+StubClient), comparing `policy=None` (body-only) vs
+`policy={"multi_route": True}` (body+title+phrase+core_keyword merged
+through `filter_by_title_relevance`):
+
+| Q | no-policy | multi_route | verdict |
+|---|---|---|---|
+| Soviet Union dissolve | Soviet Union | Soviet Union | preserved |
+| Mount Kilimanjaro | Mount Kilimanjaro | Mount Kilimanjaro | preserved |
+| Mona Lisa painter | Mona Lisa | Mona Lisa | preserved |
+| Mercury Seven astronauts | **Mercury Seven** | **Sam T. Beddingfield** | **REGRESSED** |
+| Dinosaurs extinct | **Dinosaur** | **Paul Austin Kelly** | **REGRESSED** |
+
+Diagnosis: the phrase route (`fts_phrase` on 4-grams from the
+question) surfaces oddly-titled stubs whose body contains the
+literal question phrase. The 5-accept-path filter's path-4
+(`phrase_match_roots`) lets them through with no body-density
+gate, and `apply_title_boost` doesn't rerank them out because their
+titles share zero tokens with the question.
+
+Legacy `query()` suppresses these noisy phrase hits via downstream
+stages that Phase 1 has NOT ported:
+  1. `_body_density_passes` — co-occurrence breadth check on
+     full doc body (3+ token queries require N-1 of N tokens
+     present at ≥3 mentions each)
+  2. `_rerank_by_body_coverage` — sqrt-rerank that counters BM25
+     short-doc bias
+  3. `_rerank_by_source_role` — role-weighted score multiplier
+     (primary 2.0×, noisy 0.3×)
+  4. `_rerank_by_title_purity` — `(1+overlap)*(1+purity)` boost
+  5. `_rerank_by_ordered_token_match` — longest-ordered-subsequence
+     match between question tokens and title tokens
+
+## Phase 2 — BLOCKED until the 5 stages above land
+
+Phase 2 was meant to land:
+  - `providence_query.py` cache wrapper (skeleton done, persist
+    deferred)
+  - Legacy `query()` collapsed to a thin adapter (50-line shell that
+    delegates to `providence_query`)
+  - Bench parity proof on smoke + bench/qa_questions.txt themed subsets
+
+But Phase 2 cannot ship without the 5 missing rerank stages — the
+new orchestrator currently produces WORSE primary picks than legacy
+on 2/5 smoke questions. Promoting it to default would degrade the
+user surface. Fox decision 2026-05-31: defer Phase 2; do NOT promote
+multi_route to default until A lands.
+
+## Path A — port the missing rerank stages (multi-day)
+
+Estimated scope:
+  1. `retrieval_routes.body_density_passes(corpus, hit, qtokens, *,
+     min_mentions=3)` — Corpus exposes `doc_body(document_root)` since
+     step 4 (e322bbd); fn already shaped for hits that carry
+     `document_root`. ~30 LOC.
+  2. `retrieval_routes.rerank_by_body_coverage(hits, question, corpus)`
+     — uses `corpus.doc_body` per hit; sqrt-rerank multiplier.
+     ~60 LOC.
+  3. `retrieval_routes.rerank_by_source_role(hits, question)` — uses
+     `classify_source_role` from source_roles.py + RANK_WEIGHTS.
+     ~30 LOC.
+  4. `retrieval_routes.rerank_by_title_purity(hits, question, *,
+     shards_dir=None)` — lifts the (1+overlap)*(1+purity) shape with
+     stem-aware matching; synonym_expand_strict gated on shards_dir.
+     ~80 LOC.
+  5. `retrieval_routes.rerank_by_ordered_token_match(hits, question)`
+     — longest-common-subsequence over query tokens vs title tokens.
+     ~70 LOC.
+  6. `run_query` policy-gated chain that invokes them in order:
+     filter → body_density → body_coverage → source_role → title_purity
+     → ordered_token → apply_title_boost (existing tail). ~50 LOC.
+  7. Re-bench: smoke (must hit 5/5), bench/qa_questions.txt themed
+     subsets (no >5pp drop vs legacy).
+  8. Only after bench-parity: continue Phase 2 cache persist +
+     legacy collapse.
+
+Risk: byte-identity gate (step 5) catches cache_key drift, but
+SEMANTIC drift (different primary source picks) requires the bench
+gate. Legacy bench data in `bench/three_way_results/*.jsonl` is the
+reference baseline.
+
+## Path B (chosen 2026-05-31)
+
+Stop here. Document the regression. Leave `policy.multi_route=False`
+as the default behavior (which it is — `policy=None` is default and
+preserves body-only retrieval). Phase 1 infrastructure stays in tree
+as the FOUNDATION for a future A-path effort; it is not the
+production user surface today.
+
+Practical implications:
+  - `arborist query` (legacy) keeps producing the same answers it
+    always has, INCLUDING the wrong "Edwina" pick on dinosaur Q5.
+  - `arborist corpus-query` and `arborist cloud query` keep
+    producing the better answers via the unified `run_query` path
+    (no-policy mode, which IS already shipped).
+  - `providence_query` exists as a skeleton callers can opt into but
+    is incomplete (cache persist missing).
+  - No commits revert; the Phase 1 work is genuinely useful even
+    without Phase 2.
+
+## When to reopen
+
+When fox wants to either:
+  - Take a focused multi-day session to land Path A
+  - Investigate a specific bench regression that suggests the
+    Phase 1 foundation is itself wrong (vs. just incomplete)
+
+Until then this ticket sits open as the gap-tracker. Pattern from
+CLAUDE.md: "An open ticket whose code already shipped is a stale
+map" — but here the OPEN-ness is the point. Phase 1 shipped, Phase 2
+hasn't. The ticket records the bridge that wasn't built yet.