From 4098e41563d7e1aec03310b5b04c895da00e0f2f Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 31 May 2026 13:06:12 -0400 Subject: [PATCH] #000072: open ticket for legacy query() collapse + multi_route regression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the work shipped this session (Phase 1 foundation, 10 commits 9ba6317..20faae0) and the blocker that stops Phase 2: the multi_route pipeline regresses on 2/5 smoke fixture questions (Mercury Seven → Sam T. Beddingfield; dinosaurs extinct → Paul Austin Kelly) because Phase 1 hasn't ported the 5 downstream rerank stages legacy query() uses to suppress noisy phrase-route hits — body-density check, body-coverage sqrt rerank, source-role rerank, title-purity rerank, ordered-token-match rerank. Path B chosen 2026-05-31: stop here, leave multi_route off by default (which IS default — policy=None preserves body-only behavior). Phase 1 foundation stays in tree as future-ready infrastructure; user surface unchanged. Path A (port the 5 stages, multi-day effort) reserved for a future focused session. Until then, legacy query() keeps producing the same answers it always has, INCLUDING the wrong "Edwina" pick on dinosaur Q5. The slim-FTS5 cloud path already fixes that bug for `arborist cloud query` / `arborist corpus-query` callers (proven 5/5 source parity in d9fb6a9). Bumps Next ID 000072 → 000073. --- docs/TICKETS.md | 3 +- ...72-collapse-legacy-query-into-run-query.md | 163 ++++++++++++++++++ 2 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 5dfbbd2..c7c72de 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -111,6 +111,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| +| #000072 | Collapse legacy `query()` into the unified `run_query` orchestrator | **open · Phase 1 foundation shipped 2026-05-31 (10 commits) · Phase 2 BLOCKED on 5 missing rerank stages** — multi_route smoke probe regressed on 2/5 questions (Mercury Seven → Sam T. Beddingfield; dinosaurs → Paul Austin Kelly), so Phase 1 is structurally complete but not production-quality. Phase 1 shipped: `source_roles.py` extraction (9ba6317), canonical `stem_for_match` in `_text_norm.py` (a83e47b), `retrieval_routes.py` with `filter_by_title_relevance` (056d785), Corpus protocol gains `core_keyword_match` + `doc_body` (e322bbd), byte-identity test fixture as Phase 2 safety gate (1822741), `run_query` gains `policy=` kwarg threading 12 verifier kwargs (72d1117), role-classified + role-weighted budget (03f248c), multi-route retrieval body+title+phrase+core_keyword merge (5fdd573), filter wired into multi_route path (6c2ec1b), wikitext-strip when `policy.base_version` set (b8bd9d6), `providence_query.py` cache-wrapper skeleton with cache persist deferred (20faae0). Every commit ships with `policy=None` preserving pre-step byte-identity behavior — existing `arborist corpus-query` and `arborist cloud query` callers are unaffected; the slim-FTS5 cloud path already wins on the dinosaur Q5 fixture (picks "Dinosaur" not legacy's "Edwina"). Phase 2 blocked because Phase 1's `multi_route` pipeline lacks the 5 downstream rerank stages legacy uses to suppress noisy phrase-route hits: body-density check (`_body_density_passes`), body-coverage sqrt rerank (`_rerank_by_body_coverage`), source-role rerank (`_rerank_by_source_role` with RANK_WEIGHTS), title-purity rerank ((1+overlap)*(1+purity)), ordered-token-match rerank (longest-common-subsequence). **Path A** = port the 5 stages → re-bench → resume Phase 2 cache persist + legacy-query() collapse to 50-line adapter. **Path B chosen 2026-05-31** = stop here, defer Phase 2, leave multi_route off by default (which IS the default — `policy=None` preserves body-only retrieval). Until Path A: Phase 1 foundation stays in tree as future-ready infrastructure; user surface unchanged (legacy `query()` still produces same answers, INCLUDING the wrong "Edwina" pick on dinosaur Q5). Full spec + commit log + bench data + the missing-rerank-stages inventory in `docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md`. | 2026-05-31 | — | | #000071 | World-bridge grammar (Joseph 3-bridge inter-chain framework) | **open · awaiting dav1d review + fox go/no-go · doc-only Phase 1** (2026-05-31; sibling to #000070, opened same day after Joseph @TrudoJo posted his bridge-framework extension to the 6-dim spatial-ontology — "chain A has its own world rules, chain B has its own world rules, a bridge exists only where a deterministic agreement layer can be proven"). Three bridge kinds: **Agreement** (rules match, direct passage), **Translation** (rules differ but a hash-pinned adapter exists), **Embassy** (foreign-object representation with limited rights). Bridge identity: `bridge_seed = hash(src_chain_id ‖ src_node ‖ src_rule_v ‖ tgt_chain_id ‖ tgt_rule_v ‖ translator_id ‖ invariant_set)`. Joseph's sovereignty rule (hard): chain law remains local, bridge law governs translation only, **no bridge may overwrite native chain law** — guard against forced-unity failure mode. **Primary contribution: retroactive-validation appendix** naming three already-shipped arborist primitives as Joseph-bridge-shape patterns one-for-one — **#000066 graft-receipt mode = Agreement bridge**, **#000056 Operation Sandwich = Translation bridge** (opus-mt hash IS the translator_id), **#000059 bounded-ingestion graveyard = Embassy bridge** (POINTER-LINKED only, never promotes, banner-labelled). The framework is not a new requirement; it's a unifying spec that names what arborist already does in 3+ places, plus surfaces a fourth pattern (bridge atlas for multi-instance federation with potentially-different `governance_policy_hash` peers) as genuinely new work for `arborist/mesh/`. **Phase 1 is doc-only** — paper extension §7 in `docs/_source/merkle-agi-v7w-spatial-temporal.rst` + `arborist/world/bridge/__init__.py` namespace stub mirroring `arborist/world/__init__.py` discipline + Joseph-memory extension. No kernel code, no SQL change, no `cache_key` extension, no new `audit_mode` token, no bench (doc-only review surface). Phase 2 = `pi_star/bridge-seed@v1` canonicalization deliberately deferred to avoid premature locking before three implementation tickets opine. Phase 3 = per-bridge-type kernels + bridge atlas + `event_type='bridge'` writer + mesh integration. Phase 4 = cross-domain bridges (language ↔ spatial via #000015 composition theorem). **Ten open questions for dav1d** in §8: doc-only vs registry-slot-at-Phase-1 (Q1), chain_id = governance_policy_hash vs manifest hash (Q2), three-bridge taxonomy completeness (Q3), translator integrity hashing depth (Q4), bridge witness via `event_type='bridge'` (Q5), `audit_mode` discipline for translated objects (Q6), bridge privacy class default (Q7), endianness reconfirmation (Q8), Phase 2 KAT count + adversarial vectors (Q9), paper-amendment wording (Q10). Hard constraints: A1–A3, no continuous tensors, no `audit_mode` token, no SQL change at Phase 1, topic-named under `arborist/world/bridge/`. Full spec in `docs/tickets/ticket-000071-world-bridge-grammar.md`. | 2026-05-31 | — | | #000070 | Spatial-anchor π*_w_object (Joseph 6-dim determination kernel) | **open · awaiting dav1d review + fox go/no-go · pre-review bench shipped `55b651f`** — pure-stdlib empirical-validation script (`bench/spatial_anchor_validation.py`, `make bench-spatial-anchor`, ~2 s, RNG-seed-pinned) measures: §1 avalanche mean 767.85 bits vs PRF null 768 (z = -0.49) · §2 octree chi² uniformity \|z\| < 1 at L=2,3,4 · §3 collision birthday-bound ratio 0.989 / 1.038 at L=4,6 · §4 cross-region Pearson r in [-0.018, +0.012] across all 15 pairs of H₁..H₆ (none flag at 4σ) · §5 domain separation Arm A 767.91 bits independent / Arm B **0.00 bits exact collision** confirming the dedicated-`spatial_anchor_seed` discipline. Five of ten open questions (Q1 seed source · Q2 segmentation · Q3 octree mapper · Q8 endianness · Q9 KAT adversarial vectors) now resolve by measurement; Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions for dav1d. Full empirical detail in ticket §7a. Originally surfaced 2026-05-31 when fox brought in Joseph @TrudoJo's procedural-spatial framework — "hashes do not encode the world, hashes determine the world" — and asked whether it fits arborist's substrate model. It does, cleanly. Implements the first verifier kernel under the #000013 v7-W reserved namespace (`arborist/world/__init__.py` `STATUS = "namespace_reserved"` → `kernel_in_progress` on land): a single committed 32-byte SHA-256 hash deterministically expands into six named 32-byte regions `H₁..H₆` via the HMAC-SHA-512 KDF already shipped in `arborist/substrate/anchor_prg.py` (#000035), and each region drives one quantized object dimension under a fixed canonical mapper — `H₁→octree position`, `H₂→scale level`, `H₃→quantized SO(3) rotation`, `H₄→material palette`, `H₅→raw links (reserved for π*_w_relation sibling ticket)`, `H₆→behavior code`. Domain separation from #000035 by dedicated `spatial_anchor_seed` (manifest-published) — keeps #000035's KAT freeze + dav1d 2026-05-11 final review intact. Hard constraints: stays inside A1–A3, no continuous tensors in proof path, no new `audit_mode` token (substrate commitments are not warrants — the four-rung ladder is unchanged), no SQL schema change at Phase 1, `canonicalization_version` absorbs the new `spatial-anchor-object@v1` registry slot. Phase 1 deliverable ~150 LoC + 10 KATs + tests in one PR: `arborist/substrate/spatial_anchor.py` (segmentation), `arborist/world/pi_star/object.py` (six mappers), registry entry, KAT vectors at `bench/fixtures/spatial-anchor/known-answer-tests.jsonl` matching #000035 KAT discipline, `tests/test_spatial_anchor.py` + `tests/test_world_pi_star_object.py`, substrate-paper amendment citing Joseph (@TrudoJo) in `docs/_source/merkle-agi-v7w-spatial-temporal.rst`. Position mapper at Phase 1 = octree (substrate paper §2.1 + §A worked example); H3/S2/Hilbert/Morton siblings deferred — Morton's value is purely as the relation-kernel `pair(A,B)` cheap bit-interleave and surfaces in the sibling ticket. **Ten open questions for dav1d** in §8: seed-source choice, segmentation method, position mapper, scope split (object alone vs object+relation per CLAUDE.md memory `feedback_ticket_proliferation`), privacy-class fail-closed default, `audit_mode` discipline confirmation, manifest-validator timing, endianness reconfirmation, KAT count + adversarial vectors, paper-amendment wording. Five-step §7 deletions captured: dropped the prior `/tmp/arborist-spatial-ontology-plan.md`'s geographic-search backend (encoding-route confusion), `Document.extra` lat/lon hooks (different ticket if at all), `audit_mode=HYBRID` for spatial hits (convention-illegal), geohash (subsumed by Morton), quadtree as separate (octree at z-level-0), Hilbert at Phase 1 (deferred), tier-1/2/3 hash-suite framing (conflated hard vs soft hashes), and `arborist/spatial/` namespace (violates topic-naming rule). Full spec in `docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md`. | 2026-05-31 | — | | #000069 | Arborist VIZ / Merkle Command Center (Pyramid + six.js + SSE browser dashboard) | **open · awaiting go/no-go · doc-only scaffold** (2026-05-27; filed from `/home/fox/Downloads/TICKET_0000VIZ_*`, stack corrected same day per fox). Configurable browser dashboard for inspecting arborist's content-addressed state: Merkle root explorer, proof verifier, claim warrant + graveyard, audit timeline, run-DAG replay, cache-key explainer, root diff, 3D Merkle lattice, optional circuit/activation traces. Read-only consumer; arborist proper stays source-of-truth, dashboard projects state. **Stack pinned to unturf-native** (fox 2026-05-27, supersedes proposal §3): **Pyramid + Jinja2 + SQLAlchemy** (matches `remarkbox` / `make_post_sell` / `unhomeschool.com` idiom), **SSE** (`text/event-stream` via Pyramid streaming response) for live audit/claim/falsifier patches, **vanilla JS + six.js** (fox's patched three.js fork at `git.unturf.com/gumyum/six.js` — three.js r175 + CWE-407 patches incl. ObjectBVH O(N)→O(log N); bundles vendored from `~/git/cupPCB/cdn/six/`; third-instance MOAD-0001 dogfood alongside `java-topology` + gumyum-engine) for 3D widgets and large-graph rendering, SQLite for dashboard metadata (no PostgreSQL/ClickHouse/Redis/NATS by default — promote on measured need), no React / no Next.js / no Node build step. Server-rendered SVG (or Graphviz `.dot` per existing `docs/diagrams/*.dot` pattern) replaces React Flow for run-DAG widgets. Browser-side proof verification dropped from v1 (server-side Pyramid view returns PASS/FAIL + receipt; reinstate phase-N only if third-party-verification use case surfaces). **Three filing-note gates before phase 0** (in ticket body): **F-1** sibling-repo home — implementation lives in a new `~/git/arborist-viz` (Pyramid Python, matches existing unturf apps), not in-tree; arborist's contribution is the read-API spec + view package + arborist library import via `arborist.embed`. **F-2** scope split — proposal carries 8 phases (§17 phases 0–8); recommended cut keeps phases 0–3 (schema + shell + proof/root widgets + claim/audit/run widgets) inside #000069, and spawns sibling tickets for SSE streaming (4), 3D six.js (5), massive-graph (6, only if measured need surfaces), circuit-tracing (7, gated on #000062), embeddable widgets (8) — Dav1d-audience rule. **F-3** upstream prereqs — phase 7 (circuit/activation) consumes **#000062 Mechanistic Witness**'s `MechanisticWitnessRoot`; phase 3's claim-graveyard widget projects **#000059**'s bounded-ingestion graveyard. Hard constraints: arborist soft-vs-hard discipline applies verbatim (attribution weights renderable but never `audit_mode`, never causal without intervention/ablation evidence); private-leaf default-deny (commitments + hashes + redacted maps only without explicit auth); every widget exposes its data query + source roots. Reserved scope: NOT a replacement for `arborist controller-events` / `arborist analyze` / `arborist inspect` CLI — those stay canonical inspector surfaces; VIZ is the projection layer. | 2026-05-27 | — | @@ -185,4 +186,4 @@ Newest first. Update on every open/close. ## Next ID -`000072` +`000073` diff --git a/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md b/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md new file mode 100644 index 0000000..7759b68 --- /dev/null +++ b/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md @@ -0,0 +1,163 @@ +# Ticket #000072 — Collapse legacy `query()` into the unified `run_query` + +Status: **open · Phase 1 foundation shipped 2026-05-31 · Phase 2 BLOCKED on +the 5 missing rerank stages.** A multi_route smoke probe regressed on +2/5 questions — Phase 1 infrastructure is structurally complete but +not production-quality. Promoting `policy.multi_route=True` to default +today would degrade the user surface. + +Opened: 2026-05-31 Owner: fox/blackops Directive: cleanup + +## Why this exists + +`arborist/qa/query.py` is 4280 LOC; the `query()` function alone runs +lines 1975-4196 (2200 LOC, single function). It is the local-CLI +entry point for `arborist query` — every retrieval, rerank, cache +lookup, audit-chain append, run-DAG build, and verifier kwarg threads +through it. It also picks the WRONG primary source on the dinosaur- +extinction smoke fixture: `Edwina, the Dinosaur Who Didn't Know She +Was Extinct` (a children's book) instead of `Dinosaur` (the encyclopedia +article). The cloud-via-slim-FTS5 path landed in d9fb6a9 already picks +`Dinosaur` correctly through the unified `run_query` orchestrator. +Collapsing legacy `query()` to delegate to the same orchestrator +would fix that retrieval bug as a byproduct and delete ~1500 LOC of +duplicated retrieval/rerank logic. + +## What shipped — Phase 1 foundation (2026-05-31) + +Ten incremental commits, each behavior-additive, each with the +byte-identity gate (step 5) staying green: + +| commit | step | what landed | +|---|---|---| +| 9ba6317 | 1 | `arborist/qa/source_roles.py` — `classify_source_role`, weights, title-marker tuples extracted from query.py | +| a83e47b | 2 | `arborist/qa/_text_norm.py` gains canonical `stem_for_match`; query.py + corpus.py both import | +| 056d785 | 3 | `arborist/qa/retrieval_routes.py` with `filter_by_title_relevance` (5-accept-path filter); query.py re-exports | +| e322bbd | 4 | `Corpus` protocol gains `core_keyword_match` + `doc_body`; SqliteShard implements; SidecarBucket raises `NotSupportedError` | +| 1822741 | 5 | `tests/test_run_query_byte_identity.py` + `tests/fixtures/byte_identity/claim_lattice.json` — pins hash inputs as Phase 2 safety gate | +| 72d1117 | 6a | `run_query` gains `policy=` kwarg threading 12 verifier kwargs | +| 03f248c | 6b | role-classified + role-weighted budget when policy provided | +| 5fdd573 | 6c | multi-route retrieval (body+title+phrase+core_keyword merge) when `policy.multi_route=True` | +| 6c2ec1b | 6c-fil | wire `filter_by_title_relevance` into multi_route path | +| b8bd9d6 | 6d | wikitext-strip when `policy.base_version` is set | +| 20faae0 | P2/1 | `arborist/qa/providence_query.py` — cache-aware wrapper around `run_query` (skeleton; cache persist not yet done) | + +Every commit ships with `policy=None` preserving pre-step byte-identity +behavior — existing `arborist corpus-query` and `arborist cloud query` +callers are unaffected. 264 tests in the +query/corpus/sidecar/wallet/bucket/claim_lattice/byte_identity gate +pass at HEAD. + +## What's broken — the multi_route smoke regression (2026-05-31) + +Retrieval-only smoke against the 4-shard wikipedia corpus (no LLM, +StubClient), comparing `policy=None` (body-only) vs +`policy={"multi_route": True}` (body+title+phrase+core_keyword merged +through `filter_by_title_relevance`): + +| Q | no-policy | multi_route | verdict | +|---|---|---|---| +| Soviet Union dissolve | Soviet Union | Soviet Union | preserved | +| Mount Kilimanjaro | Mount Kilimanjaro | Mount Kilimanjaro | preserved | +| Mona Lisa painter | Mona Lisa | Mona Lisa | preserved | +| Mercury Seven astronauts | **Mercury Seven** | **Sam T. Beddingfield** | **REGRESSED** | +| Dinosaurs extinct | **Dinosaur** | **Paul Austin Kelly** | **REGRESSED** | + +Diagnosis: the phrase route (`fts_phrase` on 4-grams from the +question) surfaces oddly-titled stubs whose body contains the +literal question phrase. The 5-accept-path filter's path-4 +(`phrase_match_roots`) lets them through with no body-density +gate, and `apply_title_boost` doesn't rerank them out because their +titles share zero tokens with the question. + +Legacy `query()` suppresses these noisy phrase hits via downstream +stages that Phase 1 has NOT ported: + 1. `_body_density_passes` — co-occurrence breadth check on + full doc body (3+ token queries require N-1 of N tokens + present at ≥3 mentions each) + 2. `_rerank_by_body_coverage` — sqrt-rerank that counters BM25 + short-doc bias + 3. `_rerank_by_source_role` — role-weighted score multiplier + (primary 2.0×, noisy 0.3×) + 4. `_rerank_by_title_purity` — `(1+overlap)*(1+purity)` boost + 5. `_rerank_by_ordered_token_match` — longest-ordered-subsequence + match between question tokens and title tokens + +## Phase 2 — BLOCKED until the 5 stages above land + +Phase 2 was meant to land: + - `providence_query.py` cache wrapper (skeleton done, persist + deferred) + - Legacy `query()` collapsed to a thin adapter (50-line shell that + delegates to `providence_query`) + - Bench parity proof on smoke + bench/qa_questions.txt themed subsets + +But Phase 2 cannot ship without the 5 missing rerank stages — the +new orchestrator currently produces WORSE primary picks than legacy +on 2/5 smoke questions. Promoting it to default would degrade the +user surface. Fox decision 2026-05-31: defer Phase 2; do NOT promote +multi_route to default until A lands. + +## Path A — port the missing rerank stages (multi-day) + +Estimated scope: + 1. `retrieval_routes.body_density_passes(corpus, hit, qtokens, *, + min_mentions=3)` — Corpus exposes `doc_body(document_root)` since + step 4 (e322bbd); fn already shaped for hits that carry + `document_root`. ~30 LOC. + 2. `retrieval_routes.rerank_by_body_coverage(hits, question, corpus)` + — uses `corpus.doc_body` per hit; sqrt-rerank multiplier. + ~60 LOC. + 3. `retrieval_routes.rerank_by_source_role(hits, question)` — uses + `classify_source_role` from source_roles.py + RANK_WEIGHTS. + ~30 LOC. + 4. `retrieval_routes.rerank_by_title_purity(hits, question, *, + shards_dir=None)` — lifts the (1+overlap)*(1+purity) shape with + stem-aware matching; synonym_expand_strict gated on shards_dir. + ~80 LOC. + 5. `retrieval_routes.rerank_by_ordered_token_match(hits, question)` + — longest-common-subsequence over query tokens vs title tokens. + ~70 LOC. + 6. `run_query` policy-gated chain that invokes them in order: + filter → body_density → body_coverage → source_role → title_purity + → ordered_token → apply_title_boost (existing tail). ~50 LOC. + 7. Re-bench: smoke (must hit 5/5), bench/qa_questions.txt themed + subsets (no >5pp drop vs legacy). + 8. Only after bench-parity: continue Phase 2 cache persist + + legacy collapse. + +Risk: byte-identity gate (step 5) catches cache_key drift, but +SEMANTIC drift (different primary source picks) requires the bench +gate. Legacy bench data in `bench/three_way_results/*.jsonl` is the +reference baseline. + +## Path B (chosen 2026-05-31) + +Stop here. Document the regression. Leave `policy.multi_route=False` +as the default behavior (which it is — `policy=None` is default and +preserves body-only retrieval). Phase 1 infrastructure stays in tree +as the FOUNDATION for a future A-path effort; it is not the +production user surface today. + +Practical implications: + - `arborist query` (legacy) keeps producing the same answers it + always has, INCLUDING the wrong "Edwina" pick on dinosaur Q5. + - `arborist corpus-query` and `arborist cloud query` keep + producing the better answers via the unified `run_query` path + (no-policy mode, which IS already shipped). + - `providence_query` exists as a skeleton callers can opt into but + is incomplete (cache persist missing). + - No commits revert; the Phase 1 work is genuinely useful even + without Phase 2. + +## When to reopen + +When fox wants to either: + - Take a focused multi-day session to land Path A + - Investigate a specific bench regression that suggests the + Phase 1 foundation is itself wrong (vs. just incomplete) + +Until then this ticket sits open as the gap-tracker. Pattern from +CLAUDE.md: "An open ticket whose code already shipped is a stale +map" — but here the OPEN-ness is the point. Phase 1 shipped, Phase 2 +hasn't. The ticket records the bridge that wasn't built yet.