qa/corpus: shared title-boost (extras penalty) lifted into run_query

The Hank-Scorpio-vs-Mr.-Burns problem: body BM25 alone outranks the
canonical primary-source article ("Homer Simpson") below sibling
articles that incidentally mention the same tokens. The sidecar
pipeline already had title-boost-with-extras-penalty; lift it into
a shared utility that the orchestrator applies to ANY adapter's
fts_body output.

  arborist/qa/corpus.py:apply_title_boost(hits, query, *,
                                          higher_is_better)
    Stems both sides (possessive + plural collapse), numeral-expands
    (7↔VII), accent-folds (é→e). Effective bonus per hit:
      max(0, overlap - extras/2) * boost
    where extras = title tokens NOT in query. Score direction honors
    each adapter's convention (BM25 negative → subtract; sidecar BM25
    positive → add).

  arborist/qa/corpus_query.py:run_query
    Now oversamples fts_body by 4× and reranks via apply_title_boost
    before slicing to top_k. Without oversampling the right primary
    can sit at rank 7-15 in the body BM25 output and get cut before
    the rerank sees it.

bench/three_way_bench.py: NEW — drives same fixture through
  - `arborist query` (legacy 2000-line pipeline)
  - `arborist corpus-query` (new, local shards via Corpus + run_query)
  - `arborist cloud query` (new, sidecar via Corpus + run_query)
  prints 3-column table + flags primary-source disagreements.

Smoke fixture (5 questions, all paths):
  4/5 all-paths agree on primary source (was 2/5 pre-fix)
  1/5 disagrees: dinosaur-extinction
    legacy:  UNGROUNDED · Edwina (children's book)   ← wrong
    corpus:  STRICT     · Dinosaur (main article)    ← right
    cloud:   HYBRID     · Edwina (children's book)   ← wrong
    The new title-boost lifted corpus-query above the legacy here;
    cloud still picks Edwina because SidecarReader.search applies
    title-boost INTERNALLY (it predates the shared util), so the
    extras-penalty stacks weirdly when run_query applies it again.
    Next fix: disable internal boost in SidecarReader once shared
    util is the single source of truth.

26 corpus/wallet/sidecar tests still green.
This commit is contained in:
russell@unturf.com 2026-05-31 08:43:24 -04:00
parent 510e05b979
commit 2b8777c947
No known key found for this signature in database
6 changed files with 319 additions and 4 deletions

View file

@ -41,6 +41,97 @@ class NotSupportedError(NotImplementedError):
backend. The orchestrator should catch and skip never propagate."""
# ---------------------------------------------------------------------------
# Shared retrieval-quality post-processor: title-boost with extras penalty.
# ---------------------------------------------------------------------------
def apply_title_boost(
hits: list["Hit"],
query: str,
*,
higher_is_better: bool,
boost: float = 8.0,
) -> list["Hit"]:
"""Re-rank hits by query-vs-title token overlap, penalized by extras.
Without this, body BM25 alone consistently outranks the canonical
primary-source article ("Homer Simpson") below sibling articles
("You Only Move Twice", "Fat Tony") that incidentally mention the
same tokens. The sidecar pipeline already does this on construction;
here we lift it into a shared utility so SqliteShardCorpus +
SidecarBucketCorpus + future adapters all benefit through the
same orchestrator step.
Score direction:
- higher_is_better=True (sidecar BM25): boost ADDED to score
- higher_is_better=False (FTS5 BM25): boost SUBTRACTED from score
Effective boost magnitude:
max(0, overlap - extras/2) * boost
where overlap = |query_stems title_stems|
and extras = |title_stems - query_stems|
Stems strip possessive apostrophes + trailing-s plurals; both
sides are numeral-expanded (7VII) and accent-folded (ée).
"""
if not hits or not query.strip() or boost <= 0:
return hits
# Lazy-import: sidecar carries the tokenizer + numeral_expand;
# importing at module top would loop on the from-sidecar imports.
from arborist.wallet.sidecar import (
_WORD_RE, STOPWORDS,
fold_accents, numeral_expand, tokenize_text,
)
def _stem(t: str) -> str:
t = t.replace("'", "").replace("", "")
if len(t) > 4 and t.endswith("s") and not t.endswith("ss"):
return t[:-1]
return t
query_stems = numeral_expand({_stem(t) for t in tokenize_text(query)})
if not query_stems:
return hits
rescored: list[Hit] = []
for h in hits:
title = fold_accents((h.title or "").lower())
raw_title_tokens: set[str] = set()
for tok in _WORD_RE.findall(title):
if tok in STOPWORDS:
continue
if len(tok) <= 1 and not tok.isdigit():
continue
raw_title_tokens.add(_stem(tok))
title_tokens = numeral_expand(raw_title_tokens)
overlap = len(query_stems & title_tokens)
if not overlap:
rescored.append(h)
continue
extras = len(title_tokens - query_stems)
effective = max(0.0, overlap - extras / 2.0)
if effective <= 0:
rescored.append(h)
continue
delta = effective * boost
new_score = h.score + delta if higher_is_better else h.score - delta
rescored.append(Hit(
document_root=h.document_root,
document_uri=h.document_uri,
title=h.title,
score=new_score,
shard_id=h.shard_id,
extras={**h.extras, "title_boost_delta": round(delta, 3)},
))
# Re-sort by score in the adapter's convention.
rescored.sort(
key=lambda h: h.score,
reverse=higher_is_better,
)
return rescored
@dataclass(frozen=True)
class Hit:
"""One retrieval result, backend-agnostic.

View file

@ -31,7 +31,7 @@ from typing import Optional
from arborist.compress import unpack_chunk
from arborist.qa.client import ChatClient
from arborist.qa.corpus import Corpus, NotSupportedError
from arborist.qa.corpus import Corpus, NotSupportedError, apply_title_boost
from arborist.qa.evidence import (
build_evidence_map,
evidence_map_by_pointer_id,
@ -78,13 +78,23 @@ def run_query(
t_start = _time.time()
# 1. Retrieve — fts_body is the only route currently shared across
# adapters. Future: add fts_title / fts_phrase / core_keyword
# with NotSupported skip-and-merge logic.
# adapters. Oversample 4× so the post-retrieval title-boost has
# more candidates to rerank (without oversampling, the right
# primary-source article can be at rank 7-15 in body BM25 output
# and get cut before reranking sees it). Future: add fts_title /
# fts_phrase / core_keyword with NotSupported skip-and-merge.
ts = _time.time()
try:
hits = corpus.fts_body(question, limit=top_k)
hits = corpus.fts_body(question, limit=top_k * 4)
except NotSupportedError:
hits = []
# Shared title-boost rerank — lifts "Homer Simpson" main article
# above "You Only Move Twice" sibling on the homer query, etc.
# Same logic for SqliteShardCorpus + SidecarBucketCorpus.
hits = apply_title_boost(
hits, question,
higher_is_better=getattr(corpus, "higher_is_better", False),
)[:top_k]
timings["search"] = _time.time() - ts
if not hits:

View file

@ -0,0 +1,5 @@
{"question": "when did the soviet union dissolve?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 6.38, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 47.45, "error": null}, "regression": false, "reason": "ok"}
{"question": "where is mount kilimanjaro located?", "local": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 5.08, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 54.33, "error": null}, "regression": false, "reason": "ok"}
{"question": "who painted the mona lisa?", "local": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa", "used": true}, "elapsed_s": 5.89, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa", "used": true}, "elapsed_s": 47.75, "error": null}, "regression": false, "reason": "ok"}
{"question": "who were the original seven mercury astronauts?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven", "used": true}, "elapsed_s": 7.0, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_quotes": 8, "n_verified": 8, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven", "used": true}, "elapsed_s": 53.95, "error": null}, "regression": true, "reason": "audit_mode dropped STRICT → HYBRID"}
{"question": "why did the dinosaurs go extinct?", "local": {"audit_mode": "UNGROUNDED", "n_quotes": 1, "n_verified": 0, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct", "used": false}, "elapsed_s": 5.81, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_quotes": 4, "n_verified": 1, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct", "used": true}, "elapsed_s": 58.5, "error": null}, "regression": false, "reason": "ok"}

View file

@ -0,0 +1,2 @@
{"question": "when did the soviet union dissolve?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 4.64, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 56.16, "error": null}, "regression": false, "reason": "ok"}
{"question": "where is mount kilimanjaro located?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 3.71, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 56.46, "error": null}, "regression": false, "reason": "ok"}

202
bench/three_way_bench.py Normal file
View file

@ -0,0 +1,202 @@
"""3-way bench: legacy local vs corpus-query vs cloud-query.
Goal: every fixture question produces the SAME primary-source citation
across all three paths. Diverging primary-source picks are real
regressions even when audit_mode happens to agree.
legacy arborist --shards-dir query --json --burn --answer-mode claim_lattice ...
corpus-query arborist --shards-dir corpus-query --json ... (Corpus run_query)
cloud-query arborist cloud query --bucket-url <manifest> --json ... (Sidecar run_query)
JSONL output: one row per question with all three path results +
agreement flags so iterations can compare across bench runs.
"""
from __future__ import annotations
import argparse
import datetime as _dt
import json
import os
import subprocess
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent
ARBORIST = REPO / ".venv" / "bin" / "arborist"
DEFAULT_SHARDS = Path.home() / ".arborist" / "shards"
DEFAULT_BUCKET = (
"https://nyc3.digitaloceanspaces.com/arborist/clones/manifest-sidecar.json"
)
DEFAULT_QWEN_ENDPOINT = "https://qwen.ai.unturf.com/v1"
DEFAULT_QWEN_MODEL = "Qwen3.6-27B-UD-Q4_K_XL.gguf"
def _load_fixture(path: Path) -> list[str]:
out = []
for line in path.read_text().splitlines():
s = line.strip()
if not s or s.startswith("#"):
continue
out.append(s)
return out
def _run(cmd: list[str], *, timeout_s: int) -> dict:
t0 = time.time()
try:
proc = subprocess.run(
cmd, capture_output=True, text=True, timeout=timeout_s,
env={**os.environ, "ARBORIST_PROGRESS": "0"},
)
except subprocess.TimeoutExpired:
return {"_error": f"timeout {timeout_s}s", "_elapsed_s": timeout_s}
dt = time.time() - t0
if proc.returncode != 0:
return {"_error": f"exit {proc.returncode}",
"_stderr": proc.stderr[:300], "_elapsed_s": round(dt, 2)}
try:
d = json.loads(proc.stdout)
except json.JSONDecodeError as e:
return {"_error": f"parse: {e}", "_stdout_head": proc.stdout[:200],
"_elapsed_s": round(dt, 2)}
d["_elapsed_s"] = round(dt, 2)
return d
def _legacy(q, *, shards_dir):
return _run([
str(ARBORIST), "--shards-dir", str(shards_dir),
"query", "--top-k", "8", "--json", "--burn",
"--answer-mode", "claim_lattice", q,
], timeout_s=120)
def _corpus(q, *, shards_dir, endpoint, model):
return _run([
str(ARBORIST), "--shards-dir", str(shards_dir),
"corpus-query", q,
"--top-k", "4", "--json",
"--endpoint", endpoint, "--model", model,
], timeout_s=120)
def _cloud(q, *, bucket_url, endpoint, model):
return _run([
str(ARBORIST), "cloud", "query", q,
"--bucket-url", bucket_url,
"--endpoint", endpoint, "--model", model,
"--top-k", "4", "--json",
], timeout_s=240)
def _primary(r: dict) -> dict:
srcs = r.get("sources") or []
if not srcs:
return {"title": "", "uri": ""}
pri = next(
(s for s in srcs if s.get("source_role") == "primary_answer_source"),
srcs[0],
)
return {"title": (pri.get("title") or "")[:55],
"uri": pri.get("document_uri") or ""}
def main():
p = argparse.ArgumentParser()
p.add_argument(
"--fixture", type=Path,
default=REPO / "bench" / "qa_questions_smoke.txt",
)
p.add_argument("--shards-dir", type=Path, default=DEFAULT_SHARDS)
p.add_argument("--bucket-url", default=DEFAULT_BUCKET)
p.add_argument("--endpoint", default=DEFAULT_QWEN_ENDPOINT)
p.add_argument("--model", default=DEFAULT_QWEN_MODEL)
p.add_argument(
"--out-dir", type=Path,
default=REPO / "bench" / "three_way_results",
)
p.add_argument("--skip-legacy", action="store_true")
p.add_argument("--skip-corpus", action="store_true")
p.add_argument("--skip-cloud", action="store_true")
args = p.parse_args()
questions = _load_fixture(args.fixture)
args.out_dir.mkdir(parents=True, exist_ok=True)
stamp = _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H-%M-%SZ")
out_path = args.out_dir / f"{stamp}.jsonl"
print(f"# {len(questions)} questions from {args.fixture}", file=sys.stderr)
print(f"# shards: {args.shards_dir}", file=sys.stderr)
print(f"# bucket: {args.bucket_url}", file=sys.stderr)
print(f"# llm: {args.endpoint} / {args.model}", file=sys.stderr)
print(file=sys.stderr)
n_full_agreement = 0
n_source_disagree = 0
with open(out_path, "w") as f:
for i, q in enumerate(questions, 1):
print(f"[{i}/{len(questions)}] {q}", file=sys.stderr)
legacy = {} if args.skip_legacy else _legacy(q, shards_dir=args.shards_dir)
corpus = {} if args.skip_corpus else _corpus(
q, shards_dir=args.shards_dir,
endpoint=args.endpoint, model=args.model,
)
cloud = {} if args.skip_cloud else _cloud(
q, bucket_url=args.bucket_url,
endpoint=args.endpoint, model=args.model,
)
row = {"question": q}
for label, r in (("legacy", legacy), ("corpus", corpus), ("cloud", cloud)):
row[label] = {
"audit_mode": r.get("audit_mode"),
"n_verified": r.get("n_verified"),
"n_quotes": r.get("n_quotes"),
"primary": _primary(r),
"elapsed_s": r.get("_elapsed_s"),
"error": r.get("_error"),
}
# Agreement: same primary URI across all PRESENT paths
uris = {
lbl: row[lbl]["primary"]["uri"]
for lbl in ("legacy", "corpus", "cloud")
if row[lbl]["primary"]["uri"]
}
distinct = set(uris.values())
row["agreement"] = {
"n_paths_with_source": len(uris),
"distinct_sources": len(distinct),
"all_agree": len(distinct) <= 1,
}
if row["agreement"]["all_agree"]:
n_full_agreement += 1
else:
n_source_disagree += 1
f.write(json.dumps(row, ensure_ascii=False) + "\n")
f.flush()
for label in ("legacy", "corpus", "cloud"):
r = row[label]
badge = "" if r["primary"]["uri"] and len(distinct) > 1 and r["primary"]["uri"] != min(uris.values()) else ""
err = f" ERR={r['error']}" if r["error"] else ""
print(f" {label:7s} {(r['audit_mode'] or '?'):<25} "
f"{r['primary']['title']}{badge}{err}",
file=sys.stderr)
if not row["agreement"]["all_agree"]:
print(f" ⚠ SOURCE DISAGREEMENT: {len(distinct)} distinct primaries",
file=sys.stderr)
print(file=sys.stderr)
print(f"=== SUMMARY ===", file=sys.stderr)
print(f" {len(questions)} questions", file=sys.stderr)
print(f" {n_full_agreement} all-paths agree on primary source", file=sys.stderr)
print(f" {n_source_disagree} paths-disagree", file=sys.stderr)
print(f" results: {out_path}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,5 @@
{"question": "when did the soviet union dissolve?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union"}, "elapsed_s": 4.51, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union"}, "elapsed_s": 40.06, "error": null}, "cloud": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union"}, "elapsed_s": 49.53, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
{"question": "where is mount kilimanjaro located?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}, "elapsed_s": 3.67, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}, "elapsed_s": 46.7, "error": null}, "cloud": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}, "elapsed_s": 52.95, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
{"question": "who painted the mona lisa?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa"}, "elapsed_s": 4.06, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa"}, "elapsed_s": 11.97, "error": null}, "cloud": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa"}, "elapsed_s": 46.6, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
{"question": "who were the original seven mercury astronauts?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven"}, "elapsed_s": 7.58, "error": null}, "corpus": {"audit_mode": "HYBRID", "n_verified": 4, "n_quotes": 4, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven"}, "elapsed_s": 33.99, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_verified": 8, "n_quotes": 8, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven"}, "elapsed_s": 55.25, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
{"question": "why did the dinosaurs go extinct?", "legacy": {"audit_mode": "UNGROUNDED", "n_verified": 0, "n_quotes": 1, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct"}, "elapsed_s": 7.05, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Dinosaur", "uri": "https://en.wikipedia.org/wiki/Dinosaur"}, "elapsed_s": 42.81, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_verified": 1, "n_quotes": 4, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct"}, "elapsed_s": 115.13, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 2, "all_agree": false}}