qa: synthetic_elision sidecar diagnosis + role-weighted source budget

Two enhancements from the toy-Hermes design pass (2026-04-30):

A. Synthetic-elision-inside-quote diagnosis (sidecar only).

   Distinct from interior_elision (model dropped a `(...)` aside source
   carries) — synthetic_elision is the model writing literal `[...]`
   between fragments of a `"..."` span, signaling self-elision while
   claiming verbatim citation. The verifier still rejects (binary
   discipline holds), but `aborist inspect` now reports
   `diagnosis: synthetic_elision_inside_quote` with prefix/suffix
   presence flags so an operator can judge whether the elided middle
   was benign. Probe runs first in the classify-span chain (more
   specific than trailing_artifact / interior_elision / paraphrase).

   Catches the Brachiosaurus case: `"The film centers on the fictional
   Isla Nublar [...] Universal Studios..."` — both halves are in source,
   but the literal `[...]` isn't, so substring match correctly fails &
   the sidecar tells the operator why.

C. Source-role classification + role-weighted context budget.

   `_classify_source_role(title, qtokens_stem)` tags each top-K hit:
       primary_answer_source     2.0× cap   strong title-stem overlap
       secondary_context_source  1.0× cap   "list of", "characters",
                                            "franchise", "history of"
       noisy_background_source   0.5× cap   "score", "music",
                                            "video game", "merchandise"
       sequel_background_source  0.5× cap   "lost world", roman numerals
       background_source         1.0× cap   default

   Order matters: noisy/sequel/secondary markers fire before the
   primary check so peripheral pages with strong title overlap (e.g.
   `Jurassic Park (film score)` shares 3 stems with the JP-film query)
   don't claim a primary slot.

   Cap loop now applies role weight on top of the baseline
   `max_context_chars / top_k`. Total context still bounded by the
   running `char_budget` — weights just shift how the budget gets
   divided so primary pages get more text & noisy pages less, fixing
   the case where a `(film score)` page consumed a primary slot.

   `source_role` is persisted on `_Hit` and surfaces on
   `merkle_proof.sources[*].source_role` in the providence record so
   inspect & audits can see which slot each source occupied.

Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
  synthetic_elision_does_not_fire_when_source_has_brackets (false-
  positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
  background) on JP-film-style titles, role persistence in sources list.

455 tests pass (sidecar +2, query +2).
This commit is contained in:
russell@unturf.com 2026-04-29 18:32:28 -04:00
parent 631cf50690
commit d0a3d93836
No known key found for this signature in database
5 changed files with 201 additions and 3 deletions

View file

@ -333,6 +333,47 @@ def test_query_strict_fidelity_does_not_fall_back(tmp_path):
assert len(captured) == 1
def test_classify_source_role_separates_primary_from_noisy():
"""Direct unit test on the role classifier. JP film-score should be
noisy_background; JP (film) should be primary; JP franchise should
be secondary; The Lost World should be sequel; off-topic background.
Catches the case where peripheral pages with strong title overlap
used to share the primary slot."""
from aborist.qa.query import _classify_source_role
qstems = {"dinosaur", "jurassic", "park", "film"}
assert _classify_source_role("Jurassic Park (film)", qstems) == "primary_answer_source"
assert _classify_source_role("Jurassic Park (film score)", qstems) == "noisy_background_source"
assert _classify_source_role("Jurassic Park video games", qstems) == "noisy_background_source"
assert _classify_source_role("Jurassic Park (franchise)", qstems) == "secondary_context_source"
assert _classify_source_role("List of Jurassic Park characters", qstems) == "secondary_context_source"
assert _classify_source_role("The Lost World: Jurassic Park", qstems) == "sequel_background_source"
# Off-topic title (no shared stems): falls through to background.
assert _classify_source_role("Anarchism", qstems) == "background_source"
def test_query_role_weighted_budget_persists_role_on_sources(tmp_path):
"""Each source in the providence record's merkle_proof.sources gains
a `source_role` field verifies the role made it into the audit
trail so an inspector can see which slot a source occupied."""
main_db = tmp_path / "corpus.db"
qa_db = tmp_path / "qa.db"
conn = connect(main_db)
try:
ingest_source(conn, FakeSource(DOCS))
finally:
conn.close()
result = query(
question="What is anarcho-capitalism?",
qa_db=qa_db,
chat_client=StubClient(answer="x"),
model_id="m",
single_db=main_db,
top_k=3,
)
assert all("source_role" in s for s in result["sources"])
def test_query_no_sources_when_empty_corpus(tmp_path):
main_db = tmp_path / "empty.db"
qa_db = tmp_path / "qa.db"