qa: synthetic_elision sidecar diagnosis + role-weighted source budget
Two enhancements from the toy-Hermes design pass (2026-04-30):
A. Synthetic-elision-inside-quote diagnosis (sidecar only).
Distinct from interior_elision (model dropped a `(...)` aside source
carries) — synthetic_elision is the model writing literal `[...]`
between fragments of a `"..."` span, signaling self-elision while
claiming verbatim citation. The verifier still rejects (binary
discipline holds), but `aborist inspect` now reports
`diagnosis: synthetic_elision_inside_quote` with prefix/suffix
presence flags so an operator can judge whether the elided middle
was benign. Probe runs first in the classify-span chain (more
specific than trailing_artifact / interior_elision / paraphrase).
Catches the Brachiosaurus case: `"The film centers on the fictional
Isla Nublar [...] Universal Studios..."` — both halves are in source,
but the literal `[...]` isn't, so substring match correctly fails &
the sidecar tells the operator why.
C. Source-role classification + role-weighted context budget.
`_classify_source_role(title, qtokens_stem)` tags each top-K hit:
primary_answer_source 2.0× cap strong title-stem overlap
secondary_context_source 1.0× cap "list of", "characters",
"franchise", "history of"
noisy_background_source 0.5× cap "score", "music",
"video game", "merchandise"
sequel_background_source 0.5× cap "lost world", roman numerals
background_source 1.0× cap default
Order matters: noisy/sequel/secondary markers fire before the
primary check so peripheral pages with strong title overlap (e.g.
`Jurassic Park (film score)` shares 3 stems with the JP-film query)
don't claim a primary slot.
Cap loop now applies role weight on top of the baseline
`max_context_chars / top_k`. Total context still bounded by the
running `char_budget` — weights just shift how the budget gets
divided so primary pages get more text & noisy pages less, fixing
the case where a `(film score)` page consumed a primary slot.
`source_role` is persisted on `_Hit` and surfaces on
`merkle_proof.sources[*].source_role` in the providence record so
inspect & audits can see which slot each source occupied.
Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
synthetic_elision_does_not_fire_when_source_has_brackets (false-
positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
background) on JP-film-style titles, role persistence in sources list.
455 tests pass (sidecar +2, query +2).
This commit is contained in:
parent
631cf50690
commit
d0a3d93836
5 changed files with 201 additions and 3 deletions
|
|
@ -333,6 +333,47 @@ def test_query_strict_fidelity_does_not_fall_back(tmp_path):
|
|||
assert len(captured) == 1
|
||||
|
||||
|
||||
def test_classify_source_role_separates_primary_from_noisy():
|
||||
"""Direct unit test on the role classifier. JP film-score should be
|
||||
noisy_background; JP (film) should be primary; JP franchise should
|
||||
be secondary; The Lost World should be sequel; off-topic background.
|
||||
Catches the case where peripheral pages with strong title overlap
|
||||
used to share the primary slot."""
|
||||
from aborist.qa.query import _classify_source_role
|
||||
|
||||
qstems = {"dinosaur", "jurassic", "park", "film"}
|
||||
assert _classify_source_role("Jurassic Park (film)", qstems) == "primary_answer_source"
|
||||
assert _classify_source_role("Jurassic Park (film score)", qstems) == "noisy_background_source"
|
||||
assert _classify_source_role("Jurassic Park video games", qstems) == "noisy_background_source"
|
||||
assert _classify_source_role("Jurassic Park (franchise)", qstems) == "secondary_context_source"
|
||||
assert _classify_source_role("List of Jurassic Park characters", qstems) == "secondary_context_source"
|
||||
assert _classify_source_role("The Lost World: Jurassic Park", qstems) == "sequel_background_source"
|
||||
# Off-topic title (no shared stems): falls through to background.
|
||||
assert _classify_source_role("Anarchism", qstems) == "background_source"
|
||||
|
||||
|
||||
def test_query_role_weighted_budget_persists_role_on_sources(tmp_path):
|
||||
"""Each source in the providence record's merkle_proof.sources gains
|
||||
a `source_role` field — verifies the role made it into the audit
|
||||
trail so an inspector can see which slot a source occupied."""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
result = query(
|
||||
question="What is anarcho-capitalism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer="x"),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
top_k=3,
|
||||
)
|
||||
assert all("source_role" in s for s in result["sources"])
|
||||
|
||||
|
||||
def test_query_no_sources_when_empty_corpus(tmp_path):
|
||||
main_db = tmp_path / "empty.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue