qa: spotlight density rank — pick load-bearing slice, not leading match
Pre-fix: _spotlight_excerpt used first-match-of-longest-content-token.
For "who is homer simpson's boss?" the Homer Simpson chunk had
"homer" matching early in voice-actor / Castellaneta prose ("For
voicing Homer..."), so the spotlight centered there even though
the actual answer ("his boss Mr. Burns") was deeper in the chunk
where boss/burns/homer cluster together.
Post-fix: density rank. Find ALL match positions for ALL content
tokens, then pick the position whose ±half-window cluster contains
the MOST distinct tokens. Tie-break on smallest idx (deterministic,
reproducible).
Live verified:
- "who is homer simpson's boss?" — spotlight now lands on
"...At the plant, Homer is often ignored and completely
forgotten by his boss Mr. Burns..." (boss/burns/homer/plant
cluster). Pre-fix it landed on voice-actor prose with no
Mr. Burns visible.
- "name simpsons family members including pets?" — spotlight
lands on "...the family's two pets, Santa's Little Helper
and Snowball II..." (densest pets/family cluster). Already
worked pre-fix; now even tighter.
Performance: O(|positions|^2) inner ranking. Typical chunks have
4-10 content tokens and 1-3 match positions each, so 8-30 positions
and ≤900 inner iterations — trivial.
Bracket format change (in render_claim_lattice) ships alongside:
[E5: "..."] → before
[E5 | <title> | <chunk_root[:8]>: "..."] → after
Closes visual confusion where E# is a chunk pointer but readers
parsed it as a 1-indexed source rank from the sources list.
Tests: 523 passed.
This commit is contained in:
parent
182bb67996
commit
bf8d93caba
1 changed files with 51 additions and 10 deletions
|
|
@ -267,8 +267,19 @@ def render_claim_lattice(
|
|||
if obj is None:
|
||||
lines.append(f' [{eid}: ?]')
|
||||
continue
|
||||
# Provenance-clear evidence pointer:
|
||||
# [E5 | <source title> | <chunk_root prefix>: "<excerpt>"]
|
||||
# Closes the visual confusion observed 2026-05-01 on the
|
||||
# Orwell run where `[E5: "..."]` displayed alongside a
|
||||
# source list whose `[5]` slot was a different document
|
||||
# (E# is a chunk pointer, not a 1-indexed source rank).
|
||||
# Title comes from the EvidenceObject (URI tail fallback);
|
||||
# chunk_root prefix is the first 8 hex chars — enough for
|
||||
# the operator to disambiguate while staying compact.
|
||||
label = obj.title or obj.document_uri.rsplit("/", 1)[-1] or "untitled"
|
||||
chunk_prefix = (obj.chunk_root or "")[:8]
|
||||
excerpt = _spotlight_excerpt(text, obj.span, window=window)
|
||||
lines.append(f' [{eid}: "{excerpt}"]')
|
||||
lines.append(f' [{eid} | {label} | {chunk_prefix}: "{excerpt}"]')
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
|
|
@ -382,14 +393,28 @@ def _spotlight_excerpt(claim_text: str, span: str, *, window: int) -> str:
|
|||
if len(span) <= window:
|
||||
return span
|
||||
span_lower = span.lower()
|
||||
match_idx = -1
|
||||
for tok in _content_tokens(claim_text):
|
||||
idx = span_lower.find(tok)
|
||||
if idx >= 0:
|
||||
match_idx = idx
|
||||
break
|
||||
tokens = _content_tokens(claim_text)
|
||||
|
||||
if match_idx < 0:
|
||||
# Find ALL match positions for ALL content tokens, then pick the
|
||||
# position whose ±half-window cluster contains the most distinct
|
||||
# tokens. The Homer-Simpson-boss case (2026-05-01) showed why
|
||||
# first-match-of-longest-token loses: "homer" matches early in
|
||||
# voice-actor prose, but the actual answer ("Mr. Burns") is
|
||||
# deeper in the chunk where boss/burns/homer all cluster
|
||||
# together. Density picks the load-bearing slice; first-match
|
||||
# picks whatever phrasing the chunk happens to lead with.
|
||||
half = window // 2
|
||||
positions: list[tuple[str, int]] = []
|
||||
for tok in tokens:
|
||||
i = 0
|
||||
while True:
|
||||
j = span_lower.find(tok, i)
|
||||
if j < 0:
|
||||
break
|
||||
positions.append((tok, j))
|
||||
i = j + len(tok)
|
||||
|
||||
if not positions:
|
||||
# No content match — return the leading sentence(s) up to
|
||||
# ~window bytes, expanded to the next sentence end so we
|
||||
# never cut a word at the boundary.
|
||||
|
|
@ -401,9 +426,25 @@ def _spotlight_excerpt(claim_text: str, span: str, *, window: int) -> str:
|
|||
suffix = "..." if new_end < len(span) else ""
|
||||
return f"{span[new_start:new_end]}{suffix}"
|
||||
|
||||
half = window // 2
|
||||
# Density rank: for each candidate position, count distinct
|
||||
# tokens whose match positions fall within ±half. Pick the
|
||||
# position with max distinct count; tie-break on smallest idx
|
||||
# (deterministic, reproducible).
|
||||
best_score = -1
|
||||
match_idx = positions[0][1]
|
||||
for _tok, idx in positions:
|
||||
distinct: set[str] = set()
|
||||
lo, hi = idx - half, idx + half
|
||||
for tok2, idx2 in positions:
|
||||
if lo <= idx2 <= hi:
|
||||
distinct.add(tok2)
|
||||
score = len(distinct)
|
||||
if score > best_score or (score == best_score and idx < match_idx):
|
||||
best_score = score
|
||||
match_idx = idx
|
||||
|
||||
raw_start = max(0, match_idx - half)
|
||||
raw_end = min(len(span), match_idx + len(_content_tokens(claim_text)[0]) + half)
|
||||
raw_end = min(len(span), match_idx + len(tokens[0]) + half)
|
||||
|
||||
# Expand outward to sentence boundaries — this is the main change.
|
||||
new_start, new_end = _expand_to_sentence_boundaries(span, raw_start, raw_end)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue