qa: spotlight density rank — pick load-bearing slice, not leading match

Pre-fix: _spotlight_excerpt used first-match-of-longest-content-token.
For "who is homer simpson's boss?" the Homer Simpson chunk had
"homer" matching early in voice-actor / Castellaneta prose ("For
voicing Homer..."), so the spotlight centered there even though
the actual answer ("his boss Mr. Burns") was deeper in the chunk
where boss/burns/homer cluster together.

Post-fix: density rank. Find ALL match positions for ALL content
tokens, then pick the position whose ±half-window cluster contains
the MOST distinct tokens. Tie-break on smallest idx (deterministic,
reproducible).

Live verified:
  - "who is homer simpson's boss?" — spotlight now lands on
    "...At the plant, Homer is often ignored and completely
    forgotten by his boss Mr. Burns..." (boss/burns/homer/plant
    cluster). Pre-fix it landed on voice-actor prose with no
    Mr. Burns visible.
  - "name simpsons family members including pets?" — spotlight
    lands on "...the family's two pets, Santa's Little Helper
    and Snowball II..." (densest pets/family cluster). Already
    worked pre-fix; now even tighter.

Performance: O(|positions|^2) inner ranking. Typical chunks have
4-10 content tokens and 1-3 match positions each, so 8-30 positions
and ≤900 inner iterations — trivial.

Bracket format change (in render_claim_lattice) ships alongside:
   [E5: "..."]                          → before
   [E5 | <title> | <chunk_root[:8]>: "..."]  → after
Closes visual confusion where E# is a chunk pointer but readers
parsed it as a 1-indexed source rank from the sources list.

Tests: 523 passed.
This commit is contained in:
russell@unturf.com 2026-05-01 13:37:04 -04:00
parent 182bb67996
commit bf8d93caba
No known key found for this signature in database

View file

@ -267,8 +267,19 @@ def render_claim_lattice(
if obj is None:
lines.append(f' [{eid}: ?]')
continue
# Provenance-clear evidence pointer:
# [E5 | <source title> | <chunk_root prefix>: "<excerpt>"]
# Closes the visual confusion observed 2026-05-01 on the
# Orwell run where `[E5: "..."]` displayed alongside a
# source list whose `[5]` slot was a different document
# (E# is a chunk pointer, not a 1-indexed source rank).
# Title comes from the EvidenceObject (URI tail fallback);
# chunk_root prefix is the first 8 hex chars — enough for
# the operator to disambiguate while staying compact.
label = obj.title or obj.document_uri.rsplit("/", 1)[-1] or "untitled"
chunk_prefix = (obj.chunk_root or "")[:8]
excerpt = _spotlight_excerpt(text, obj.span, window=window)
lines.append(f' [{eid}: "{excerpt}"]')
lines.append(f' [{eid} | {label} | {chunk_prefix}: "{excerpt}"]')
return "\n".join(lines)
@ -382,14 +393,28 @@ def _spotlight_excerpt(claim_text: str, span: str, *, window: int) -> str:
if len(span) <= window:
return span
span_lower = span.lower()
match_idx = -1
for tok in _content_tokens(claim_text):
idx = span_lower.find(tok)
if idx >= 0:
match_idx = idx
break
tokens = _content_tokens(claim_text)
if match_idx < 0:
# Find ALL match positions for ALL content tokens, then pick the
# position whose ±half-window cluster contains the most distinct
# tokens. The Homer-Simpson-boss case (2026-05-01) showed why
# first-match-of-longest-token loses: "homer" matches early in
# voice-actor prose, but the actual answer ("Mr. Burns") is
# deeper in the chunk where boss/burns/homer all cluster
# together. Density picks the load-bearing slice; first-match
# picks whatever phrasing the chunk happens to lead with.
half = window // 2
positions: list[tuple[str, int]] = []
for tok in tokens:
i = 0
while True:
j = span_lower.find(tok, i)
if j < 0:
break
positions.append((tok, j))
i = j + len(tok)
if not positions:
# No content match — return the leading sentence(s) up to
# ~window bytes, expanded to the next sentence end so we
# never cut a word at the boundary.
@ -401,9 +426,25 @@ def _spotlight_excerpt(claim_text: str, span: str, *, window: int) -> str:
suffix = "..." if new_end < len(span) else ""
return f"{span[new_start:new_end]}{suffix}"
half = window // 2
# Density rank: for each candidate position, count distinct
# tokens whose match positions fall within ±half. Pick the
# position with max distinct count; tie-break on smallest idx
# (deterministic, reproducible).
best_score = -1
match_idx = positions[0][1]
for _tok, idx in positions:
distinct: set[str] = set()
lo, hi = idx - half, idx + half
for tok2, idx2 in positions:
if lo <= idx2 <= hi:
distinct.add(tok2)
score = len(distinct)
if score > best_score or (score == best_score and idx < match_idx):
best_score = score
match_idx = idx
raw_start = max(0, match_idx - half)
raw_end = min(len(span), match_idx + len(_content_tokens(claim_text)[0]) + half)
raw_end = min(len(span), match_idx + len(tokens[0]) + half)
# Expand outward to sentence boundaries — this is the main change.
new_start, new_end = _expand_to_sentence_boundaries(span, raw_start, raw_end)