qa(prompt): pointer-mode chunk-specificity rule (Sprint 2)

Sprint 2 — add Rule 9 to the claim_lattice_system_prompt to
discourage lazy-anchoring. Targets the structural pattern surfaced
by the 2026-05-02 bench: 49/142 pointer-mode rows (35%) had
lazy_anchor_ratio >= 0.75, and 19/71 questions show pointer fully
HHHHH while JSON aces SSSSS — pointer cites a topic-overview chunk
where JSON's structured per-claim linkage forces specificity.

The new rule (positive form per docs/bench-maxing.md):

  9. Pick the pointer whose own text contains the claim's
     specific facts — its dates, names, places, numbers, and
     verbs. A chunk whose title matches the question's topic is
     the right anchor only when that chunk's text also states
     the claim's facts; when a different chunk states the fact
     more directly, cite that one instead.

Six lines, no negation, named failure mode. Aligns with Rule 8
(title-relevance) by definition: a chunk whose body states the
fact almost always shares stems with its title.

The pointer prompt is duplicated between aborist/qa/runner.py
DEFAULT_POLICY and aborist/qa/query.py DEFAULT_QUERY_POLICY.
Both updated in sync. (DRY violation worth cleaning up later —
the duplication can drift; one source of truth would be safer.)

Smoke (5-question fixture, n=1) — no regression, n=1 too noisy
to detect signal:

  quote   5/5 → 4/5..5/5  (oscillation, single dinosaurs flip)
  pointer 1/5 → 0/5..1/5  (sampling noise around the floor)
  JSON    5/5 → 5/5       (held)

Real signal lives in the full 71-question n=3 bench (~50 min).
The change is principled, no calibration cost, and folds into
governance_policy_hash so prior cached records auto-partition.
This commit is contained in:
russell@unturf.com 2026-05-02 09:48:19 -04:00
parent e582b2e93a
commit e7650b8eee
No known key found for this signature in database
2 changed files with 14 additions and 2 deletions

View file

@ -379,7 +379,13 @@ DEFAULT_QUERY_POLICY = {
"three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms, or split the "
"claim into two lines."
"claim into two lines.\n"
"9. Pick the pointer whose own text contains the claim's "
"specific facts — its dates, names, places, numbers, and "
"verbs. A chunk whose title matches the question's topic is "
"the right anchor only when that chunk's text also states "
"the claim's facts; when a different chunk states the fact "
"more directly, cite that one instead."
),
"claim_lattice_grounding_reminder": (
"REMINDER: format = pointer-line — `Claim text. [E1]` per "

View file

@ -170,7 +170,13 @@ DEFAULT_POLICY = {
"three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms, or split the "
"claim into two lines."
"claim into two lines.\n"
"9. Pick the pointer whose own text contains the claim's "
"specific facts — its dates, names, places, numbers, and "
"verbs. A chunk whose title matches the question's topic is "
"the right anchor only when that chunk's text also states "
"the claim's facts; when a different chunk states the fact "
"more directly, cite that one instead."
),
"claim_lattice_grounding_reminder": (
"REMINDER: format = pointer-line — `Claim text. [E1]` per "