qa(prompt): pointer-mode chunk-specificity rule (Sprint 2)
Sprint 2 — add Rule 9 to the claim_lattice_system_prompt to
discourage lazy-anchoring. Targets the structural pattern surfaced
by the 2026-05-02 bench: 49/142 pointer-mode rows (35%) had
lazy_anchor_ratio >= 0.75, and 19/71 questions show pointer fully
HHHHH while JSON aces SSSSS — pointer cites a topic-overview chunk
where JSON's structured per-claim linkage forces specificity.
The new rule (positive form per docs/bench-maxing.md):
9. Pick the pointer whose own text contains the claim's
specific facts — its dates, names, places, numbers, and
verbs. A chunk whose title matches the question's topic is
the right anchor only when that chunk's text also states
the claim's facts; when a different chunk states the fact
more directly, cite that one instead.
Six lines, no negation, named failure mode. Aligns with Rule 8
(title-relevance) by definition: a chunk whose body states the
fact almost always shares stems with its title.
The pointer prompt is duplicated between aborist/qa/runner.py
DEFAULT_POLICY and aborist/qa/query.py DEFAULT_QUERY_POLICY.
Both updated in sync. (DRY violation worth cleaning up later —
the duplication can drift; one source of truth would be safer.)
Smoke (5-question fixture, n=1) — no regression, n=1 too noisy
to detect signal:
quote 5/5 → 4/5..5/5 (oscillation, single dinosaurs flip)
pointer 1/5 → 0/5..1/5 (sampling noise around the floor)
JSON 5/5 → 5/5 (held)
Real signal lives in the full 71-question n=3 bench (~50 min).
The change is principled, no calibration cost, and folds into
governance_policy_hash so prior cached records auto-partition.
This commit is contained in:
parent
e582b2e93a
commit
e7650b8eee
2 changed files with 14 additions and 2 deletions
|
|
@ -379,7 +379,13 @@ DEFAULT_QUERY_POLICY = {
|
|||
"three or more pointers are rejected. When more than two "
|
||||
"evidence blocks support a claim, pick the two that most "
|
||||
"directly contain the claim's key terms, or split the "
|
||||
"claim into two lines."
|
||||
"claim into two lines.\n"
|
||||
"9. Pick the pointer whose own text contains the claim's "
|
||||
"specific facts — its dates, names, places, numbers, and "
|
||||
"verbs. A chunk whose title matches the question's topic is "
|
||||
"the right anchor only when that chunk's text also states "
|
||||
"the claim's facts; when a different chunk states the fact "
|
||||
"more directly, cite that one instead."
|
||||
),
|
||||
"claim_lattice_grounding_reminder": (
|
||||
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
|
||||
|
|
|
|||
|
|
@ -170,7 +170,13 @@ DEFAULT_POLICY = {
|
|||
"three or more pointers are rejected. When more than two "
|
||||
"evidence blocks support a claim, pick the two that most "
|
||||
"directly contain the claim's key terms, or split the "
|
||||
"claim into two lines."
|
||||
"claim into two lines.\n"
|
||||
"9. Pick the pointer whose own text contains the claim's "
|
||||
"specific facts — its dates, names, places, numbers, and "
|
||||
"verbs. A chunk whose title matches the question's topic is "
|
||||
"the right anchor only when that chunk's text also states "
|
||||
"the claim's facts; when a different chunk states the fact "
|
||||
"more directly, cite that one instead."
|
||||
),
|
||||
"claim_lattice_grounding_reminder": (
|
||||
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue