From e7650b8eee88b2129706b154b8e84186b95c2052 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 2 May 2026 09:48:19 -0400 Subject: [PATCH] qa(prompt): pointer-mode chunk-specificity rule (Sprint 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sprint 2 — add Rule 9 to the claim_lattice_system_prompt to discourage lazy-anchoring. Targets the structural pattern surfaced by the 2026-05-02 bench: 49/142 pointer-mode rows (35%) had lazy_anchor_ratio >= 0.75, and 19/71 questions show pointer fully HHHHH while JSON aces SSSSS — pointer cites a topic-overview chunk where JSON's structured per-claim linkage forces specificity. The new rule (positive form per docs/bench-maxing.md): 9. Pick the pointer whose own text contains the claim's specific facts — its dates, names, places, numbers, and verbs. A chunk whose title matches the question's topic is the right anchor only when that chunk's text also states the claim's facts; when a different chunk states the fact more directly, cite that one instead. Six lines, no negation, named failure mode. Aligns with Rule 8 (title-relevance) by definition: a chunk whose body states the fact almost always shares stems with its title. The pointer prompt is duplicated between aborist/qa/runner.py DEFAULT_POLICY and aborist/qa/query.py DEFAULT_QUERY_POLICY. Both updated in sync. (DRY violation worth cleaning up later — the duplication can drift; one source of truth would be safer.) Smoke (5-question fixture, n=1) — no regression, n=1 too noisy to detect signal: quote 5/5 → 4/5..5/5 (oscillation, single dinosaurs flip) pointer 1/5 → 0/5..1/5 (sampling noise around the floor) JSON 5/5 → 5/5 (held) Real signal lives in the full 71-question n=3 bench (~50 min). The change is principled, no calibration cost, and folds into governance_policy_hash so prior cached records auto-partition. --- aborist/qa/query.py | 8 +++++++- aborist/qa/runner.py | 8 +++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/aborist/qa/query.py b/aborist/qa/query.py index 3476ce3..c8247e3 100644 --- a/aborist/qa/query.py +++ b/aborist/qa/query.py @@ -379,7 +379,13 @@ DEFAULT_QUERY_POLICY = { "three or more pointers are rejected. When more than two " "evidence blocks support a claim, pick the two that most " "directly contain the claim's key terms, or split the " - "claim into two lines." + "claim into two lines.\n" + "9. Pick the pointer whose own text contains the claim's " + "specific facts — its dates, names, places, numbers, and " + "verbs. A chunk whose title matches the question's topic is " + "the right anchor only when that chunk's text also states " + "the claim's facts; when a different chunk states the fact " + "more directly, cite that one instead." ), "claim_lattice_grounding_reminder": ( "REMINDER: format = pointer-line — `Claim text. [E1]` per " diff --git a/aborist/qa/runner.py b/aborist/qa/runner.py index e6b8bc3..9f578ec 100644 --- a/aborist/qa/runner.py +++ b/aborist/qa/runner.py @@ -170,7 +170,13 @@ DEFAULT_POLICY = { "three or more pointers are rejected. When more than two " "evidence blocks support a claim, pick the two that most " "directly contain the claim's key terms, or split the " - "claim into two lines." + "claim into two lines.\n" + "9. Pick the pointer whose own text contains the claim's " + "specific facts — its dates, names, places, numbers, and " + "verbs. A chunk whose title matches the question's topic is " + "the right anchor only when that chunk's text also states " + "the claim's facts; when a different chunk states the fact " + "more directly, cite that one instead." ), "claim_lattice_grounding_reminder": ( "REMINDER: format = pointer-line — `Claim text. [E1]` per "