From 4b356030fbb09e5157334cd826803c10197e1181 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 2 May 2026 09:58:45 -0400 Subject: [PATCH] qa(prompts): collapse lattice prompts to single source of truth MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four claim-lattice prompt strings — CLAIM_LATTICE_SYSTEM_PROMPT CLAIM_LATTICE_GROUNDING_REMINDER CLAIM_LATTICE_JSON_SYSTEM_PROMPT CLAIM_LATTICE_JSON_GROUNDING_REMINDER — were duplicated byte-for-byte between aborist/qa/runner.py DEFAULT_POLICY (single-document path) and aborist/qa/query.py DEFAULT_QUERY_POLICY (multi-source path). Sprint 2 surfaced the DRY violation when its agent had to update both files in lockstep to add Rule 9. Two sources of truth = one source of drift. Lift the four strings to aborist/qa/prompts.py as module-level constants. Both DEFAULT_* dicts import them and reference by name. The surrounding policy entries legitimately differ between the two paths (different system_prompt/grounding_reminder for single-doc vs multi-source framing, different max_tokens, different chunk caps) — only the lattice prompts collapse. Net shrinkage: -240 duplicated lines across runner.py + query.py, +153 in prompts.py = ~87 lines removed. Sole authoritative copy lives in prompts.py. Verified via runtime equality check: both DEFAULT_POLICY and DEFAULT_QUERY_POLICY resolve to the same string objects post- import, matching the prompts.py constants byte-for-byte. Test suite (751/34) passes clean in 11s with pytest -n auto. No behavioural change. governance_policy_hash unchanged because the strings themselves are unchanged. Cache namespace unaffected. --- aborist/qa/prompts.py | 153 ++++++++++++++++++++++++++++++++++++++++++ aborist/qa/query.py | 120 +++------------------------------ aborist/qa/runner.py | 120 +++------------------------------ 3 files changed, 173 insertions(+), 220 deletions(-) create mode 100644 aborist/qa/prompts.py diff --git a/aborist/qa/prompts.py b/aborist/qa/prompts.py new file mode 100644 index 0000000..c597e08 --- /dev/null +++ b/aborist/qa/prompts.py @@ -0,0 +1,153 @@ +"""Shared claim-lattice prompt strings. + +Single source of truth for the four prompt strings that travel +across both QA paths: + +- ``aborist.qa.runner.ask()`` — single-document path +- ``aborist.qa.query.query()`` — multi-source retrieval path + +Both paths use the same lattice prompts (pointer + JSON variants, +system + grounding-reminder each), so DRY them here. The +surrounding ``DEFAULT_POLICY`` / ``DEFAULT_QUERY_POLICY`` dicts +legitimately differ — different ``system_prompt`` / ``grounding_reminder`` +for the single-document vs multi-source framing, different +``max_tokens``, different per-mode chunk caps. Only the +claim-lattice prompts (and the rules they encode) collapse to one +copy. + +Bumping any of these strings folds into ``governance_policy_hash`` +in both code paths simultaneously, so the cache namespace +partitions cleanly across the substrate. See +``docs/qa-modes-bench-2026-05-02.md`` for the bench-driven +hardening history. +""" +from __future__ import annotations + + +# Pointer-line variant — the model emits prose-with-bracket-tags. +# The runtime owns quote text; the model never types a quote string +# (synthetic-elision-by-construction-impossible). Two-layer id +# discipline (pointer_id E1/E2/… vs content-addressed evidence_id) +# keeps the cache & run-DAG keyed on stable hashes while the model +# sees BPE-friendly short tags. +CLAIM_LATTICE_SYSTEM_PROMPT = ( + "You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. " + "Answer using natural-language pointer-lines: one claim per " + "line, followed by a bracket tag with the pointer IDs that " + "directly support that claim.\n\n" + "WORKED EXAMPLE 1 — narrow factoid\n" + "---------------------------------\n" + "EVIDENCE:\n\n" + "=== E1 (Apple_Inc | primary_answer_source) ===\n" + "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and " + "Ronald Wayne in April 1976.\n\n" + "=== E2 (Steve_Wozniak | secondary_context_source) ===\n" + "Steve Wozniak co-founded Apple Computer Company alongside " + "Steve Jobs in 1976 and designed the Apple I.\n\n" + "QUESTION: who founded Apple?\n\n" + "ANSWER:\n" + "Steve Jobs co-founded Apple. [E1]\n" + "Steve Wozniak co-founded Apple. [E1,E2]\n" + "Ronald Wayne co-founded Apple. [E1]\n\n" + "WORKED EXAMPLE 2 — broad descriptive\n" + "------------------------------------\n" + "EVIDENCE:\n\n" + "=== E1 (Mars | primary_answer_source) ===\n" + "Mars is the fourth planet from the Sun. It has two moons, " + "Phobos and Deimos. Mars has a thin atmosphere of carbon " + "dioxide. Average surface temperature is around -60 " + "degrees Celsius.\n\n" + "QUESTION: tell me about Mars\n\n" + "ANSWER:\n" + "Mars is the fourth planet from the Sun. [E1]\n" + "Mars has two moons, Phobos and Deimos. [E1]\n" + "Mars has a thin atmosphere of carbon dioxide. [E1]\n" + "The average surface temperature on Mars is around -60 " + "degrees Celsius. [E1]\n\n" + "END OF EXAMPLES\n\n" + "RULES (each rule says what TO do):\n" + "1. Reference evidence by pointer ID. The runtime displays " + "the literal source span beside each claim — referencing is " + "your job; quoting is the runtime's job.\n" + "2. Use only pointer IDs that appear in the EVIDENCE blocks " + "above.\n" + "3. Cite 1 or 2 pointers per claim — the blocks whose text " + "directly contains the claim's key terms.\n" + "4. End every claim line with [E#] or [E#,E#].\n" + "5. Make a claim only when an EVIDENCE block textually " + "supports it. Stop when the evidence runs out — a short " + "answer is the right answer when only short evidence " + "exists.\n" + "6. Write each claim as one plain-prose sentence on its own " + "line.\n" + "7. Each line begins with the claim text and ends with a " + "bracket tag of pointer ids. The headers above the QUESTION " + "(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for " + "the input only — your answer is the claim lines themselves.\n" + "8. Each claim line cites at most two pointers. Lines with " + "three or more pointers are rejected. When more than two " + "evidence blocks support a claim, pick the two that most " + "directly contain the claim's key terms, or split the " + "claim into two lines.\n" + "9. Pick the pointer whose own text contains the claim's " + "specific facts — its dates, names, places, numbers, and " + "verbs. A chunk whose title matches the question's topic is " + "the right anchor only when that chunk's text also states " + "the claim's facts; when a different chunk states the fact " + "more directly, cite that one instead." +) + + +CLAIM_LATTICE_GROUNDING_REMINDER = ( + "REMINDER: format = pointer-line — `Claim text. [E1]` per " + "line, plain prose with bracket tags. Pointer IDs come from " + "the EVIDENCE blocks above. At most two pointers per claim. " + "Begin each line with the claim text. Now answer the question " + "on the next message." +) + + +# JSON variant — pairs with grammar-constrained inference (vLLM +# guided_json, Claude/GPT-4 native JSON, Qwen 3.6 reasoner). +# Lenient pre-parser in ``aborist.qa.verify.verify_claim_lattice_json`` +# keeps the path survivable on inference paths without grammar +# guidance. +CLAIM_LATTICE_JSON_SYSTEM_PROMPT = ( + "You will see numbered EVIDENCE blocks tagged with short " + "pointer IDs: E1, E2, E3, etc. Answer as a single JSON " + "object using EXACTLY this schema. Your output begins with " + "`{` and ends with `}`:\n\n" + ' {"claims":[{"text":"","evidence_ids":["E1"]}]}\n\n' + "RULES:\n" + "1. Your output is exactly one JSON object — first character " + "is `{`, last character is `}`.\n" + "2. `text` is plain prose. Punctuation (including any quoted " + "names from the source) appears in the source span the " + "runtime renders for you, so write the text using only " + "letters, digits, spaces, and basic sentence punctuation " + "(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n" + "3. `evidence_ids` are pointer IDs that appear verbatim in " + "the EVIDENCE block headers above (E1, E2, …). Use only IDs " + "you can see in the headers above. At most two IDs per claim.\n" + "4. Each claim references at least one evidence_id.\n" + "5. Emit a claim only when an EVIDENCE block directly " + "supports it; stop emitting claims when supporting evidence " + "runs out — a short answer is the right answer when only " + "short evidence exists.\n" + "6. Each claim is one focused fact citing the source that " + "directly contains it. Multiple distinct facts become " + "multiple claim entries, each citing the source for that " + "fact. Three short claims, each citing one pointer, beat " + "one long claim citing five." +) + + +CLAIM_LATTICE_JSON_GROUNDING_REMINDER = ( + "REMINDER: emit one JSON object — first character `{`, last " + "character `}` — with the schema " + '`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. ' + "evidence_ids are pointer IDs (E1, E2, …) that appear " + "verbatim in the EVIDENCE block headers above. At most two " + "per claim. Each claim is one focused fact. Now answer the " + "question on the next message." +) diff --git a/aborist/qa/query.py b/aborist/qa/query.py index c8247e3..b8dda9a 100644 --- a/aborist/qa/query.py +++ b/aborist/qa/query.py @@ -47,6 +47,12 @@ from aborist import ( from aborist.compress import unpack_chunk from aborist.merkle import MerkleTree from aborist.qa.client import ChatClient +from aborist.qa.prompts import ( + CLAIM_LATTICE_GROUNDING_REMINDER, + CLAIM_LATTICE_JSON_GROUNDING_REMINDER, + CLAIM_LATTICE_JSON_SYSTEM_PROMPT, + CLAIM_LATTICE_SYSTEM_PROMPT, +) from aborist.qa.concepts import ( has_compare_phrasing, rivalry_excluded, @@ -321,79 +327,8 @@ DEFAULT_QUERY_POLICY = { # output ("Claim. [E12]") that the verifier maps back to # content-addressed evidence_ids for the cache & run-DAG. "answer_mode": DEFAULT_ANSWER_MODE, - "claim_lattice_system_prompt": ( - "You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. " - "Answer using natural-language pointer-lines: one claim per " - "line, followed by a bracket tag with the pointer IDs that " - "directly support that claim.\n\n" - "WORKED EXAMPLE 1 — narrow factoid\n" - "---------------------------------\n" - "EVIDENCE:\n\n" - "=== E1 (Apple_Inc | primary_answer_source) ===\n" - "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and " - "Ronald Wayne in April 1976.\n\n" - "=== E2 (Steve_Wozniak | secondary_context_source) ===\n" - "Steve Wozniak co-founded Apple Computer Company alongside " - "Steve Jobs in 1976 and designed the Apple I.\n\n" - "QUESTION: who founded Apple?\n\n" - "ANSWER:\n" - "Steve Jobs co-founded Apple. [E1]\n" - "Steve Wozniak co-founded Apple. [E1,E2]\n" - "Ronald Wayne co-founded Apple. [E1]\n\n" - "WORKED EXAMPLE 2 — broad descriptive\n" - "------------------------------------\n" - "EVIDENCE:\n\n" - "=== E1 (Mars | primary_answer_source) ===\n" - "Mars is the fourth planet from the Sun. It has two moons, " - "Phobos and Deimos. Mars has a thin atmosphere of carbon " - "dioxide. Average surface temperature is around -60 " - "degrees Celsius.\n\n" - "QUESTION: tell me about Mars\n\n" - "ANSWER:\n" - "Mars is the fourth planet from the Sun. [E1]\n" - "Mars has two moons, Phobos and Deimos. [E1]\n" - "Mars has a thin atmosphere of carbon dioxide. [E1]\n" - "The average surface temperature on Mars is around -60 " - "degrees Celsius. [E1]\n\n" - "END OF EXAMPLES\n\n" - "RULES (each rule says what TO do):\n" - "1. Reference evidence by pointer ID. The runtime displays " - "the literal source span beside each claim — referencing is " - "your job; quoting is the runtime's job.\n" - "2. Use only pointer IDs that appear in the EVIDENCE blocks " - "above.\n" - "3. Cite 1 or 2 pointers per claim — the blocks whose text " - "directly contains the claim's key terms.\n" - "4. End every claim line with [E#] or [E#,E#].\n" - "5. Make a claim only when an EVIDENCE block textually " - "supports it. Stop when the evidence runs out — a short " - "answer is the right answer when only short evidence " - "exists.\n" - "6. Write each claim as one plain-prose sentence on its own " - "line.\n" - "7. Each line begins with the claim text and ends with a " - "bracket tag of pointer ids. The headers above the QUESTION " - "(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for " - "the input only — your answer is the claim lines themselves.\n" - "8. Each claim line cites at most two pointers. Lines with " - "three or more pointers are rejected. When more than two " - "evidence blocks support a claim, pick the two that most " - "directly contain the claim's key terms, or split the " - "claim into two lines.\n" - "9. Pick the pointer whose own text contains the claim's " - "specific facts — its dates, names, places, numbers, and " - "verbs. A chunk whose title matches the question's topic is " - "the right anchor only when that chunk's text also states " - "the claim's facts; when a different chunk states the fact " - "more directly, cite that one instead." - ), - "claim_lattice_grounding_reminder": ( - "REMINDER: format = pointer-line — `Claim text. [E1]` per " - "line, plain prose with bracket tags. Pointer IDs come from " - "the EVIDENCE blocks above. At most two pointers per claim. " - "Begin each line with the claim text. Now answer the question " - "on the next message." - ), + "claim_lattice_system_prompt": CLAIM_LATTICE_SYSTEM_PROMPT, + "claim_lattice_grounding_reminder": CLAIM_LATTICE_GROUNDING_REMINDER, # Reference-frame polarity preamble (Ticket #000002 / Module L). # Injected as an additional user-role message before the # grounding_reminder when `detect_frame` classifies the query as @@ -458,43 +393,8 @@ DEFAULT_QUERY_POLICY = { # survivable on inference paths without grammar guidance. Toggling # `claim_lattice_use_guided_json=False` disables the extra_body # pass for endpoints that 400 on unknown fields. - "claim_lattice_json_system_prompt": ( - "You will see numbered EVIDENCE blocks tagged with short " - "pointer IDs: E1, E2, E3, etc. Answer as a single JSON " - "object using EXACTLY this schema. Your output begins with " - "`{` and ends with `}`:\n\n" - ' {"claims":[{"text":"","evidence_ids":["E1"]}]}\n\n' - "RULES:\n" - "1. Your output is exactly one JSON object — first character " - "is `{`, last character is `}`.\n" - "2. `text` is plain prose. Punctuation (including any quoted " - "names from the source) appears in the source span the " - "runtime renders for you, so write the text using only " - "letters, digits, spaces, and basic sentence punctuation " - "(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n" - "3. `evidence_ids` are pointer IDs that appear verbatim in " - "the EVIDENCE block headers above (E1, E2, …). Use only IDs " - "you can see in the headers above. At most two IDs per claim.\n" - "4. Each claim references at least one evidence_id.\n" - "5. Emit a claim only when an EVIDENCE block directly " - "supports it; stop emitting claims when supporting evidence " - "runs out — a short answer is the right answer when only " - "short evidence exists.\n" - "6. Each claim is one focused fact citing the source that " - "directly contains it. Multiple distinct facts become " - "multiple claim entries, each citing the source for that " - "fact. Three short claims, each citing one pointer, beat " - "one long claim citing five." - ), - "claim_lattice_json_grounding_reminder": ( - "REMINDER: emit one JSON object — first character `{`, last " - "character `}` — with the schema " - '`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. ' - "evidence_ids are pointer IDs (E1, E2, …) that appear " - "verbatim in the EVIDENCE block headers above. At most two " - "per claim. Each claim is one focused fact. Now answer the " - "question on the next message." - ), + "claim_lattice_json_system_prompt": CLAIM_LATTICE_JSON_SYSTEM_PROMPT, + "claim_lattice_json_grounding_reminder": CLAIM_LATTICE_JSON_GROUNDING_REMINDER, "claim_lattice_use_guided_json": True, "claim_lattice_json_stop_sequences": ["\n\n"], # Per-mode context-budget defaults. Sprint 1b (2026-05-02) bench diff --git a/aborist/qa/runner.py b/aborist/qa/runner.py index 9f578ec..67d8c83 100644 --- a/aborist/qa/runner.py +++ b/aborist/qa/runner.py @@ -22,6 +22,12 @@ from aborist import ( from aborist.compress import unpack_chunk from aborist.merkle import MerkleTree, proof_to_dict from aborist.qa.client import ChatClient +from aborist.qa.prompts import ( + CLAIM_LATTICE_GROUNDING_REMINDER, + CLAIM_LATTICE_JSON_GROUNDING_REMINDER, + CLAIM_LATTICE_JSON_SYSTEM_PROMPT, + CLAIM_LATTICE_SYSTEM_PROMPT, +) from aborist.qa.keys import ( DEFAULT_FIDELITY, DEFAULT_QUESTION_DEDUP, @@ -112,79 +118,8 @@ DEFAULT_POLICY = { # different cache_keys and never alias. No iterative repair in # pointer mode (one-shot benchmark discipline). "answer_mode": DEFAULT_ANSWER_MODE, - "claim_lattice_system_prompt": ( - "You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. " - "Answer using natural-language pointer-lines: one claim per " - "line, followed by a bracket tag with the pointer IDs that " - "directly support that claim.\n\n" - "WORKED EXAMPLE 1 — narrow factoid\n" - "---------------------------------\n" - "EVIDENCE:\n\n" - "=== E1 (Apple_Inc | primary_answer_source) ===\n" - "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and " - "Ronald Wayne in April 1976.\n\n" - "=== E2 (Steve_Wozniak | secondary_context_source) ===\n" - "Steve Wozniak co-founded Apple Computer Company alongside " - "Steve Jobs in 1976 and designed the Apple I.\n\n" - "QUESTION: who founded Apple?\n\n" - "ANSWER:\n" - "Steve Jobs co-founded Apple. [E1]\n" - "Steve Wozniak co-founded Apple. [E1,E2]\n" - "Ronald Wayne co-founded Apple. [E1]\n\n" - "WORKED EXAMPLE 2 — broad descriptive\n" - "------------------------------------\n" - "EVIDENCE:\n\n" - "=== E1 (Mars | primary_answer_source) ===\n" - "Mars is the fourth planet from the Sun. It has two moons, " - "Phobos and Deimos. Mars has a thin atmosphere of carbon " - "dioxide. Average surface temperature is around -60 " - "degrees Celsius.\n\n" - "QUESTION: tell me about Mars\n\n" - "ANSWER:\n" - "Mars is the fourth planet from the Sun. [E1]\n" - "Mars has two moons, Phobos and Deimos. [E1]\n" - "Mars has a thin atmosphere of carbon dioxide. [E1]\n" - "The average surface temperature on Mars is around -60 " - "degrees Celsius. [E1]\n\n" - "END OF EXAMPLES\n\n" - "RULES (each rule says what TO do):\n" - "1. Reference evidence by pointer ID. The runtime displays " - "the literal source span beside each claim — referencing is " - "your job; quoting is the runtime's job.\n" - "2. Use only pointer IDs that appear in the EVIDENCE blocks " - "above.\n" - "3. Cite 1 or 2 pointers per claim — the blocks whose text " - "directly contains the claim's key terms.\n" - "4. End every claim line with [E#] or [E#,E#].\n" - "5. Make a claim only when an EVIDENCE block textually " - "supports it. Stop when the evidence runs out — a short " - "answer is the right answer when only short evidence " - "exists.\n" - "6. Write each claim as one plain-prose sentence on its own " - "line.\n" - "7. Each line begins with the claim text and ends with a " - "bracket tag of pointer ids. The headers above the QUESTION " - "(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for " - "the input only — your answer is the claim lines themselves.\n" - "8. Each claim line cites at most two pointers. Lines with " - "three or more pointers are rejected. When more than two " - "evidence blocks support a claim, pick the two that most " - "directly contain the claim's key terms, or split the " - "claim into two lines.\n" - "9. Pick the pointer whose own text contains the claim's " - "specific facts — its dates, names, places, numbers, and " - "verbs. A chunk whose title matches the question's topic is " - "the right anchor only when that chunk's text also states " - "the claim's facts; when a different chunk states the fact " - "more directly, cite that one instead." - ), - "claim_lattice_grounding_reminder": ( - "REMINDER: format = pointer-line — `Claim text. [E1]` per " - "line, plain prose with bracket tags. Pointer IDs come from " - "the EVIDENCE blocks above. At most two pointers per claim. " - "Begin each line with the claim text. Now answer the question " - "on the next message." - ), + "claim_lattice_system_prompt": CLAIM_LATTICE_SYSTEM_PROMPT, + "claim_lattice_grounding_reminder": CLAIM_LATTICE_GROUNDING_REMINDER, # Allowed source roles for claim-lattice verification. Roles outside # this set get classified SOURCE_ROLE_BLOCKED and downgrade the # verdict. Mirrors aborist.qa.verify.DEFAULT_ALLOWED_SOURCE_ROLES; @@ -263,43 +198,8 @@ DEFAULT_POLICY = { # (vLLM guided_json, Claude/GPT-4 native JSON, Qwen 3.6 reasoner). # Lenient pre-parser in verify_claim_lattice_json keeps the path # survivable on inference paths without grammar guidance. - "claim_lattice_json_system_prompt": ( - "You will see numbered EVIDENCE blocks tagged with short " - "pointer IDs: E1, E2, E3, etc. Answer as a single JSON " - "object using EXACTLY this schema. Your output begins with " - "`{` and ends with `}`:\n\n" - ' {"claims":[{"text":"","evidence_ids":["E1"]}]}\n\n' - "RULES:\n" - "1. Your output is exactly one JSON object — first character " - "is `{`, last character is `}`.\n" - "2. `text` is plain prose. Punctuation (including any quoted " - "names from the source) appears in the source span the " - "runtime renders for you, so write the text using only " - "letters, digits, spaces, and basic sentence punctuation " - "(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n" - "3. `evidence_ids` are pointer IDs that appear verbatim in " - "the EVIDENCE block headers above (E1, E2, …). Use only IDs " - "you can see in the headers above. At most two IDs per claim.\n" - "4. Each claim references at least one evidence_id.\n" - "5. Emit a claim only when an EVIDENCE block directly " - "supports it; stop emitting claims when supporting evidence " - "runs out — a short answer is the right answer when only " - "short evidence exists.\n" - "6. Each claim is one focused fact citing the source that " - "directly contains it. Multiple distinct facts become " - "multiple claim entries, each citing the source for that " - "fact. Three short claims, each citing one pointer, beat " - "one long claim citing five." - ), - "claim_lattice_json_grounding_reminder": ( - "REMINDER: emit one JSON object — first character `{`, last " - "character `}` — with the schema " - '`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. ' - "evidence_ids are pointer IDs (E1, E2, …) that appear " - "verbatim in the EVIDENCE block headers above. At most two " - "per claim. Each claim is one focused fact. Now answer the " - "question on the next message." - ), + "claim_lattice_json_system_prompt": CLAIM_LATTICE_JSON_SYSTEM_PROMPT, + "claim_lattice_json_grounding_reminder": CLAIM_LATTICE_JSON_GROUNDING_REMINDER, "claim_lattice_use_guided_json": True, # JSON-mode stop sequences. Hermes-3-8B sometimes spams whitespace # / newlines after the closing brace on broad-descriptive shapes