engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa(prompts): collapse lattice prompts to single source of truth

Browse source 
The four claim-lattice prompt strings —

  CLAIM_LATTICE_SYSTEM_PROMPT
  CLAIM_LATTICE_GROUNDING_REMINDER
  CLAIM_LATTICE_JSON_SYSTEM_PROMPT
  CLAIM_LATTICE_JSON_GROUNDING_REMINDER

— were duplicated byte-for-byte between aborist/qa/runner.py
DEFAULT_POLICY (single-document path) and aborist/qa/query.py
DEFAULT_QUERY_POLICY (multi-source path). Sprint 2 surfaced the
DRY violation when its agent had to update both files in lockstep
to add Rule 9. Two sources of truth = one source of drift.

Lift the four strings to aborist/qa/prompts.py as module-level
constants. Both DEFAULT_* dicts import them and reference by
name. The surrounding policy entries legitimately differ between
the two paths (different system_prompt/grounding_reminder for
single-doc vs multi-source framing, different max_tokens,
different chunk caps) — only the lattice prompts collapse.

Net shrinkage: -240 duplicated lines across runner.py + query.py,
+153 in prompts.py = ~87 lines removed. Sole authoritative copy
lives in prompts.py.

Verified via runtime equality check: both DEFAULT_POLICY and
DEFAULT_QUERY_POLICY resolve to the same string objects post-
import, matching the prompts.py constants byte-for-byte. Test
suite (751/34) passes clean in 11s with pytest -n auto.

No behavioural change. governance_policy_hash unchanged because
the strings themselves are unchanged. Cache namespace
unaffected.
This commit is contained in:
russell@unturf.com 2026-05-02 09:58:45 -04:00
parent e7650b8eee
commit 4b356030fb
No known key found for this signature in database
3 changed files with 173 additions and 220 deletions
Download patch file Download diff file Expand all files Collapse all files

153
aborist/qa/prompts.py Normal file
View file

@ -0,0 +1,153 @@
"""Shared claim-lattice prompt strings.
Single source of truth for the four prompt strings that travel
across both QA paths:
- ``aborist.qa.runner.ask()`` — single-document path
- ``aborist.qa.query.query()`` — multi-source retrieval path
Both paths use the same lattice prompts (pointer + JSON variants,
system + grounding-reminder each), so DRY them here. The
surrounding ``DEFAULT_POLICY`` / ``DEFAULT_QUERY_POLICY`` dicts
legitimately differ — different ``system_prompt`` / ``grounding_reminder``
for the single-document vs multi-source framing, different
``max_tokens``, different per-mode chunk caps. Only the
claim-lattice prompts (and the rules they encode) collapse to one
copy.
Bumping any of these strings folds into ``governance_policy_hash``
in both code paths simultaneously, so the cache namespace
partitions cleanly across the substrate. See
``docs/qa-modes-bench-2026-05-02.md`` for the bench-driven
hardening history.
"""
from __future__ import annotations
# Pointer-line variant — the model emits prose-with-bracket-tags.
# The runtime owns quote text; the model never types a quote string
# (synthetic-elision-by-construction-impossible). Two-layer id
# discipline (pointer_id E1/E2/… vs content-addressed evidence_id)
# keeps the cache & run-DAG keyed on stable hashes while the model
# sees BPE-friendly short tags.
CLAIM_LATTICE_SYSTEM_PROMPT = (
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
"Answer using natural-language pointer-lines: one claim per "
"line, followed by a bracket tag with the pointer IDs that "
"directly support that claim.\n\n"
"WORKED EXAMPLE 1 — narrow factoid\n"
"---------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Apple_Inc | primary_answer_source) ===\n"
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and "
"Ronald Wayne in April 1976.\n\n"
"=== E2 (Steve_Wozniak | secondary_context_source) ===\n"
"Steve Wozniak co-founded Apple Computer Company alongside "
"Steve Jobs in 1976 and designed the Apple I.\n\n"
"QUESTION: who founded Apple?\n\n"
"ANSWER:\n"
"Steve Jobs co-founded Apple. [E1]\n"
"Steve Wozniak co-founded Apple. [E1,E2]\n"
"Ronald Wayne co-founded Apple. [E1]\n\n"
"WORKED EXAMPLE 2 — broad descriptive\n"
"------------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Mars | primary_answer_source) ===\n"
"Mars is the fourth planet from the Sun. It has two moons, "
"Phobos and Deimos. Mars has a thin atmosphere of carbon "
"dioxide. Average surface temperature is around -60 "
"degrees Celsius.\n\n"
"QUESTION: tell me about Mars\n\n"
"ANSWER:\n"
"Mars is the fourth planet from the Sun. [E1]\n"
"Mars has two moons, Phobos and Deimos. [E1]\n"
"Mars has a thin atmosphere of carbon dioxide. [E1]\n"
"The average surface temperature on Mars is around -60 "
"degrees Celsius. [E1]\n\n"
"END OF EXAMPLES\n\n"
"RULES (each rule says what TO do):\n"
"1. Reference evidence by pointer ID. The runtime displays "
"the literal source span beside each claim — referencing is "
"your job; quoting is the runtime's job.\n"
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
"above.\n"
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
"directly contains the claim's key terms.\n"
"4. End every claim line with [E#] or [E#,E#].\n"
"5. Make a claim only when an EVIDENCE block textually "
"supports it. Stop when the evidence runs out — a short "
"answer is the right answer when only short evidence "
"exists.\n"
"6. Write each claim as one plain-prose sentence on its own "
"line.\n"
"7. Each line begins with the claim text and ends with a "
"bracket tag of pointer ids. The headers above the QUESTION "
"(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
"the input only — your answer is the claim lines themselves.\n"
"8. Each claim line cites at most two pointers. Lines with "
"three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms, or split the "
"claim into two lines.\n"
"9. Pick the pointer whose own text contains the claim's "
"specific facts — its dates, names, places, numbers, and "
"verbs. A chunk whose title matches the question's topic is "
"the right anchor only when that chunk's text also states "
"the claim's facts; when a different chunk states the fact "
"more directly, cite that one instead."
)
CLAIM_LATTICE_GROUNDING_REMINDER = (
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. At most two pointers per claim. "
"Begin each line with the claim text. Now answer the question "
"on the next message."
)
# JSON variant — pairs with grammar-constrained inference (vLLM
# guided_json, Claude/GPT-4 native JSON, Qwen 3.6 reasoner).
# Lenient pre-parser in ``aborist.qa.verify.verify_claim_lattice_json``
# keeps the path survivable on inference paths without grammar
# guidance.
CLAIM_LATTICE_JSON_SYSTEM_PROMPT = (
"You will see numbered EVIDENCE blocks tagged with short "
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
"object using EXACTLY this schema. Your output begins with "
"`{` and ends with `}`:\n\n"
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
"RULES:\n"
"1. Your output is exactly one JSON object — first character "
"is `{`, last character is `}`.\n"
"2. `text` is plain prose. Punctuation (including any quoted "
"names from the source) appears in the source span the "
"runtime renders for you, so write the text using only "
"letters, digits, spaces, and basic sentence punctuation "
"(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
"3. `evidence_ids` are pointer IDs that appear verbatim in "
"the EVIDENCE block headers above (E1, E2, …). Use only IDs "
"you can see in the headers above. At most two IDs per claim.\n"
"4. Each claim references at least one evidence_id.\n"
"5. Emit a claim only when an EVIDENCE block directly "
"supports it; stop emitting claims when supporting evidence "
"runs out — a short answer is the right answer when only "
"short evidence exists.\n"
"6. Each claim is one focused fact citing the source that "
"directly contains it. Multiple distinct facts become "
"multiple claim entries, each citing the source for that "
"fact. Three short claims, each citing one pointer, beat "
"one long claim citing five."
)
CLAIM_LATTICE_JSON_GROUNDING_REMINDER = (
"REMINDER: emit one JSON object — first character `{`, last "
"character `}` — with the schema "
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
"evidence_ids are pointer IDs (E1, E2, …) that appear "
"verbatim in the EVIDENCE block headers above. At most two "
"per claim. Each claim is one focused fact. Now answer the "
"question on the next message."
)

120
aborist/qa/query.py
View file

@ -47,6 +47,12 @@ from aborist import (
from aborist.compress import unpack_chunk
from aborist.merkle import MerkleTree
from aborist.qa.client import ChatClient
from aborist.qa.prompts import (
CLAIM_LATTICE_GROUNDING_REMINDER,
CLAIM_LATTICE_JSON_GROUNDING_REMINDER,
CLAIM_LATTICE_JSON_SYSTEM_PROMPT,
CLAIM_LATTICE_SYSTEM_PROMPT,
)
from aborist.qa.concepts import (
has_compare_phrasing,
rivalry_excluded,
@ -321,79 +327,8 @@ DEFAULT_QUERY_POLICY = {
# output ("Claim. [E12]") that the verifier maps back to
# content-addressed evidence_ids for the cache & run-DAG.
"answer_mode": DEFAULT_ANSWER_MODE,
"claim_lattice_system_prompt": (
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
"Answer using natural-language pointer-lines: one claim per "
"line, followed by a bracket tag with the pointer IDs that "
"directly support that claim.\n\n"
"WORKED EXAMPLE 1 — narrow factoid\n"
"---------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Apple_Inc | primary_answer_source) ===\n"
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and "
"Ronald Wayne in April 1976.\n\n"
"=== E2 (Steve_Wozniak | secondary_context_source) ===\n"
"Steve Wozniak co-founded Apple Computer Company alongside "
"Steve Jobs in 1976 and designed the Apple I.\n\n"
"QUESTION: who founded Apple?\n\n"
"ANSWER:\n"
"Steve Jobs co-founded Apple. [E1]\n"
"Steve Wozniak co-founded Apple. [E1,E2]\n"
"Ronald Wayne co-founded Apple. [E1]\n\n"
"WORKED EXAMPLE 2 — broad descriptive\n"
"------------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Mars | primary_answer_source) ===\n"
"Mars is the fourth planet from the Sun. It has two moons, "
"Phobos and Deimos. Mars has a thin atmosphere of carbon "
"dioxide. Average surface temperature is around -60 "
"degrees Celsius.\n\n"
"QUESTION: tell me about Mars\n\n"
"ANSWER:\n"
"Mars is the fourth planet from the Sun. [E1]\n"
"Mars has two moons, Phobos and Deimos. [E1]\n"
"Mars has a thin atmosphere of carbon dioxide. [E1]\n"
"The average surface temperature on Mars is around -60 "
"degrees Celsius. [E1]\n\n"
"END OF EXAMPLES\n\n"
"RULES (each rule says what TO do):\n"
"1. Reference evidence by pointer ID. The runtime displays "
"the literal source span beside each claim — referencing is "
"your job; quoting is the runtime's job.\n"
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
"above.\n"
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
"directly contains the claim's key terms.\n"
"4. End every claim line with [E#] or [E#,E#].\n"
"5. Make a claim only when an EVIDENCE block textually "
"supports it. Stop when the evidence runs out — a short "
"answer is the right answer when only short evidence "
"exists.\n"
"6. Write each claim as one plain-prose sentence on its own "
"line.\n"
"7. Each line begins with the claim text and ends with a "
"bracket tag of pointer ids. The headers above the QUESTION "
"(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
"the input only — your answer is the claim lines themselves.\n"
"8. Each claim line cites at most two pointers. Lines with "
"three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms, or split the "
"claim into two lines.\n"
"9. Pick the pointer whose own text contains the claim's "
"specific facts — its dates, names, places, numbers, and "
"verbs. A chunk whose title matches the question's topic is "
"the right anchor only when that chunk's text also states "
"the claim's facts; when a different chunk states the fact "
"more directly, cite that one instead."
),
"claim_lattice_grounding_reminder": (
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. At most two pointers per claim. "
"Begin each line with the claim text. Now answer the question "
"on the next message."
),
"claim_lattice_system_prompt": CLAIM_LATTICE_SYSTEM_PROMPT,
"claim_lattice_grounding_reminder": CLAIM_LATTICE_GROUNDING_REMINDER,
# Reference-frame polarity preamble (Ticket #000002 / Module L).
# Injected as an additional user-role message before the
# grounding_reminder when `detect_frame` classifies the query as
@ -458,43 +393,8 @@ DEFAULT_QUERY_POLICY = {
# survivable on inference paths without grammar guidance. Toggling
# `claim_lattice_use_guided_json=False` disables the extra_body
# pass for endpoints that 400 on unknown fields.
"claim_lattice_json_system_prompt": (
"You will see numbered EVIDENCE blocks tagged with short "
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
"object using EXACTLY this schema. Your output begins with "
"`{` and ends with `}`:\n\n"
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
"RULES:\n"
"1. Your output is exactly one JSON object — first character "
"is `{`, last character is `}`.\n"
"2. `text` is plain prose. Punctuation (including any quoted "
"names from the source) appears in the source span the "
"runtime renders for you, so write the text using only "
"letters, digits, spaces, and basic sentence punctuation "
"(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
"3. `evidence_ids` are pointer IDs that appear verbatim in "
"the EVIDENCE block headers above (E1, E2, …). Use only IDs "
"you can see in the headers above. At most two IDs per claim.\n"
"4. Each claim references at least one evidence_id.\n"
"5. Emit a claim only when an EVIDENCE block directly "
"supports it; stop emitting claims when supporting evidence "
"runs out — a short answer is the right answer when only "
"short evidence exists.\n"
"6. Each claim is one focused fact citing the source that "
"directly contains it. Multiple distinct facts become "
"multiple claim entries, each citing the source for that "
"fact. Three short claims, each citing one pointer, beat "
"one long claim citing five."
),
"claim_lattice_json_grounding_reminder": (
"REMINDER: emit one JSON object — first character `{`, last "
"character `}` — with the schema "
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
"evidence_ids are pointer IDs (E1, E2, …) that appear "
"verbatim in the EVIDENCE block headers above. At most two "
"per claim. Each claim is one focused fact. Now answer the "
"question on the next message."
),
"claim_lattice_json_system_prompt": CLAIM_LATTICE_JSON_SYSTEM_PROMPT,
"claim_lattice_json_grounding_reminder": CLAIM_LATTICE_JSON_GROUNDING_REMINDER,
"claim_lattice_use_guided_json": True,
"claim_lattice_json_stop_sequences": ["\n\n"],
# Per-mode context-budget defaults. Sprint 1b (2026-05-02) bench

120
aborist/qa/runner.py
View file

@ -22,6 +22,12 @@ from aborist import (
from aborist.compress import unpack_chunk
from aborist.merkle import MerkleTree, proof_to_dict
from aborist.qa.client import ChatClient
from aborist.qa.prompts import (
CLAIM_LATTICE_GROUNDING_REMINDER,
CLAIM_LATTICE_JSON_GROUNDING_REMINDER,
CLAIM_LATTICE_JSON_SYSTEM_PROMPT,
CLAIM_LATTICE_SYSTEM_PROMPT,
)
from aborist.qa.keys import (
DEFAULT_FIDELITY,
DEFAULT_QUESTION_DEDUP,
@ -112,79 +118,8 @@ DEFAULT_POLICY = {
# different cache_keys and never alias. No iterative repair in
# pointer mode (one-shot benchmark discipline).
"answer_mode": DEFAULT_ANSWER_MODE,
"claim_lattice_system_prompt": (
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
"Answer using natural-language pointer-lines: one claim per "
"line, followed by a bracket tag with the pointer IDs that "
"directly support that claim.\n\n"
"WORKED EXAMPLE 1 — narrow factoid\n"
"---------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Apple_Inc | primary_answer_source) ===\n"
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and "
"Ronald Wayne in April 1976.\n\n"
"=== E2 (Steve_Wozniak | secondary_context_source) ===\n"
"Steve Wozniak co-founded Apple Computer Company alongside "
"Steve Jobs in 1976 and designed the Apple I.\n\n"
"QUESTION: who founded Apple?\n\n"
"ANSWER:\n"
"Steve Jobs co-founded Apple. [E1]\n"
"Steve Wozniak co-founded Apple. [E1,E2]\n"
"Ronald Wayne co-founded Apple. [E1]\n\n"
"WORKED EXAMPLE 2 — broad descriptive\n"
"------------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Mars | primary_answer_source) ===\n"
"Mars is the fourth planet from the Sun. It has two moons, "
"Phobos and Deimos. Mars has a thin atmosphere of carbon "
"dioxide. Average surface temperature is around -60 "
"degrees Celsius.\n\n"
"QUESTION: tell me about Mars\n\n"
"ANSWER:\n"
"Mars is the fourth planet from the Sun. [E1]\n"
"Mars has two moons, Phobos and Deimos. [E1]\n"
"Mars has a thin atmosphere of carbon dioxide. [E1]\n"
"The average surface temperature on Mars is around -60 "
"degrees Celsius. [E1]\n\n"
"END OF EXAMPLES\n\n"
"RULES (each rule says what TO do):\n"
"1. Reference evidence by pointer ID. The runtime displays "
"the literal source span beside each claim — referencing is "
"your job; quoting is the runtime's job.\n"
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
"above.\n"
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
"directly contains the claim's key terms.\n"
"4. End every claim line with [E#] or [E#,E#].\n"
"5. Make a claim only when an EVIDENCE block textually "
"supports it. Stop when the evidence runs out — a short "
"answer is the right answer when only short evidence "
"exists.\n"
"6. Write each claim as one plain-prose sentence on its own "
"line.\n"
"7. Each line begins with the claim text and ends with a "
"bracket tag of pointer ids. The headers above the QUESTION "
"(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
"the input only — your answer is the claim lines themselves.\n"
"8. Each claim line cites at most two pointers. Lines with "
"three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms, or split the "
"claim into two lines.\n"
"9. Pick the pointer whose own text contains the claim's "
"specific facts — its dates, names, places, numbers, and "
"verbs. A chunk whose title matches the question's topic is "
"the right anchor only when that chunk's text also states "
"the claim's facts; when a different chunk states the fact "
"more directly, cite that one instead."
),
"claim_lattice_grounding_reminder": (
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. At most two pointers per claim. "
"Begin each line with the claim text. Now answer the question "
"on the next message."
),
"claim_lattice_system_prompt": CLAIM_LATTICE_SYSTEM_PROMPT,
"claim_lattice_grounding_reminder": CLAIM_LATTICE_GROUNDING_REMINDER,
# Allowed source roles for claim-lattice verification. Roles outside
# this set get classified SOURCE_ROLE_BLOCKED and downgrade the
# verdict. Mirrors aborist.qa.verify.DEFAULT_ALLOWED_SOURCE_ROLES;
@ -263,43 +198,8 @@ DEFAULT_POLICY = {
# (vLLM guided_json, Claude/GPT-4 native JSON, Qwen 3.6 reasoner).
# Lenient pre-parser in verify_claim_lattice_json keeps the path
# survivable on inference paths without grammar guidance.
"claim_lattice_json_system_prompt": (
"You will see numbered EVIDENCE blocks tagged with short "
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
"object using EXACTLY this schema. Your output begins with "
"`{` and ends with `}`:\n\n"
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
"RULES:\n"
"1. Your output is exactly one JSON object — first character "
"is `{`, last character is `}`.\n"
"2. `text` is plain prose. Punctuation (including any quoted "
"names from the source) appears in the source span the "
"runtime renders for you, so write the text using only "
"letters, digits, spaces, and basic sentence punctuation "
"(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
"3. `evidence_ids` are pointer IDs that appear verbatim in "
"the EVIDENCE block headers above (E1, E2, …). Use only IDs "
"you can see in the headers above. At most two IDs per claim.\n"
"4. Each claim references at least one evidence_id.\n"
"5. Emit a claim only when an EVIDENCE block directly "
"supports it; stop emitting claims when supporting evidence "
"runs out — a short answer is the right answer when only "
"short evidence exists.\n"
"6. Each claim is one focused fact citing the source that "
"directly contains it. Multiple distinct facts become "
"multiple claim entries, each citing the source for that "
"fact. Three short claims, each citing one pointer, beat "
"one long claim citing five."
),
"claim_lattice_json_grounding_reminder": (
"REMINDER: emit one JSON object — first character `{`, last "
"character `}` — with the schema "
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
"evidence_ids are pointer IDs (E1, E2, …) that appear "
"verbatim in the EVIDENCE block headers above. At most two "
"per claim. Each claim is one focused fact. Now answer the "
"question on the next message."
),
"claim_lattice_json_system_prompt": CLAIM_LATTICE_JSON_SYSTEM_PROMPT,
"claim_lattice_json_grounding_reminder": CLAIM_LATTICE_JSON_GROUNDING_REMINDER,
"claim_lattice_use_guided_json": True,
# JSON-mode stop sequences. Hermes-3-8B sometimes spams whitespace
# / newlines after the closing brace on broad-descriptive shapes