The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
153 lines
7 KiB
Python
153 lines
7 KiB
Python
"""Shared claim-lattice prompt strings.
|
|
|
|
Single source of truth for the four prompt strings that travel
|
|
across both QA paths:
|
|
|
|
- ``aborist.qa.runner.ask()`` — single-document path
|
|
- ``aborist.qa.query.query()`` — multi-source retrieval path
|
|
|
|
Both paths use the same lattice prompts (pointer + JSON variants,
|
|
system + grounding-reminder each), so DRY them here. The
|
|
surrounding ``DEFAULT_POLICY`` / ``DEFAULT_QUERY_POLICY`` dicts
|
|
legitimately differ — different ``system_prompt`` / ``grounding_reminder``
|
|
for the single-document vs multi-source framing, different
|
|
``max_tokens``, different per-mode chunk caps. Only the
|
|
claim-lattice prompts (and the rules they encode) collapse to one
|
|
copy.
|
|
|
|
Bumping any of these strings folds into ``governance_policy_hash``
|
|
in both code paths simultaneously, so the cache namespace
|
|
partitions cleanly across the substrate. See
|
|
``docs/qa-modes-bench.md`` for the bench-driven
|
|
hardening history.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
|
|
# Pointer-line variant — the model emits prose-with-bracket-tags.
|
|
# The runtime owns quote text; the model never types a quote string
|
|
# (synthetic-elision-by-construction-impossible). Two-layer id
|
|
# discipline (pointer_id E1/E2/… vs content-addressed evidence_id)
|
|
# keeps the cache & run-DAG keyed on stable hashes while the model
|
|
# sees BPE-friendly short tags.
|
|
CLAIM_LATTICE_SYSTEM_PROMPT = (
|
|
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
|
|
"Answer using natural-language pointer-lines: one claim per "
|
|
"line, followed by a bracket tag with the pointer IDs that "
|
|
"directly support that claim.\n\n"
|
|
"WORKED EXAMPLE 1 — narrow factoid\n"
|
|
"---------------------------------\n"
|
|
"EVIDENCE:\n\n"
|
|
"=== E1 (Apple_Inc | primary_answer_source) ===\n"
|
|
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and "
|
|
"Ronald Wayne in April 1976.\n\n"
|
|
"=== E2 (Steve_Wozniak | secondary_context_source) ===\n"
|
|
"Steve Wozniak co-founded Apple Computer Company alongside "
|
|
"Steve Jobs in 1976 and designed the Apple I.\n\n"
|
|
"QUESTION: who founded Apple?\n\n"
|
|
"ANSWER:\n"
|
|
"Steve Jobs co-founded Apple. [E1]\n"
|
|
"Steve Wozniak co-founded Apple. [E1,E2]\n"
|
|
"Ronald Wayne co-founded Apple. [E1]\n\n"
|
|
"WORKED EXAMPLE 2 — broad descriptive\n"
|
|
"------------------------------------\n"
|
|
"EVIDENCE:\n\n"
|
|
"=== E1 (Mars | primary_answer_source) ===\n"
|
|
"Mars is the fourth planet from the Sun. It has two moons, "
|
|
"Phobos and Deimos. Mars has a thin atmosphere of carbon "
|
|
"dioxide. Average surface temperature is around -60 "
|
|
"degrees Celsius.\n\n"
|
|
"QUESTION: tell me about Mars\n\n"
|
|
"ANSWER:\n"
|
|
"Mars is the fourth planet from the Sun. [E1]\n"
|
|
"Mars has two moons, Phobos and Deimos. [E1]\n"
|
|
"Mars has a thin atmosphere of carbon dioxide. [E1]\n"
|
|
"The average surface temperature on Mars is around -60 "
|
|
"degrees Celsius. [E1]\n\n"
|
|
"END OF EXAMPLES\n\n"
|
|
"RULES (each rule says what TO do):\n"
|
|
"1. Reference evidence by pointer ID. The runtime displays "
|
|
"the literal source span beside each claim — referencing is "
|
|
"your job; quoting is the runtime's job.\n"
|
|
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
|
|
"above.\n"
|
|
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
|
|
"directly contains the claim's key terms.\n"
|
|
"4. End every claim line with [E#] or [E#,E#].\n"
|
|
"5. Make a claim only when an EVIDENCE block textually "
|
|
"supports it. Stop when the evidence runs out — a short "
|
|
"answer is the right answer when only short evidence "
|
|
"exists.\n"
|
|
"6. Write each claim as one plain-prose sentence on its own "
|
|
"line.\n"
|
|
"7. Each line begins with the claim text and ends with a "
|
|
"bracket tag of pointer ids. The headers above the QUESTION "
|
|
"(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
|
|
"the input only — your answer is the claim lines themselves.\n"
|
|
"8. Each claim line cites at most two pointers. Lines with "
|
|
"three or more pointers are rejected. When more than two "
|
|
"evidence blocks support a claim, pick the two that most "
|
|
"directly contain the claim's key terms, or split the "
|
|
"claim into two lines.\n"
|
|
"9. Pick the pointer whose own text contains the claim's "
|
|
"specific facts — its dates, names, places, numbers, and "
|
|
"verbs. A chunk whose title matches the question's topic is "
|
|
"the right anchor only when that chunk's text also states "
|
|
"the claim's facts; when a different chunk states the fact "
|
|
"more directly, cite that one instead."
|
|
)
|
|
|
|
|
|
CLAIM_LATTICE_GROUNDING_REMINDER = (
|
|
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
|
|
"line, plain prose with bracket tags. Pointer IDs come from "
|
|
"the EVIDENCE blocks above. At most two pointers per claim. "
|
|
"Begin each line with the claim text. Now answer the question "
|
|
"on the next message."
|
|
)
|
|
|
|
|
|
# JSON variant — pairs with grammar-constrained inference (vLLM
|
|
# guided_json, Claude/GPT-4 native JSON, Qwen 3.6 reasoner).
|
|
# Lenient pre-parser in ``aborist.qa.verify.verify_claim_lattice_json``
|
|
# keeps the path survivable on inference paths without grammar
|
|
# guidance.
|
|
CLAIM_LATTICE_JSON_SYSTEM_PROMPT = (
|
|
"You will see numbered EVIDENCE blocks tagged with short "
|
|
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
|
|
"object using EXACTLY this schema. Your output begins with "
|
|
"`{` and ends with `}`:\n\n"
|
|
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
|
|
"RULES:\n"
|
|
"1. Your output is exactly one JSON object — first character "
|
|
"is `{`, last character is `}`.\n"
|
|
"2. `text` is plain prose. Punctuation (including any quoted "
|
|
"names from the source) appears in the source span the "
|
|
"runtime renders for you, so write the text using only "
|
|
"letters, digits, spaces, and basic sentence punctuation "
|
|
"(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
|
|
"3. `evidence_ids` are pointer IDs that appear verbatim in "
|
|
"the EVIDENCE block headers above (E1, E2, …). Use only IDs "
|
|
"you can see in the headers above. At most two IDs per claim.\n"
|
|
"4. Each claim references at least one evidence_id.\n"
|
|
"5. Emit a claim only when an EVIDENCE block directly "
|
|
"supports it; stop emitting claims when supporting evidence "
|
|
"runs out — a short answer is the right answer when only "
|
|
"short evidence exists.\n"
|
|
"6. Each claim is one focused fact citing the source that "
|
|
"directly contains it. Multiple distinct facts become "
|
|
"multiple claim entries, each citing the source for that "
|
|
"fact. Three short claims, each citing one pointer, beat "
|
|
"one long claim citing five."
|
|
)
|
|
|
|
|
|
CLAIM_LATTICE_JSON_GROUNDING_REMINDER = (
|
|
"REMINDER: emit one JSON object — first character `{`, last "
|
|
"character `}` — with the schema "
|
|
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
|
|
"evidence_ids are pointer IDs (E1, E2, …) that appear "
|
|
"verbatim in the EVIDENCE block headers above. At most two "
|
|
"per claim. Each claim is one focused fact. Now answer the "
|
|
"question on the next message."
|
|
)
|