arborist/aborist/qa/prompts.py
russell@unturf.com 2bbd12a270
docs(bench): rename qa-modes-bench-2026-05-02.md → qa-modes-bench.md
The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.

Naming pattern moving forward:

  docs/qa-modes-bench.md             — living journal (latest run)
  docs/qa-modes-bench-2026-04-30.md  — historical snapshot
                                       (frozen for the JSON-mode
                                       hardening day)

Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.

Updated references in:
  - CLAUDE.md (× 2)
  - aborist/qa/prompts.py
  - aborist/qa/query.py
  - docs/bench-maxing.md

The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
2026-05-02 12:27:38 -04:00

153 lines
7 KiB
Python

"""Shared claim-lattice prompt strings.
Single source of truth for the four prompt strings that travel
across both QA paths:
- ``aborist.qa.runner.ask()`` — single-document path
- ``aborist.qa.query.query()`` — multi-source retrieval path
Both paths use the same lattice prompts (pointer + JSON variants,
system + grounding-reminder each), so DRY them here. The
surrounding ``DEFAULT_POLICY`` / ``DEFAULT_QUERY_POLICY`` dicts
legitimately differ — different ``system_prompt`` / ``grounding_reminder``
for the single-document vs multi-source framing, different
``max_tokens``, different per-mode chunk caps. Only the
claim-lattice prompts (and the rules they encode) collapse to one
copy.
Bumping any of these strings folds into ``governance_policy_hash``
in both code paths simultaneously, so the cache namespace
partitions cleanly across the substrate. See
``docs/qa-modes-bench.md`` for the bench-driven
hardening history.
"""
from __future__ import annotations
# Pointer-line variant — the model emits prose-with-bracket-tags.
# The runtime owns quote text; the model never types a quote string
# (synthetic-elision-by-construction-impossible). Two-layer id
# discipline (pointer_id E1/E2/… vs content-addressed evidence_id)
# keeps the cache & run-DAG keyed on stable hashes while the model
# sees BPE-friendly short tags.
CLAIM_LATTICE_SYSTEM_PROMPT = (
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
"Answer using natural-language pointer-lines: one claim per "
"line, followed by a bracket tag with the pointer IDs that "
"directly support that claim.\n\n"
"WORKED EXAMPLE 1 — narrow factoid\n"
"---------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Apple_Inc | primary_answer_source) ===\n"
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and "
"Ronald Wayne in April 1976.\n\n"
"=== E2 (Steve_Wozniak | secondary_context_source) ===\n"
"Steve Wozniak co-founded Apple Computer Company alongside "
"Steve Jobs in 1976 and designed the Apple I.\n\n"
"QUESTION: who founded Apple?\n\n"
"ANSWER:\n"
"Steve Jobs co-founded Apple. [E1]\n"
"Steve Wozniak co-founded Apple. [E1,E2]\n"
"Ronald Wayne co-founded Apple. [E1]\n\n"
"WORKED EXAMPLE 2 — broad descriptive\n"
"------------------------------------\n"
"EVIDENCE:\n\n"
"=== E1 (Mars | primary_answer_source) ===\n"
"Mars is the fourth planet from the Sun. It has two moons, "
"Phobos and Deimos. Mars has a thin atmosphere of carbon "
"dioxide. Average surface temperature is around -60 "
"degrees Celsius.\n\n"
"QUESTION: tell me about Mars\n\n"
"ANSWER:\n"
"Mars is the fourth planet from the Sun. [E1]\n"
"Mars has two moons, Phobos and Deimos. [E1]\n"
"Mars has a thin atmosphere of carbon dioxide. [E1]\n"
"The average surface temperature on Mars is around -60 "
"degrees Celsius. [E1]\n\n"
"END OF EXAMPLES\n\n"
"RULES (each rule says what TO do):\n"
"1. Reference evidence by pointer ID. The runtime displays "
"the literal source span beside each claim — referencing is "
"your job; quoting is the runtime's job.\n"
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
"above.\n"
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
"directly contains the claim's key terms.\n"
"4. End every claim line with [E#] or [E#,E#].\n"
"5. Make a claim only when an EVIDENCE block textually "
"supports it. Stop when the evidence runs out — a short "
"answer is the right answer when only short evidence "
"exists.\n"
"6. Write each claim as one plain-prose sentence on its own "
"line.\n"
"7. Each line begins with the claim text and ends with a "
"bracket tag of pointer ids. The headers above the QUESTION "
"(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
"the input only — your answer is the claim lines themselves.\n"
"8. Each claim line cites at most two pointers. Lines with "
"three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms, or split the "
"claim into two lines.\n"
"9. Pick the pointer whose own text contains the claim's "
"specific facts — its dates, names, places, numbers, and "
"verbs. A chunk whose title matches the question's topic is "
"the right anchor only when that chunk's text also states "
"the claim's facts; when a different chunk states the fact "
"more directly, cite that one instead."
)
CLAIM_LATTICE_GROUNDING_REMINDER = (
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. At most two pointers per claim. "
"Begin each line with the claim text. Now answer the question "
"on the next message."
)
# JSON variant — pairs with grammar-constrained inference (vLLM
# guided_json, Claude/GPT-4 native JSON, Qwen 3.6 reasoner).
# Lenient pre-parser in ``aborist.qa.verify.verify_claim_lattice_json``
# keeps the path survivable on inference paths without grammar
# guidance.
CLAIM_LATTICE_JSON_SYSTEM_PROMPT = (
"You will see numbered EVIDENCE blocks tagged with short "
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
"object using EXACTLY this schema. Your output begins with "
"`{` and ends with `}`:\n\n"
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
"RULES:\n"
"1. Your output is exactly one JSON object — first character "
"is `{`, last character is `}`.\n"
"2. `text` is plain prose. Punctuation (including any quoted "
"names from the source) appears in the source span the "
"runtime renders for you, so write the text using only "
"letters, digits, spaces, and basic sentence punctuation "
"(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
"3. `evidence_ids` are pointer IDs that appear verbatim in "
"the EVIDENCE block headers above (E1, E2, …). Use only IDs "
"you can see in the headers above. At most two IDs per claim.\n"
"4. Each claim references at least one evidence_id.\n"
"5. Emit a claim only when an EVIDENCE block directly "
"supports it; stop emitting claims when supporting evidence "
"runs out — a short answer is the right answer when only "
"short evidence exists.\n"
"6. Each claim is one focused fact citing the source that "
"directly contains it. Multiple distinct facts become "
"multiple claim entries, each citing the source for that "
"fact. Three short claims, each citing one pointer, beat "
"one long claim citing five."
)
CLAIM_LATTICE_JSON_GROUNDING_REMINDER = (
"REMINDER: emit one JSON object — first character `{`, last "
"character `}` — with the schema "
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
"evidence_ids are pointer IDs (E1, E2, …) that appear "
"verbatim in the EVIDENCE block headers above. At most two "
"per claim. Each claim is one focused fact. Now answer the "
"question on the next message."
)