engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa: positive-form prompts + atomic-claim rule; great-wall fixture

Browse source 
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:

  "Do NOT invent IDs" → "Use only the IDs that appear in the
                         EVIDENCE blocks above"
  "Never begin a line with E1:" → "Each line begins with the
                                    claim text, then a bracket tag"
  "No commentary, no preamble" → "Your output begins with `{`
                                   and ends with `}`"
  "If no evidence supports a claim, omit the claim" →
    "Emit only claims that an EVIDENCE block directly supports"

Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.

Bench delta (n=3, 25 questions, 75 runs per mode):

  quote   30S 28H 17U → 40S 23H 12U   +10 STRICT, +13pp rate
  pointer 27S 37H 11U → 23S 37H 15U   -4 STRICT  (acceptable —
                                       JSON is now the daily
                                       driver; pointer is
                                       fallback/diagnostic)
  JSON    36S 28H 11U → 44S 26H  5U   +8 STRICT, +11pp rate,
                                       -6 UNGROUNDED, -1.8s latency

Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.

Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.

CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
This commit is contained in:
russell@unturf.com 2026-04-30 18:00:22 -04:00
parent 96edaef27f
commit b5925c8cce
No known key found for this signature in database
5 changed files with 168 additions and 79 deletions
Download patch file Download diff file Expand all files Collapse all files

17
CLAUDE.md
View file

@ -146,10 +146,23 @@ during the JSON-mode hardening journey:
prompt change "should help" are routinely wrong. prompt change "should help" are routinely wrong.
- **Slight prompt adjustments are fine — local-minima are not.** - **Slight prompt adjustments are fine — local-minima are not.**
Prompts can be nudged with a worked example or a single-sentence Prompts can be nudged with a worked example or a single-sentence
rule. They should NOT be padded with N rules that incentivize rule. They should NOT be padded with many rules that incentivize
empty / over-cautious output ("if uncertain, omit" can collapse empty / over-cautious output ("if uncertain, omit" can collapse
the model into refusing valid answers). When in doubt, change the model into refusing valid answers). When in doubt, change
ONE thing per bench cycle. one thing per bench cycle.
- **Avoid negation in every prompt.** Small instruction-tuned models
(Hermes-3-8B observed) struggle with "do NOT X" / "never Y" /
"without Z" — the negation often pattern-matches away or inverts
under attention. Rewrite every rule to its positive form. Some
swaps that work:
- "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
- "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
- "No commentary, no preamble" → "Your output begins with `{` and ends with `}`"
- "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
- "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"
Audit all the `*_system_prompt` and `*_grounding_reminder` policy
fields for residual negation when a model behaves erratically —
it is the cheapest fix in the prompt-iteration toolkit.
- **Bench is the scoreboard; live fixtures are the gates.** Bench - **Bench is the scoreboard; live fixtures are the gates.** Bench
measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q
samples). Live fixtures (`tests/test_qa_quality_live.py`) gate samples). Live fixtures (`tests/test_qa_quality_live.py`) gate

104
aborist/qa/query.py
View file

@ -219,24 +219,26 @@ DEFAULT_QUERY_POLICY = {
"GROUNDING RULE (most important):\n" "GROUNDING RULE (most important):\n"
"For EVERY factual claim in your answer, include a verbatim quote " "For EVERY factual claim in your answer, include a verbatim quote "
"from a source enclosed in double quotes (\"...\"). The quoted span " "from a source enclosed in double quotes (\"...\"). The quoted span "
"must appear word-for-word in one of the sources. If you cannot " "must appear word-for-word in one of the sources. Make a claim "
"find a verbatim quote that supports a claim, do not make the " "only when a verbatim quote in one of the sources supports it. "
"claim. Do not paraphrase inside the quotes.\n\n" "Quote the source word-for-word inside the double quotes.\n\n"
"ATTRIBUTION RULES:\n" "ATTRIBUTION RULES:\n"
"1. A single source may discuss MULTIPLE products, companies, or " "1. A single source may discuss MULTIPLE products, companies, or "
"competitors. Read carefully and only attribute facts to the entity " "competitors. Read carefully and attribute facts only to the entity "
"the source explicitly names for that fact.\n" "the source explicitly names for that fact.\n"
"2. If the question asks about ONE specific entity (e.g., 'fastest " "2. When the question asks about ONE specific entity (e.g., 'fastest "
"AMD CPU'), do NOT include facts about competitors (Intel, Pentium) " "AMD CPU'), keep facts about that entity in the answer; leave facts "
"as if they were facts about the asked entity. They are different " "about competitors (Intel, Pentium) out, even when the source "
"products even when discussed in the same article.\n" "discusses both — they are different products that happen to share "
"an article.\n"
"3. When referencing model numbers like 'Athlon XP 3200+', remember " "3. When referencing model numbers like 'Athlon XP 3200+', remember "
"that AMD's PR-rating numbers (3200+, 2500+) are NOT the clock speed " "that AMD's PR-rating numbers (3200+, 2500+) are model labels, not "
"in MHz. Quote the source's own wording for clock speeds rather than " "the clock speed in MHz. Quote the source's own wording for clock "
"interpreting model numbers.\n" "speeds; let the source state the speed.\n"
"4. If the sources do not contain the answer, say 'I don't know " "4. When the sources contain the answer, write it. When the answer "
"based on the provided sources.' Do not fall back on background " "is absent from the sources, say 'I don't know based on the "
"knowledge from training." "provided sources.' and stop there. Stay inside the sources at "
"all times — let the corpus speak."
), ),
# Restated rule fired as a user message immediately before the sources # Restated rule fired as a user message immediately before the sources
# turn. Hermes (and most instruction-tuned 8B models) follow recent # turn. Hermes (and most instruction-tuned 8B models) follow recent
@ -244,9 +246,9 @@ DEFAULT_QUERY_POLICY = {
# decays under long context. Repetition is not redundancy — it raises # decays under long context. Repetition is not redundancy — it raises
# the prior on the response shape we want. # the prior on the response shape we want.
"grounding_reminder": ( "grounding_reminder": (
"REMINDER: every factual claim in your reply must be wrapped in " "REMINDER: wrap every factual claim in double quotes (\"...\") "
"double quotes (\"...\") and the quoted span must appear " "and the quoted span must appear word-for-word in one of the "
"word-for-word in one of the sources. No quote, no claim. " "sources. Each claim earns a verbatim quote. "
"Example:\n\n" "Example:\n\n"
" Q: who founded Apple?\n" " Q: who founded Apple?\n"
" Sources: ...Apple Inc. was founded by Steve Jobs, Steve " " Sources: ...Apple Inc. was founded by Steve Jobs, Steve "
@ -345,22 +347,26 @@ DEFAULT_QUERY_POLICY = {
"exists.\n" "exists.\n"
"6. Write each claim as one plain-prose sentence on its own " "6. Write each claim as one plain-prose sentence on its own "
"line.\n" "line.\n"
"7. Your answer goes only in the ANSWER position. Never " "7. Each line begins with the claim text and ends with a "
"begin a line with the word EVIDENCE or with a pointer-id " "bracket tag of pointer ids. The headers above the QUESTION "
"prefix like E1: or E2: — those tags belong only above the " "(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
"QUESTION, never in your output.\n" "the input only — your answer is the claim lines themselves.\n"
"8. Each claim line cites at most two pointers. Lines with " "8. Each claim line cites at most two pointers. Lines with "
"three or more pointers are rejected. If more than two " "three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most " "evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms; or split the " "directly contain the claim's key terms, or split the "
"claim into two lines." "claim into two lines.\n"
"9. Each claim is one focused fact. Multiple distinct facts "
"become multiple claim lines, each citing the source that "
"directly contains it. Three short claims, each citing one "
"pointer, beat one long claim citing five."
), ),
"claim_lattice_grounding_reminder": ( "claim_lattice_grounding_reminder": (
"REMINDER: format = pointer-line — `Claim text. [E1]` per " "REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from " "line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. At most two pointers per claim. " "the EVIDENCE blocks above. At most two pointers per claim. "
"No `EVIDENCE:` header in your answer, no `E#:` line " "Begin each line with the claim text. Now answer the question "
"prefix. Now answer the question on the next message." "on the next message."
), ),
"claim_lattice_allowed_source_roles": [ "claim_lattice_allowed_source_roles": [
"primary_answer_source", "primary_answer_source",
@ -393,31 +399,39 @@ DEFAULT_QUERY_POLICY = {
"claim_lattice_json_system_prompt": ( "claim_lattice_json_system_prompt": (
"You will see numbered EVIDENCE blocks tagged with short " "You will see numbered EVIDENCE blocks tagged with short "
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON " "pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
"object using EXACTLY this schema, with NO prose, NO " "object using EXACTLY this schema. Your output begins with "
"markdown fences, NO preamble:\n\n" "`{` and ends with `}`:\n\n"
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n' ' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
"RULES:\n" "RULES:\n"
"1. Output a single JSON object. No code fences. No commentary.\n" "1. Your output is exactly one JSON object — first character "
"2. `text` is plain prose with NO double-quote characters " "is `{`, last character is `}`.\n"
"anywhere — the JSON string-quotes are not the same as quoted " "2. `text` is plain prose. Punctuation (including any quoted "
"spans inside the text. If you need to mention a name " "names from the source) appears in the source span the "
"containing punctuation, use the source's own form without " "runtime renders for you, so write the text using only "
"wrapping it in quotes.\n" "letters, digits, spaces, and basic sentence punctuation "
"3. `evidence_ids` MUST be pointer IDs that appear verbatim " "(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
"in the EVIDENCE block headers above (E1, E2, …). Do NOT " "3. `evidence_ids` are pointer IDs that appear verbatim in "
"invent IDs — only IDs that already exist above. At most " "the EVIDENCE block headers above (E1, E2, …). Use only IDs "
"two IDs per claim.\n" "you can see in the headers above. At most two IDs per claim.\n"
"4. Each claim must reference at least one evidence_id.\n" "4. Each claim references at least one evidence_id.\n"
"5. If no evidence supports a claim, omit the claim." "5. Emit a claim only when an EVIDENCE block directly "
"supports it; stop emitting claims when supporting evidence "
"runs out — a short answer is the right answer when only "
"short evidence exists.\n"
"6. Each claim is one focused fact citing the source that "
"directly contains it. Multiple distinct facts become "
"multiple claim entries, each citing the source for that "
"fact. Three short claims, each citing one pointer, beat "
"one long claim citing five."
), ),
"claim_lattice_json_grounding_reminder": ( "claim_lattice_json_grounding_reminder": (
"REMINDER: emit a single JSON object with the exact schema " "REMINDER: emit one JSON object — first character `{`, last "
"character `}` — with the schema "
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. ' '`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
"evidence_ids are pointer IDs (E1, E2, …) that appear " "evidence_ids are pointer IDs (E1, E2, …) that appear "
"verbatim in the EVIDENCE block headers — never invented. " "verbatim in the EVIDENCE block headers above. At most two "
"At most two per claim. Text has no double-quote characters. " "per claim. Each claim is one focused fact. Now answer the "
"No code fences, no preamble. " "question on the next message."
"Now answer the question on the next message."
), ),
"claim_lattice_use_guided_json": True, "claim_lattice_use_guided_json": True,
"claim_lattice_json_stop_sequences": ["\n\n"], "claim_lattice_json_stop_sequences": ["\n\n"],

80
aborist/qa/runner.py
View file

@ -65,19 +65,21 @@ DEFAULT_POLICY = {
"Answer the user's question based ONLY on the document below. " "Answer the user's question based ONLY on the document below. "
"For EVERY factual claim, include a verbatim quote from the " "For EVERY factual claim, include a verbatim quote from the "
"document enclosed in double quotes (\"...\"). The quoted span " "document enclosed in double quotes (\"...\"). The quoted span "
"must appear word-for-word. If you cannot find a verbatim quote " "must appear word-for-word. Make a claim only when a verbatim "
"supporting a claim, do not make the claim. " "quote in the document directly supports it. "
"If the answer is not in the document, say 'I don't know based " "If the answer is in the document, write it. "
"on the provided document.' Do not speculate." "If the answer is absent from the document, say 'I don't know "
"based on the provided document.' and stop there. "
"Stay inside the document at all times."
), ),
# Restated rule fired as a user message right before the document + # Restated rule fired as a user message right before the document +
# question arrive. See aborist/qa/query.py for the rationale (recent # question arrive. See aborist/qa/query.py for the rationale (recent
# user-turn instructions outweigh decayed system-turn rules in 8B # user-turn instructions outweigh decayed system-turn rules in 8B
# instruction-tuned models). # instruction-tuned models).
"grounding_reminder": ( "grounding_reminder": (
"REMINDER: every factual claim in your reply must be wrapped in " "REMINDER: wrap every factual claim in double quotes (\"...\") "
"double quotes (\"...\") and the quoted span must appear " "and the quoted span must appear word-for-word in the "
"word-for-word in the document. No quote, no claim. " "document. Each claim earns a verbatim quote. "
"Now answer the question on the next message." "Now answer the question on the next message."
), ),
"temperature": 0.1, "temperature": 0.1,
@ -159,22 +161,26 @@ DEFAULT_POLICY = {
"exists.\n" "exists.\n"
"6. Write each claim as one plain-prose sentence on its own " "6. Write each claim as one plain-prose sentence on its own "
"line.\n" "line.\n"
"7. Your answer goes only in the ANSWER position. Never " "7. Each line begins with the claim text and ends with a "
"begin a line with the word EVIDENCE or with a pointer-id " "bracket tag of pointer ids. The headers above the QUESTION "
"prefix like E1: or E2: — those tags belong only above the " "(`EVIDENCE:`, `=== E1 (Title) ===`) are scaffolding for "
"QUESTION, never in your output.\n" "the input only — your answer is the claim lines themselves.\n"
"8. Each claim line cites at most two pointers. Lines with " "8. Each claim line cites at most two pointers. Lines with "
"three or more pointers are rejected. If more than two " "three or more pointers are rejected. When more than two "
"evidence blocks support a claim, pick the two that most " "evidence blocks support a claim, pick the two that most "
"directly contain the claim's key terms; or split the " "directly contain the claim's key terms, or split the "
"claim into two lines." "claim into two lines.\n"
"9. Each claim is one focused fact. Multiple distinct facts "
"become multiple claim lines, each citing the source that "
"directly contains it. Three short claims, each citing one "
"pointer, beat one long claim citing five."
), ),
"claim_lattice_grounding_reminder": ( "claim_lattice_grounding_reminder": (
"REMINDER: format = pointer-line — `Claim text. [E1]` per " "REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from " "line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. At most two pointers per claim. " "the EVIDENCE blocks above. At most two pointers per claim. "
"No `EVIDENCE:` header in your answer, no `E#:` line " "Begin each line with the claim text. Now answer the question "
"prefix. Now answer the question on the next message." "on the next message."
), ),
# Allowed source roles for claim-lattice verification. Roles outside # Allowed source roles for claim-lattice verification. Roles outside
# this set get classified SOURCE_ROLE_BLOCKED and downgrade the # this set get classified SOURCE_ROLE_BLOCKED and downgrade the
@ -231,25 +237,39 @@ DEFAULT_POLICY = {
"claim_lattice_json_system_prompt": ( "claim_lattice_json_system_prompt": (
"You will see numbered EVIDENCE blocks tagged with short " "You will see numbered EVIDENCE blocks tagged with short "
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON " "pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
"object using EXACTLY this schema, with NO prose, NO " "object using EXACTLY this schema. Your output begins with "
"markdown fences, NO preamble:\n\n" "`{` and ends with `}`:\n\n"
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n' ' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
"RULES:\n" "RULES:\n"
"1. Output a single JSON object. No code fences. No commentary.\n" "1. Your output is exactly one JSON object — first character "
"2. `text` is plain prose with NO double-quote characters.\n" "is `{`, last character is `}`.\n"
"3. `evidence_ids` MUST be pointer IDs that appear verbatim in " "2. `text` is plain prose. Punctuation (including any quoted "
"the EVIDENCE block headers above (E1, E2, …). Do NOT invent " "names from the source) appears in the source span the "
"new IDs. At most two IDs per claim.\n" "runtime renders for you, so write the text using only "
"4. Each claim must reference at least one evidence_id.\n" "letters, digits, spaces, and basic sentence punctuation "
"5. If no evidence supports a claim, omit the claim." "(`,`, `.`, `;`, `:`, `?`, `!`, `(`, `)`, `-`).\n"
"3. `evidence_ids` are pointer IDs that appear verbatim in "
"the EVIDENCE block headers above (E1, E2, …). Use only IDs "
"you can see in the headers above. At most two IDs per claim.\n"
"4. Each claim references at least one evidence_id.\n"
"5. Emit a claim only when an EVIDENCE block directly "
"supports it; stop emitting claims when supporting evidence "
"runs out — a short answer is the right answer when only "
"short evidence exists.\n"
"6. Each claim is one focused fact citing the source that "
"directly contains it. Multiple distinct facts become "
"multiple claim entries, each citing the source for that "
"fact. Three short claims, each citing one pointer, beat "
"one long claim citing five."
), ),
"claim_lattice_json_grounding_reminder": ( "claim_lattice_json_grounding_reminder": (
"REMINDER: emit a single JSON object with the exact schema " "REMINDER: emit one JSON object — first character `{`, last "
"character `}` — with the schema "
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. ' '`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
"No code fences, no preamble. evidence_ids must be pointer " "evidence_ids are pointer IDs (E1, E2, …) that appear "
"IDs (E1, E2, …) from the EVIDENCE blocks above — do not " "verbatim in the EVIDENCE block headers above. At most two "
"invent IDs. At most two per claim. Text has no double-quote " "per claim. Each claim is one focused fact. Now answer the "
"characters. Now answer the question on the next message." "question on the next message."
), ),
"claim_lattice_use_guided_json": True, "claim_lattice_use_guided_json": True,
# JSON-mode stop sequences. Hermes-3-8B sometimes spams whitespace # JSON-mode stop sequences. Hermes-3-8B sometimes spams whitespace

10
bench/qa_questions.txt
View file

@ -16,6 +16,7 @@ a bridge between new london & groton?
tell me about connecticut tell me about connecticut
tell me about the C programming language tell me about the C programming language
tell me about method man? tell me about method man?
tell me all there is to know about york england?
# entity list — invites lazy-anchor on a magnet chunk # entity list — invites lazy-anchor on a magnet chunk
what dinosaurs were in the first jurassic park film? what dinosaurs were in the first jurassic park film?
@ -37,8 +38,13 @@ how does intel compare to amd?
what is the boltzmann constant? what is the boltzmann constant?
who invented the doppler effect? who invented the doppler effect?
# leading / adversarial — model should refuse over-claim # leading / forensic — probe whether retrieval surfaces the right
isn't it true that the great wall of china is visible from space? # fact when the question's premise contradicts the popular narrative.
# Elevation question expected answer: north (per geographic surveys);
# corpus grounding may be shaky — coverage check can pass on
# topical-token overlap without the cited span containing the
# elevation fact, so this also stress-tests lazy-anchor STRICT.
which side is the ground elevation highest throughout the span of the great wall of china, the north or south?
whales are endangered from over hunting whales are endangered from over hunting
# out-of-corpus — should land UNGROUNDED honestly # out-of-corpus — should land UNGROUNDED honestly

36
tests/test_qa_quality_live.py
View file

@ -267,6 +267,42 @@ def test_homer_simpson_boss_is_mr_burns():
assert "burns" in txt, f"answer missing 'Burns': {txt[:300]}" assert "burns" in txt, f"answer missing 'Burns': {txt[:300]}"
# ---------------------------------------------------------------- forensic / leading
def test_great_wall_elevation_north_side_higher():
"""Forensic-shape question: which side of the Great Wall has higher
ground elevation? Expected answer: north (per geographic surveys).
The corpus may not directly contain the elevation fact, in which
case Hermes-3-8B can lazy-anchor a "north" claim onto any
Great-Wall span by token coincidence. The system today returns
STRICT 1/1 with the right answer ("northern side higher") cited
to a chunk about "Outer China beyond the Great Wall" — coverage
threshold passes via topical-token overlap (north / Great Wall /
China) without the cited span actually containing elevation
information.
Until the verifier grows a semantic-frame check, this fixture
gates on entity-presence only: the answer must mention "north"
(or "northern" / "northward"). STRICT or HYBRID is fine; the
model sometimes deflects to other Great-Wall facts.
"""
r = _ask(
"which side is the ground elevation highest throughout the "
"span of the great wall of china, the north or south?"
)
txt = _answer_lower(r)
# Either grounded with a directional answer, OR honestly UNGROUNDED.
# The fail mode is: grounded WITHOUT mentioning a direction.
if r.get("audit_mode") == "UNGROUNDED":
return
direction_markers = ("north", "northern", "northward")
assert any(m in txt for m in direction_markers), (
f"audit={r.get('audit_mode')} but answer doesn't name a "
f"direction: {txt[:300]}"
)
# ---------------------------------------------------------------- honest refusal # ---------------------------------------------------------------- honest refusal