engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

strict attribution rules in query system prompt

Browse source 
Even with rivalry exclusion, sources like 'CPU_monthly' (a roundup
discussing both AMD and Intel CPUs in the same article) sneak through
because their title matches "CPU" without naming a competitor. Hermes
then conflated facts: "the fastest AMD CPU is the 3.2 GHz Pentium 4."

Tightened the DEFAULT_QUERY_POLICY system prompt with four explicit
rules:

1. A single source may discuss multiple competitors. Attribute facts
   only to the entity the source explicitly names.
2. If the question asks about ONE entity (e.g., "fastest AMD CPU"),
   do not include competitor facts as if they were facts about the
   asked entity.
3. AMD PR-rating numbers (3200+, 2500+) are NOT clock speeds in MHz.
   Quote the source's own wording rather than interpreting model
   numbers.
4. Say "I don't know based on the provided sources" when the sources
   don't cover the answer. Don't speculate.

Demo on the same AMD query (post-distill, cores now in retrieval):
  before: "the fastest AMD CPU is the 3.2 GHz Pentium 4 from May 2003"
  after:  "the fastest AMD CPU mentioned is the Athlon XP 3200+ which
           was released in May 2003. The sources do not contain
           information about CPUs released after 2003."

Sources include AMD, AMD_5x86, AMD_Duron + their cores, CPU_design,
CPU_monthly. Hermes correctly attributes Athlon XP 3200+ to AMD and
honestly bounds the corpus's coverage at 2003.

Bumping the system prompt invalidates the governance_policy_hash and
therefore every prior cache_key — by design, per v9.8 admissibility
(prompt change = different governance = fresh inference).

66 tests still passing.
This commit is contained in:
russell@unturf.com 2026-04-27 13:04:31 -04:00
parent 1abe6d4adc
commit 787c243179
No known key found for this signature in database
1 changed files with 17 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

21
aborist/qa/query.py
View file

@ -132,10 +132,23 @@ def _filter_by_title_relevance(hits: list, question: str) -> list:
DEFAULT_QUERY_POLICY = {
"system_prompt": (
"You are answering a question using ONLY the sources provided below. "
"Each source is delimited by '=== Source: <URI> ===' headers. "
"Cite which source supports each claim by quoting it inline. "
"If the sources do not contain the answer, say 'I don't know based on "
"the provided sources.' Do not speculate beyond what is written."
"Each source is delimited by '=== Source: <URI> ===' headers.\n\n"
"ATTRIBUTION RULES:\n"
"1. A single source may discuss MULTIPLE products, companies, or "
"competitors. Read carefully and only attribute facts to the entity "
"the source explicitly names for that fact.\n"
"2. If the question asks about ONE specific entity (e.g., 'fastest "
"AMD CPU'), do NOT include facts about competitors (Intel, Pentium) "
"as if they were facts about the asked entity. They are different "
"products even when discussed in the same article.\n"
"3. When referencing model numbers like 'Athlon XP 3200+', remember "
"that AMD's PR-rating numbers (3200+, 2500+) are NOT the clock speed "
"in MHz. Quote the source's own wording for clock speeds rather than "
"interpreting model numbers.\n"
"4. Cite which source supports each claim by quoting it inline.\n"
"5. If the sources do not contain the answer, say 'I don't know "
"based on the provided sources.' Do not speculate beyond what the "
"sources say."
),
"temperature": 0.1,
"top_p": 1.0,