engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa: positive-form prompts + source-role rank boost + smell-line floor

Browse source 
Three fixes that meaningfully improve claim_lattice_pointer behavior
on real queries.

1. Prompts in positive form. Hermes-3-8B (and instruction-tuned 8Bs
   in general) follow positive directives ("do X") much more
   reliably than negations ("don't do Y"). Rewrote the
   claim_lattice_system_prompt and claim_lattice_grounding_reminder
   in DEFAULT_POLICY and DEFAULT_QUERY_POLICY so every rule says
   what TO do — "Reference evidence by pointer ID", "Use only
   pointer IDs that appear in the EVIDENCE blocks", "Cite 1 or 2
   pointers per claim", "Stop when the evidence runs out — a short
   answer is the right answer when only short evidence exists",
   etc. The cite-count constraint (1-2 per claim) directly
   addresses the spray-anchor failure where Hermes attached every
   available pointer to one claim line ("where is florida"
   produced 1 claim with 33 cites pre-fix).

2. Source-role rank boost. _rerank() now ends with a new
   _rerank_by_source_role stage that classifies each hit's role
   (mutating h.source_role for downstream reuse) and rescales the
   score by SOURCE_ROLE_RANK_WEIGHTS — primary 2.0, secondary 0.7,
   noisy/sequel 0.3. The Florida defect: list-pages
   (List_of_State_Roads_in_Florida, List_of_places_in_Florida:S/C/B,
   Florida_locations_by_per_capita_income) all match
   _SECONDARY_TITLE_MARKERS for "list of" and now sort below the
   actual Florida article instead of dominating the top-8 by body
   density. Affects ranking for both quote and pointer modes; the
   earlier per-source budget weights stay (still primary gets 2x
   the cap), they just don't have to fight a sort order that put
   list-pages first.

3. Lazy-anchor smell line gates on n_verified >= 3. With one
   verified pair the ratio is trivially 1.00 ("1 of 1 verified
   pairs cite [E11]") which is vacuous. Below 3 pairs the metric
   has no comparison surface; suppress the warning in those cases.

Live results on the 'where is florida' query went 33/33 with
cite-spray to 1/1 with a single targeted citation; spotlight
now hits "...largest metropolitan area in the state as well as
the entire southeastern United States is the South Florida..."
instead of state-road-number tables. The JP-dinosaurs benchmark
went from 11/11 with all cites at [E1] (ratio 1.00) to 13/14
with cites distributed across [E8], [E9], [E13] (ratio 0.77),
spotlight finding film-specific spans like "the film's
Dilophosaurus stands about 1.2 meters (4 ft) tall."

442 tests still passing. All 7 production shards report 0 chain
breaks.
This commit is contained in:
russell@unturf.com 2026-04-29 21:42:35 -04:00
parent 37ac523bba
commit 8fb5148e8e
No known key found for this signature in database
3 changed files with 100 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

11
aborist/cli.py
View file

@ -469,20 +469,21 @@ def _render_query_human(result: dict, question: str) -> str:
# Anchor-smell sidecar (claim_lattice mode only). Soft signal —
# never persisted, never in cache_key. Surfaces when ≥50% of
# verified claim-pointer pairs share one pointer_id, which is
# the lazy-anchor failure mode the spotlight render also exposes.
# verified claim-pointer pairs share one pointer_id AND there
# are at least 3 verified pairs to compare; below 3 the ratio is
# vacuous (1/1 always = 1.00 even when nothing is wrong).
ratio = result.get("lazy_anchor_ratio")
distribution = result.get("pointer_id_distribution") or {}
total_pairs = sum(distribution.values()) if distribution else 0
if (
method == "claim_lattice"
and isinstance(ratio, (int, float))
and ratio >= 0.5
and distribution
and total_pairs >= 3
):
top_pid, top_count = max(distribution.items(), key=lambda kv: kv[1])
total = sum(distribution.values())
lines.append(
f"lazy-anchor smell: {top_count} of {total} verified "
f"lazy-anchor smell: {top_count} of {total_pairs} verified "
f"pairs cite [{top_pid}] (ratio {ratio:.2f}); "
f"distinct pointers cited: {len(distribution)}"
)

101
aborist/qa/query.py
View file

@ -294,9 +294,9 @@ DEFAULT_QUERY_POLICY = {
"answer_mode": DEFAULT_ANSWER_MODE,
"claim_lattice_system_prompt": (
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
"Answer the question using natural-language pointer-lines: one "
"claim per line, each followed by [E#] or [E#,E#,...] "
"referencing the EVIDENCE blocks above.\n\n"
"Answer using natural-language pointer-lines: one claim per "
"line, followed by a bracket tag with the pointer IDs that "
"directly support that claim.\n\n"
"WORKED EXAMPLE\n"
"--------------\n"
"EVIDENCE:\n\n"
@ -312,24 +312,27 @@ DEFAULT_QUERY_POLICY = {
"Steve Wozniak co-founded Apple. [E1,E2]\n"
"Ronald Wayne co-founded Apple. [E1]\n\n"
"END OF EXAMPLE\n\n"
"RULES:\n"
"1. Do not type quotation marks anywhere. Do not quote source "
"text manually. The runtime will display literal evidence "
"spans next to each claim — your job is to reference evidence "
"by pointer id, not to reproduce its text.\n"
"2. Pointer ids must be from the EVIDENCE blocks. Do not "
"invent ids.\n"
"3. Every claim line must end with [E#] or [E#,E#,...].\n"
"4. If no evidence supports a claim, omit that claim. If no "
"evidence supports any claim, output nothing.\n"
"5. No JSON. No markdown tables. No headings. No preamble. "
"Plain prose lines only, each ending with a bracket tag."
"RULES (each rule says what TO do):\n"
"1. Reference evidence by pointer ID. The runtime displays "
"the literal source span beside each claim — referencing is "
"your job; quoting is the runtime's job.\n"
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
"above.\n"
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
"directly contains the claim's key terms.\n"
"4. End every claim line with [E#] or [E#,E#].\n"
"5. Make a claim only when an EVIDENCE block textually "
"supports it. Stop when the evidence runs out — a short "
"answer is the right answer when only short evidence "
"exists.\n"
"6. Write each claim as one plain-prose sentence on its own "
"line."
),
"claim_lattice_grounding_reminder": (
"REMINDER: pointer-line format only — `Claim. [E1]` per line. "
"No quotation marks anywhere. No JSON. Pointer ids must be "
"from the EVIDENCE blocks. Now answer the question on the next "
"message."
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. Cite 1 or 2 pointers per claim. "
"Now answer the question on the next message."
),
"claim_lattice_allowed_source_roles": [
"primary_answer_source",
@ -723,12 +726,18 @@ def _rerank(
core_match_roots: set[str] | None = None,
body_density_check: callable | None = None,
) -> list[_Hit]:
"""Filter off-topic, then layer in body-coverage and title-overlap boosts.
"""Filter off-topic, then layer in body-coverage, title-overlap, and
source-role rank boosts.
Order matters: filter first (drops noise), body-coverage rerank next
(counters BM25's short-doc bias by rewarding topical density across
distinct query tokens), title-overlap rerank last (breaks ties when
a doc IS the named topic).
distinct query tokens), title-overlap rerank (breaks ties when a doc
IS the named topic), source-role rank-boost last so a primary
answer source outranks a list-page even when the list-page won on
BM25 + title-overlap (caught the "where is florida" defect:
``List_of_places_in_Florida`` and ``List_of_State_Roads_in_Florida``
each contain "Florida" hundreds of times in row markup, scoring
above the actual ``Florida`` article on body density).
"""
hits = _filter_by_title_relevance(
hits,
@ -737,7 +746,43 @@ def _rerank(
body_density_check=body_density_check,
)
hits = _rerank_by_body_coverage(hits, question)
return _rerank_by_title(hits, question)
hits = _rerank_by_title(hits, question)
return _rerank_by_source_role(hits, question)
# Per-role rank multiplier. Affects sort order, NOT just per-source
# context cap (the latter is SOURCE_ROLE_BUDGET_WEIGHTS, applied later).
# Defaults skew strongly toward primary so a real topic article beats
# list-pages and franchise/sequel siblings even when the list-page wins
# on body-density. Tuned against the JP-dinosaurs and "where is florida"
# defects.
SOURCE_ROLE_RANK_WEIGHTS = {
"primary_answer_source": 2.0,
"secondary_context_source": 0.7,
"background_source": 0.9,
"noisy_background_source": 0.3,
"sequel_background_source": 0.3,
"unclassified": 1.0,
}
def _rerank_by_source_role(hits: list[_Hit], question: str) -> list[_Hit]:
"""Classify each hit by source role and rescale score by role weight.
Mutates ``h.source_role`` so the classification happens once and
downstream context-build code can reuse the value (instead of
re-classifying at cap time). Stable sort by score desc.
"""
qtokens_stem = {
_stem_token_for_match(t.lower())
for t in _title_query_tokens(question)
}
for h in hits:
h.source_role = _classify_source_role(h.title, qtokens_stem)
weight = SOURCE_ROLE_RANK_WEIGHTS.get(h.source_role, 1.0)
h.score = h.score * weight
hits.sort(key=lambda h: -h.score)
return hits
def _rerank_by_body_coverage(
@ -952,16 +997,14 @@ def query(
context_parts: list[str] = []
per_source_cap = max(1, max_context_chars // max(1, top_k))
char_budget = max_context_chars
qtokens_stem = {_stem_token_for_match(t) for t in _title_query_tokens(question)}
for h in hits[:top_k]:
text = _load_doc_text(h.shard_path, h.document_root)
if not text:
continue
# Tag by role & apply role-weighted cap. Primary answer source
# gets 2× the baseline cap, noisy/sequel get 0.5×, secondary &
# background get 1×. Total context still bounded by
# `char_budget` (running cap to `max_context_chars`).
h.source_role = _classify_source_role(h.title, qtokens_stem)
# source_role is already set by _rerank_by_source_role; reuse
# it for the per-source cap. Primary answer source gets 2× the
# baseline cap, noisy/sequel get 0.5×, secondary & background
# get 1×. Total context still bounded by `char_budget`.
weight = SOURCE_ROLE_BUDGET_WEIGHTS.get(h.source_role, 1.0)
hit_cap = max(1, int(per_source_cap * weight))
if len(text) > hit_cap:

41
aborist/qa/runner.py
View file

@ -108,9 +108,9 @@ DEFAULT_POLICY = {
"answer_mode": DEFAULT_ANSWER_MODE,
"claim_lattice_system_prompt": (
"You will see numbered EVIDENCE blocks tagged E1, E2, E3, etc. "
"Answer the question using natural-language pointer-lines: one "
"claim per line, each followed by [E#] or [E#,E#,...] "
"referencing the EVIDENCE blocks above.\n\n"
"Answer using natural-language pointer-lines: one claim per "
"line, followed by a bracket tag with the pointer IDs that "
"directly support that claim.\n\n"
"WORKED EXAMPLE\n"
"--------------\n"
"EVIDENCE:\n\n"
@ -126,24 +126,27 @@ DEFAULT_POLICY = {
"Steve Wozniak co-founded Apple. [E1,E2]\n"
"Ronald Wayne co-founded Apple. [E1]\n\n"
"END OF EXAMPLE\n\n"
"RULES:\n"
"1. Do not type quotation marks anywhere. Do not quote source "
"text manually. The runtime will display literal evidence "
"spans next to each claim — your job is to reference evidence "
"by pointer id, not to reproduce its text.\n"
"2. Pointer ids must be from the EVIDENCE blocks. Do not "
"invent ids.\n"
"3. Every claim line must end with [E#] or [E#,E#,...].\n"
"4. If no evidence supports a claim, omit that claim. If no "
"evidence supports any claim, output nothing.\n"
"5. No JSON. No markdown tables. No headings. No preamble. "
"Plain prose lines only, each ending with a bracket tag."
"RULES (each rule says what TO do):\n"
"1. Reference evidence by pointer ID. The runtime displays "
"the literal source span beside each claim — referencing is "
"your job; quoting is the runtime's job.\n"
"2. Use only pointer IDs that appear in the EVIDENCE blocks "
"above.\n"
"3. Cite 1 or 2 pointers per claim — the blocks whose text "
"directly contains the claim's key terms.\n"
"4. End every claim line with [E#] or [E#,E#].\n"
"5. Make a claim only when an EVIDENCE block textually "
"supports it. Stop when the evidence runs out — a short "
"answer is the right answer when only short evidence "
"exists.\n"
"6. Write each claim as one plain-prose sentence on its own "
"line."
),
"claim_lattice_grounding_reminder": (
"REMINDER: pointer-line format only — `Claim. [E1]` per line. "
"No quotation marks anywhere. No JSON. Pointer ids must be "
"from the EVIDENCE blocks. Now answer the question on the next "
"message."
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
"line, plain prose with bracket tags. Pointer IDs come from "
"the EVIDENCE blocks above. Cite 1 or 2 pointers per claim. "
"Now answer the question on the next message."
),
# Allowed source roles for claim-lattice verification. Roles outside
# this set get classified SOURCE_ROLE_BLOCKED and downgrade the