qa: per-mode max_context_chars defaults from 2026-05-02 bench

Sprint 1b — replace the flat max_context_chars=60000 default with
per-mode budgets matching the peak bucket each mode achieved on
the 2026-05-02 bench:

  quote                  → 24000 (peak 16-32KB)
  claim_lattice_pointer  → 24000 (peak 16-32KB)
  claim_lattice (JSON)   → 48000 (peak 32-64KB)

The flat 60K was past peak for quote/pointer (which degrade after
32 KB) and only marginally above peak for JSON. The bench's
recommended-context-budget table now flows back into the policy
defaults — operator-driven landing per the five-step algorithm
step 5: surface, don't auto-apply. The mapping lives in
DEFAULT_QUERY_POLICY so any tuning bumps governance_policy_hash
and partitions the cache namespace cleanly.

API boundary stays backward-compatible: an explicit
max_context_chars= caller value still wins. The per-mode default
fires only when the caller passes None (or omits the arg).

CLI: --max-context-chars default flips to None; help text spells
out the per-mode fallback.

Smoke (5-question fixture) confirmed no regressions vs the
pre-change baseline:
  quote   2/5 → 5/5  (+3)
  pointer 0/5 → 1/5  (+1)
  JSON    5/5 → 5/5  (flat — already at ceiling on this fixture)

Quote's +3 is the unexpected win: the smaller 24K budget surfaces
tighter-relevance top-K instead of drowning the model in filler.
The full bench (~50 min) is the real scoreboard but smoke shape
matches the bench's predicted lift direction.
This commit is contained in:
russell@unturf.com 2026-05-02 09:39:40 -04:00
parent 3b9122395c
commit e582b2e93a
No known key found for this signature in database
2 changed files with 39 additions and 3 deletions

View file

@ -3196,8 +3196,13 @@ def build_parser() -> argparse.ArgumentParser:
help="FTS5 hits to fetch per shard before dedup (default 32)",
)
query_cmd.add_argument(
"--max-context-chars", dest="max_context_chars", type=int, default=60000,
help="cap on assembled context bytes (default 60000)",
"--max-context-chars", dest="max_context_chars", type=int, default=None,
help=(
"cap on assembled context bytes. When omitted, falls back to "
"the per-mode default in DEFAULT_QUERY_POLICY['max_context_chars_by_mode'] "
"(quote=24000, claim_lattice_pointer=24000, claim_lattice=48000). "
"Sprint 1b 2026-05-02 — peaks measured per mode."
),
)
query_cmd.add_argument(
"--question-dedup", dest="question_dedup", default=None,

View file

@ -491,6 +491,28 @@ DEFAULT_QUERY_POLICY = {
),
"claim_lattice_use_guided_json": True,
"claim_lattice_json_stop_sequences": ["\n\n"],
# Per-mode context-budget defaults. Sprint 1b (2026-05-02) bench
# measured each answer mode's peak strict-rate bucket on a sweep
# over 8 KB → 1 MB context budgets. Previous flat default of
# 60 KB was past peak for both quote and pointer modes (which
# degrade after 32 KB) and only marginally above peak for the
# JSON variant (which keeps climbing into the 32-64 KB bucket).
# Defaults below are the mid of each mode's peak bucket rounded
# to nice numbers:
# quote → 16-32KB peak → 24000
# claim_lattice_pointer → 16-32KB peak → 24000
# claim_lattice (JSON) → 32-64KB peak → 48000
# Selected by `query()` when the caller does not pass an explicit
# `max_context_chars`. Any explicit caller value still wins
# (backward-compatible at the API boundary). The mapping itself
# lives in policy so changes fold into governance_policy_hash and
# partition the cache namespace cleanly. See
# docs/qa-modes-bench-2026-05-02.md "recommended context budget".
"max_context_chars_by_mode": {
"quote": 24000,
"claim_lattice_pointer": 24000,
"claim_lattice": 48000,
},
}
@ -1529,7 +1551,7 @@ def query(
single_db: Path | None = None,
top_k: int = 8,
over_fetch: int = 32,
max_context_chars: int = 60000,
max_context_chars: int | None = None,
policy: dict | None = None,
chain: str = "private",
fidelity: str | None = None,
@ -1588,6 +1610,15 @@ def query(
raise ValueError(
f"policy['answer_mode'] must be one of {ANSWER_MODES}, got {answer_mode!r}"
)
# Resolve per-mode context budget when the caller didn't pass one
# explicitly. Sprint 1b (2026-05-02) — different answer modes peak
# at different budgets; quote/pointer at 24 KB, JSON at 48 KB.
# Explicit caller value always wins (backward-compatible).
if max_context_chars is None:
by_mode = policy.get("max_context_chars_by_mode") or {}
max_context_chars = int(
by_mode.get(answer_mode, policy.get("max_context_chars", 60000))
)
t_start = time.monotonic()
# 1. Search.