qa: per-mode max_context_chars defaults from 2026-05-02 bench
Sprint 1b — replace the flat max_context_chars=60000 default with per-mode budgets matching the peak bucket each mode achieved on the 2026-05-02 bench: quote → 24000 (peak 16-32KB) claim_lattice_pointer → 24000 (peak 16-32KB) claim_lattice (JSON) → 48000 (peak 32-64KB) The flat 60K was past peak for quote/pointer (which degrade after 32 KB) and only marginally above peak for JSON. The bench's recommended-context-budget table now flows back into the policy defaults — operator-driven landing per the five-step algorithm step 5: surface, don't auto-apply. The mapping lives in DEFAULT_QUERY_POLICY so any tuning bumps governance_policy_hash and partitions the cache namespace cleanly. API boundary stays backward-compatible: an explicit max_context_chars= caller value still wins. The per-mode default fires only when the caller passes None (or omits the arg). CLI: --max-context-chars default flips to None; help text spells out the per-mode fallback. Smoke (5-question fixture) confirmed no regressions vs the pre-change baseline: quote 2/5 → 5/5 (+3) pointer 0/5 → 1/5 (+1) JSON 5/5 → 5/5 (flat — already at ceiling on this fixture) Quote's +3 is the unexpected win: the smaller 24K budget surfaces tighter-relevance top-K instead of drowning the model in filler. The full bench (~50 min) is the real scoreboard but smoke shape matches the bench's predicted lift direction.
This commit is contained in:
parent
3b9122395c
commit
e582b2e93a
2 changed files with 39 additions and 3 deletions
|
|
@ -3196,8 +3196,13 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
help="FTS5 hits to fetch per shard before dedup (default 32)",
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--max-context-chars", dest="max_context_chars", type=int, default=60000,
|
||||
help="cap on assembled context bytes (default 60000)",
|
||||
"--max-context-chars", dest="max_context_chars", type=int, default=None,
|
||||
help=(
|
||||
"cap on assembled context bytes. When omitted, falls back to "
|
||||
"the per-mode default in DEFAULT_QUERY_POLICY['max_context_chars_by_mode'] "
|
||||
"(quote=24000, claim_lattice_pointer=24000, claim_lattice=48000). "
|
||||
"Sprint 1b 2026-05-02 — peaks measured per mode."
|
||||
),
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--question-dedup", dest="question_dedup", default=None,
|
||||
|
|
|
|||
|
|
@ -491,6 +491,28 @@ DEFAULT_QUERY_POLICY = {
|
|||
),
|
||||
"claim_lattice_use_guided_json": True,
|
||||
"claim_lattice_json_stop_sequences": ["\n\n"],
|
||||
# Per-mode context-budget defaults. Sprint 1b (2026-05-02) bench
|
||||
# measured each answer mode's peak strict-rate bucket on a sweep
|
||||
# over 8 KB → 1 MB context budgets. Previous flat default of
|
||||
# 60 KB was past peak for both quote and pointer modes (which
|
||||
# degrade after 32 KB) and only marginally above peak for the
|
||||
# JSON variant (which keeps climbing into the 32-64 KB bucket).
|
||||
# Defaults below are the mid of each mode's peak bucket rounded
|
||||
# to nice numbers:
|
||||
# quote → 16-32KB peak → 24000
|
||||
# claim_lattice_pointer → 16-32KB peak → 24000
|
||||
# claim_lattice (JSON) → 32-64KB peak → 48000
|
||||
# Selected by `query()` when the caller does not pass an explicit
|
||||
# `max_context_chars`. Any explicit caller value still wins
|
||||
# (backward-compatible at the API boundary). The mapping itself
|
||||
# lives in policy so changes fold into governance_policy_hash and
|
||||
# partition the cache namespace cleanly. See
|
||||
# docs/qa-modes-bench-2026-05-02.md "recommended context budget".
|
||||
"max_context_chars_by_mode": {
|
||||
"quote": 24000,
|
||||
"claim_lattice_pointer": 24000,
|
||||
"claim_lattice": 48000,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -1529,7 +1551,7 @@ def query(
|
|||
single_db: Path | None = None,
|
||||
top_k: int = 8,
|
||||
over_fetch: int = 32,
|
||||
max_context_chars: int = 60000,
|
||||
max_context_chars: int | None = None,
|
||||
policy: dict | None = None,
|
||||
chain: str = "private",
|
||||
fidelity: str | None = None,
|
||||
|
|
@ -1588,6 +1610,15 @@ def query(
|
|||
raise ValueError(
|
||||
f"policy['answer_mode'] must be one of {ANSWER_MODES}, got {answer_mode!r}"
|
||||
)
|
||||
# Resolve per-mode context budget when the caller didn't pass one
|
||||
# explicitly. Sprint 1b (2026-05-02) — different answer modes peak
|
||||
# at different budgets; quote/pointer at 24 KB, JSON at 48 KB.
|
||||
# Explicit caller value always wins (backward-compatible).
|
||||
if max_context_chars is None:
|
||||
by_mode = policy.get("max_context_chars_by_mode") or {}
|
||||
max_context_chars = int(
|
||||
by_mode.get(answer_mode, policy.get("max_context_chars", 60000))
|
||||
)
|
||||
t_start = time.monotonic()
|
||||
|
||||
# 1. Search.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue