qa/tests: cover capacity metrics + retrieval-keywords across all 3 layers
Three layers, 14 new tests, full suite 611 passed (was 597):
UNIT — tests/test_query.py
test_query_returns_prompt_chars_breakdown
Asserts the result dict's prompt_chars carries exactly the five
expected keys and messages_total equals sum of message contents
the StubClient saw.
test_query_answer_chars_matches_answer_text
answer_chars == len(answer_text) — drift check.
test_query_cache_hit_also_returns_capacity_metrics
Cache-hit path populates prompt_chars + answer_chars (operators
inspecting cached records still want the breakdown).
test_query_evidence_chars_grows_with_topk
Sanity: more sources / larger budget → more evidence chars
(the metric tracks actual context build, not a stale constant).
INTEGRATION — tests/test_query.py (retrieval_keywords)
test_retrieval_keywords_does_not_alter_question_to_llm
Keywords don't appear in the LLM-facing question segment;
system prompt unchanged across runs. Pins the substrate
contract: keywords are FTS5/title-filter-only.
test_retrieval_keywords_changes_retrieved_sources
Different keyword sets surface different docs (the actual
user-visible behavior).
UNIT — tests/test_cli_render.py
test_render_shows_capacity_line_when_prompt_chars_present
Capacity line appears with messages_total + breakdown when
prompt_chars is in the result dict.
test_render_omits_capacity_line_on_legacy_results_without_prompt_chars
Backwards-compat: legacy results render cleanly without the
capacity line — no KeyError, no '0 chars' noise.
test_render_capacity_thousand_separators
61,550 not 61550 — operator legibility on daily renders.
UNIT/INTEGRATION — tests/test_bench_qa_sweep.py (NEW FILE)
Imports bench/qa_sweep.py via importlib.util so the module's
not in the Python path doesn't matter. Five tests:
- _summarize counts verdicts by mode
- deflections counted only on STRICT/HYBRID rows (not UNGROUNDED)
- rendered markdown has the headline summary + size buckets
- size buckets correctly stratify strict-rate by prompt_chars_total
- empty buckets are skipped (no '0 runs' noise)
FUNCTIONAL — live verification (no automated test, manual)
`make query Q="what is the capital of france?" BURN=1` confirmed
in commit f927298 to render the capacity one-liner under the
source list. Documented in that commit's body.
Also corrected the docstring on query()'s `retrieval_keywords` to
reflect that keywords don't enter cache_key DIRECTLY but do change
context_root + conversation_hash via source selection — so the same
question with different keywords lands under different cache_keys
(legitimately, since the LLM saw different contexts).
This commit is contained in:
parent
f927298353
commit
d9b05c586e
3 changed files with 439 additions and 0 deletions
|
|
@ -146,3 +146,60 @@ def test_json_mode_also_uses_unicode():
|
|||
s = json.dumps(payload, ensure_ascii=False)
|
||||
assert "Pokémon" in s
|
||||
assert "\\u00e9" not in s
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# capacity metrics — human render surfaces prompt_chars breakdown
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_render_shows_capacity_line_when_prompt_chars_present():
|
||||
"""When the result carries `prompt_chars`, the human render
|
||||
emits a one-liner summarizing prompt size + answer size so an
|
||||
operator can tell at a glance whether STRICT came from a small
|
||||
prompt or a context-stuffed one."""
|
||||
r = _result(
|
||||
prompt_chars={
|
||||
"system_prompt": 1292,
|
||||
"grounding_reminder": 342,
|
||||
"user_question": 30,
|
||||
"evidence_or_context": 20304,
|
||||
"messages_total": 21996,
|
||||
},
|
||||
answer_chars=247,
|
||||
)
|
||||
out = _render_query_human(r, "what is the capital of france?")
|
||||
assert "capacity:" in out
|
||||
assert "21,996 chars" in out # messages_total
|
||||
assert "sys 1,292" in out
|
||||
assert "evidence 20,304" in out
|
||||
assert "answer 247 chars" in out
|
||||
|
||||
|
||||
def test_render_omits_capacity_line_on_legacy_results_without_prompt_chars():
|
||||
"""Backwards-compat: results that pre-date capacity metrics (no
|
||||
prompt_chars key) render without the capacity line — no KeyError,
|
||||
no awkward 'capacity: 0 chars' noise."""
|
||||
r = _result() # no prompt_chars
|
||||
out = _render_query_human(r, "q")
|
||||
assert "capacity:" not in out
|
||||
|
||||
|
||||
def test_render_capacity_thousand_separators():
|
||||
"""Large prompts must format with thousand-separator commas so
|
||||
a 60K prompt reads as '60,000' not '60000' — operator legibility
|
||||
on the daily render."""
|
||||
r = _result(
|
||||
prompt_chars={
|
||||
"system_prompt": 1500,
|
||||
"grounding_reminder": 0,
|
||||
"user_question": 50,
|
||||
"evidence_or_context": 60000,
|
||||
"messages_total": 61550,
|
||||
},
|
||||
answer_chars=1234,
|
||||
)
|
||||
out = _render_query_human(r, "q")
|
||||
assert "61,550 chars" in out
|
||||
assert "60,000" in out
|
||||
assert "1,234 chars" in out
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue