qa/corpus_query: wikitext-strip when policy.base_version is set (6d)

Closes Phase 1 step 6 of #53. When the caller passes
policy={"base_version": "wikitext-base-v1"} (or any truthy value),
run_query now invokes arborist.wikitext.to_base() on each chunk
span BEFORE truncating to per_doc_budget, so a 24 kB raw-template
span gets compressed first and the LLM/verifier see the same
prose. Same shape legacy query() applies at three sites.

policy=None / policy without base_version: no-op, preserves the
byte-identity gate. mwparserfromhell missing (the optional dep):
import fails, no-op — already the legacy convention.

policy["base_version"] folds into governance_policy_hash via the
caller's policy dict — opting in DOES rotate cache_key for that
policy partition, which is the right semantics (different prose →
different cache).

264 tests still pass. Phase 1 substantively complete; Phase 2
(providence_query.py + legacy query() collapse) starts next.
This commit is contained in:
russell@unturf.com 2026-05-31 12:53:42 -04:00
parent 6c2ec1b6c0
commit b8bd9d6ddb
No known key found for this signature in database

View file

@ -303,6 +303,19 @@ def run_query(
per_hit_budgets = [
max(1000, max_context_chars // max(1, len(hits)))
] * len(hits)
# Wikitext-strip (step 6d): when policy["base_version"] is set
# AND the wikitext module imports, run to_base() on each span
# BEFORE truncating to per_doc_budget so a 24kB raw-template span
# gets compressed first. Same shape legacy query() uses at three
# sites; folds into governance_policy_hash via policy["base_version"]
# — caller's responsibility to pin the version field if they want
# cache-key partitioning across the strip.
wikitext_strip_fn = None
if policy and policy.get("base_version"):
try:
from arborist.wikitext import to_base as wikitext_strip_fn
except ImportError:
wikitext_strip_fn = None
chunks_for_evidence: list[dict] = []
total_chars = 0
for rank, (h, role, per_doc_budget) in enumerate(
@ -313,6 +326,8 @@ def run_query(
continue
r = rows[0]
span = r.content
if wikitext_strip_fn is not None:
span = wikitext_strip_fn(span)
if len(span) > per_doc_budget:
span = span[:per_doc_budget]
chunks_for_evidence.append({