qa/corpus_query: wikitext-strip when policy.base_version is set (6d)
Closes Phase 1 step 6 of #53. When the caller passes policy={"base_version": "wikitext-base-v1"} (or any truthy value), run_query now invokes arborist.wikitext.to_base() on each chunk span BEFORE truncating to per_doc_budget, so a 24 kB raw-template span gets compressed first and the LLM/verifier see the same prose. Same shape legacy query() applies at three sites. policy=None / policy without base_version: no-op, preserves the byte-identity gate. mwparserfromhell missing (the optional dep): import fails, no-op — already the legacy convention. policy["base_version"] folds into governance_policy_hash via the caller's policy dict — opting in DOES rotate cache_key for that policy partition, which is the right semantics (different prose → different cache). 264 tests still pass. Phase 1 substantively complete; Phase 2 (providence_query.py + legacy query() collapse) starts next.
This commit is contained in:
parent
6c2ec1b6c0
commit
b8bd9d6ddb
1 changed files with 15 additions and 0 deletions
|
|
@ -303,6 +303,19 @@ def run_query(
|
|||
per_hit_budgets = [
|
||||
max(1000, max_context_chars // max(1, len(hits)))
|
||||
] * len(hits)
|
||||
# Wikitext-strip (step 6d): when policy["base_version"] is set
|
||||
# AND the wikitext module imports, run to_base() on each span
|
||||
# BEFORE truncating to per_doc_budget so a 24kB raw-template span
|
||||
# gets compressed first. Same shape legacy query() uses at three
|
||||
# sites; folds into governance_policy_hash via policy["base_version"]
|
||||
# — caller's responsibility to pin the version field if they want
|
||||
# cache-key partitioning across the strip.
|
||||
wikitext_strip_fn = None
|
||||
if policy and policy.get("base_version"):
|
||||
try:
|
||||
from arborist.wikitext import to_base as wikitext_strip_fn
|
||||
except ImportError:
|
||||
wikitext_strip_fn = None
|
||||
chunks_for_evidence: list[dict] = []
|
||||
total_chars = 0
|
||||
for rank, (h, role, per_doc_budget) in enumerate(
|
||||
|
|
@ -313,6 +326,8 @@ def run_query(
|
|||
continue
|
||||
r = rows[0]
|
||||
span = r.content
|
||||
if wikitext_strip_fn is not None:
|
||||
span = wikitext_strip_fn(span)
|
||||
if len(span) > per_doc_budget:
|
||||
span = span[:per_doc_budget]
|
||||
chunks_for_evidence.append({
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue