Phase 1 step 5 of #53 — the load-bearing piece. Freezes the cache-
identity byte shape today so the run_query rewrite landing in Phase 2
can be checked against it.
What's pinned (tests/fixtures/byte_identity/claim_lattice.json):
- SHA-256 of CLAIM_LATTICE_SYSTEM_PROMPT (drift = every cache rotates)
- SHA-256 of CLAIM_LATTICE_GROUNDING_REMINDER (same)
- Per-question question_hash (strict + equivalence_class modes)
- Per-question conversation_hash on the synthetic 2-message array
[system + user(EVIDENCE+QUESTION+grounding_reminder)] — the EXACT
shape arborist/qa/corpus_query.py:run_query builds
- governance_policy_hash on three reference policy shapes
- model_profile_hash for hermes / qwen / stub
Plus a determinism sanity test that pins the algos themselves
(SHA-256, _canonical_json key-sorting, dedup-mode question canonical).
Risk class addressed (Plan §6 risks #1+#2): conversation_hash takes
the FULL OpenAI messages array. Any drift — message reorder, whitespace
shift, optional message gated on a different condition — rotates every
cache_key in the world and orphans every providence_cache record on
re-lookup. Same for governance_policy_hash on the policy dict (a new
field rotates everything). The fixture catches a drift the SECOND it
happens, with a diff-style failure naming the path that drifted.
Re-capture mode: `CAPTURE=1 pytest tests/test_run_query_byte_identity.py`
rewrites the fixture. Only do this on deliberate prompt-shape or
policy-shape changes that are treated as cache-invalidation events.
Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).
Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.
BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).
Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
audit-chain integrity probe, counts dangling prev_event_hash
references; 0 = chain intact
Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
(Cloud Strife quote: VISUAL → STRICT after strip; genuine
hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
shard, used to validate the strip on actual Wikipedia content
