arborist/tests
russell@unturf.com 5990834ce8
arch: layer-cake docs + verifier_policy_hash + old-maps principle
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).

NEW DOCS
--------
docs/cti-architecture.md
  Maps today's modules onto the architectural layer cake fox named
  in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
  reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
  (evidence direction) / v9.8 Providence (admissibility ledger) /
  Hermes (weak proposer). Per-layer responsibility table + module
  map showing which existing files implement which layer. The
  architecture is real; the labels above name what's already there.

docs/naming-deferral.md
  Argues why we keep internal terms (claim_lattice,
  claim_lattice_pointer, verify_claim_lattice) instead of renaming
  to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
  cache invalidation, ~150 test fixture references, schema CHECK
  migration, blame-history disconnect, mesh peer coordination.
  The bridge is the cti-architecture mapping doc — read it once,
  then read code in code's vocabulary and prose in prose's
  vocabulary. Lists four triggers that would invert the decision.

docs/self-reference-distillation-design.md
  Successor to docs/self-reference-thought-chains-design.md (the
  flat-source MVP). Maps STRICT claims onto the existing Distiller
  / Core / derivations infrastructure: each STRICT record becomes
  a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
  Merkle inclusion proofs back to Wikipedia source_roots. The
  fact-graph grows; new claims attach by inheriting the proof
  chain. CompositionDistiller (combining facts into new claims)
  is explicitly deferred — that's reasoning machinery, not
  infrastructure.

docs/test-coverage-audit-2026-05-01.md
  Maps fox's §11 test list (16 items) against the codebase. 16/16
  covered. Documents item #9's manual-quote-rule lifecycle: the
  rule was retired from pointer mode in commit 224bfd6 after the
  bench journey identified it was rejecting factually correct
  claims; retained in JSON variant where the punctuation-rationale
  argument doesn't apply. The audit doc itself is the requirements-
  drift defense.

CODE
----
aborist/qa/keys.py:verifier_policy_hash
  New pure function. Hashes the verifier-relevant subset of policy
  (answer_mode + claim_lattice_* verifier knobs + entity_policy
  fields + base_version). Folded into cache_key as an optional
  9th dimension via a new keyword arg with backward-compat default
  None — legacy 8-dim callers continue to work and produce the
  pre-2026-05-01 hash result.

aborist/qa/{runner,query}.py
  Compute verifier_policy_hash from the policy_variant and pass
  to cache_key as the 9th dim. Records written from this commit
  forward bind to the verifier-policy identity. Pre-existing 8-dim
  records become cache-misses on next lookup — same migration cost
  as any governance_policy_hash bump. The 9-dim form makes the
  question "did the verifier rules change?" answerable from
  cache_key diff alone, separate from "did the prompt change?"

CLAUDE.md
  Adds "old maps vs runtime maps" engineering discipline to the
  bench-maxing section. Codifies the principle: every base model
  carries old maps from training time; the runtime carries the
  fresh map; when they disagree, the runtime wins. Move authority
  OUT of the model's prior and INTO runtime artifacts (pointer IDs
  the runtime mints, source spans the runtime interpolates by
  offset, evidence maps assembled per query, policy hashes that
  fold prompt/verifier/retrieval into cache_key, hard checks run
  by the verifier). Hermes' content-addressed-evidence-id
  hallucination (commit bb8450d) is the canonical case study.

TESTS
-----
tests/test_keys.py
  Six new tests for verifier_policy_hash:
    - only hashes verifier subset (non-verifier fields don't change it)
    - changes when verifier-relevant field changes
    - empty-policy → stable
    - 9-dim cache_key distinct from 8-dim
    - 9-dim distinct under different verifier hashes
    - 8-dim form preserved for legacy callers (None == omit)

DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
  logic in runner.py + query.py already IS PROMETHEUS-Σ; an
  explicit prometheus.py is a refactor with no behavior change.
  Defer until a §5-rename-trigger fires.

507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
2026-05-01 12:10:29 -04:00
..
crawler crawler: skip feeds + sitemaps at ingest (they're discovery, not knowledge) 2026-04-29 05:55:28 -04:00
fixtures verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_bench_qa_sweep.py qa/tests: cover capacity metrics + retrieval-keywords across all 3 layers 2026-05-01 11:51:30 -04:00
test_burn.py cli: 'aborist burn-kindergarten' — mass-burn fresh providence rows 2026-04-29 16:56:59 -04:00
test_burn_doc.py cli: extend 'aborist burn' to documents and cores 2026-04-28 17:32:40 -04:00
test_claim_lattice.py qa: claim-count ceiling — TOO_MANY_CLAIMS demote (default 12/answer) 2026-04-30 19:46:52 -04:00
test_cli_render.py qa/tests: cover capacity metrics + retrieval-keywords across all 3 layers 2026-05-01 11:51:30 -04:00
test_compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
test_concepts.py concept overlay: synonym expansion + rivalry exclusion 2026-04-27 12:34:00 -04:00
test_dag.py qa: chain-segment failure localization + re-prompt repair tier 2026-04-29 19:27:33 -04:00
test_distill.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill_recursive.py recursive distillation: core -> depth+1 core 2026-04-27 07:54:30 -04:00
test_evict.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_grok_source.py add Grok export source: conversations + media posts 2026-04-27 13:49:42 -04:00
test_html_source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_ingest.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
test_inspect.py qa/inspect: title-relevance sidecar — flag retrieval-driven hallucinations 2026-05-01 10:19:44 -04:00
test_keys.py arch: layer-cake docs + verifier_policy_hash + old-maps principle 2026-05-01 12:10:29 -04:00
test_llm_context_base.py qa: pipe wikitext-base-v1 into LLM context (was verifier-only) 2026-04-28 17:08:49 -04:00
test_merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_mesh.py mesh: cryptographic foundation, off by default 2026-04-27 19:00:24 -04:00
test_mesh_aead.py mesh: optional AEAD body encryption under per-epoch shared secret 2026-04-28 17:34:39 -04:00
test_mesh_chain.py mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
test_mesh_cli.py mesh: sync default-holds records younger than 1 hour (kindergarten window) 2026-04-29 16:07:24 -04:00
test_mesh_cli_pull.py mesh: 'pull' CLI verb — fetch document body from a peer 2026-04-28 17:33:27 -04:00
test_mesh_wire.py mesh: HTTP gossip wire — signed envelopes + 5 message types 2026-04-28 16:57:04 -04:00
test_mesh_wire_e2e.py mesh: HTTP gossip wire — signed envelopes + 5 message types 2026-04-28 16:57:04 -04:00
test_migration_audit_mode.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
test_providence_source.py qa: self-reference thought chains — STRICT-as-fact substrate 2026-05-01 10:16:47 -04:00
test_qa.py qa: post-LLM faithfulness verifier sets STRICT/HYBRID/VISUAL audit_mode 2026-04-28 15:45:08 -04:00
test_qa_quality_live.py qa/live: lock 3 bench standouts as STRICT-rate regression gates 2026-04-30 19:47:07 -04:00
test_query.py qa/tests: cover capacity metrics + retrieval-keywords across all 3 layers 2026-05-01 11:51:30 -04:00
test_reclassify.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
test_repair.py qa: chain-segment failure localization + re-prompt repair tier 2026-04-29 19:27:33 -04:00
test_resume.py resumable ingest + per-shard audit chain integrity 2026-04-27 11:29:27 -04:00
test_snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
test_tfidf.py add TF-IDF keyword distiller 2026-04-27 07:55:53 -04:00
test_vcs_source.py add git and mercurial repo sources for self-play 2026-04-27 18:17:39 -04:00
test_verify.py qa: per-claim status taxonomy on verifier + repair-action plans on sidecar 2026-04-29 18:37:52 -04:00
test_verify_json.py qa: claim-count ceiling — TOO_MANY_CLAIMS demote (default 12/answer) 2026-04-30 19:46:52 -04:00
test_versioned_ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
test_wikipedia_old.py prepare full Wikipedia 2003-05-16 ingest: cur + old (revisions) 2026-04-27 08:10:42 -04:00
test_wikipedia_xml.py add Phase IV Wikipedia XML + abstract sources 2026-04-27 17:24:28 -04:00
test_wikitext.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00