arborist/tests
russell@unturf.com aefc032749
qa: dedup-mode policy + JIT fidelity — agents pick fast-cache vs audit-grade
Different agents have different value functions on the same providence
records. Today aborist bakes one canonicalization policy into
governance_policy_hash and calls it universal. This adds two knobs that
let agents express their preference without breaking provenance.

- aborist/qa/keys.py: `canonical_question(q, mode=...)` and
  `question_hash(q, mode=...)` accept "strict" (NFC + ws-collapse only;
  every variant gets its own hash) or "equivalence_class" (default —
  additionally lowercase + trailing-punct strip + article strip).
  New constants: QUESTION_DEDUP_MODES, FIDELITY_MODES, defaults.

- aborist/qa/query.py + aborist/qa/runner.py: each call computes the
  primary cache_key under policy["question_dedup"]. New `fidelity`
  parameter:
    "strict"            only primary cache_key checked
    "equivalence_class" primary first; if miss AND alternate mode
                        produces a different cache_key, try alternate
  Cross-silo fallback works because _ckey_for_mode rewrites
  policy["question_dedup"] to the alternate mode before computing
  governance_policy_hash — so the fallback ckey matches what an agent
  under that mode would have written. Lookups can find each other's
  records when fidelity permits. Result dict gains `lookup_path` ∈
  {"strict", "equivalence_class", "strict_fallback",
  "equivalence_class_fallback", "miss"}.

- aborist/cli.py: `--question-dedup` and `--fidelity` flags on the
  `query` subcommand. Human render annotates cache_hit lines that
  came from a fallback ckey ("cached via equivalence_class_fallback").

- tests/test_query.py: three new regressions
    * strict-policy + strict-fidelity: variants get distinct cache_keys
    * cross-silo fallback: strict-policy reads eq-class-policy records
      via fidelity=equivalence_class
    * strict-fidelity refuses fallback (audit-grade): cache miss even
      when the alternate silo has a hit

- CLAUDE.md: dedup-mode and fidelity convention bullets updated.

This is V1 + V2 of the substrate move sketched in conversation: write
policy determines which silo a record lives in; read fidelity determines
how loosely an agent walks across silos. Records stay exactly-keyed
(provenance hard); routing is per-agent (preference soft). Same
verifier_no_diagnostics discipline holds — no soft signal enters the
hard chain.

442 tests pass.
2026-04-29 16:36:39 -04:00
..
crawler crawler: skip feeds + sitemaps at ingest (they're discovery, not knowledge) 2026-04-29 05:55:28 -04:00
fixtures verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_burn.py cli: 'aborist burn' — kindergarten leaf delete with audit event 2026-04-28 17:03:30 -04:00
test_burn_doc.py cli: extend 'aborist burn' to documents and cores 2026-04-28 17:32:40 -04:00
test_cli_render.py cli: human render for query by default; --json for raw; ensure_ascii=False 2026-04-29 10:14:36 -04:00
test_compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
test_concepts.py concept overlay: synonym expansion + rivalry exclusion 2026-04-27 12:34:00 -04:00
test_distill.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill_recursive.py recursive distillation: core -> depth+1 core 2026-04-27 07:54:30 -04:00
test_evict.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_grok_source.py add Grok export source: conversations + media posts 2026-04-27 13:49:42 -04:00
test_html_source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_ingest.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
test_inspect.py inspect: interior_elision diagnosis — model dropped a (...) aside from source 2026-04-29 16:10:51 -04:00
test_keys.py qa: question_hash strips articles + verifier coverage drops stopwords 2026-04-29 14:00:44 -04:00
test_llm_context_base.py qa: pipe wikitext-base-v1 into LLM context (was verifier-only) 2026-04-28 17:08:49 -04:00
test_merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_mesh.py mesh: cryptographic foundation, off by default 2026-04-27 19:00:24 -04:00
test_mesh_aead.py mesh: optional AEAD body encryption under per-epoch shared secret 2026-04-28 17:34:39 -04:00
test_mesh_chain.py mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
test_mesh_cli.py mesh: sync default-holds records younger than 1 hour (kindergarten window) 2026-04-29 16:07:24 -04:00
test_mesh_cli_pull.py mesh: 'pull' CLI verb — fetch document body from a peer 2026-04-28 17:33:27 -04:00
test_mesh_wire.py mesh: HTTP gossip wire — signed envelopes + 5 message types 2026-04-28 16:57:04 -04:00
test_mesh_wire_e2e.py mesh: HTTP gossip wire — signed envelopes + 5 message types 2026-04-28 16:57:04 -04:00
test_migration_audit_mode.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
test_qa.py qa: post-LLM faithfulness verifier sets STRICT/HYBRID/VISUAL audit_mode 2026-04-28 15:45:08 -04:00
test_query.py qa: dedup-mode policy + JIT fidelity — agents pick fast-cache vs audit-grade 2026-04-29 16:36:39 -04:00
test_reclassify.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
test_resume.py resumable ingest + per-shard audit chain integrity 2026-04-27 11:29:27 -04:00
test_snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
test_tfidf.py add TF-IDF keyword distiller 2026-04-27 07:55:53 -04:00
test_vcs_source.py add git and mercurial repo sources for self-play 2026-04-27 18:17:39 -04:00
test_verify.py qa: question_hash strips articles + verifier coverage drops stopwords 2026-04-29 14:00:44 -04:00
test_versioned_ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
test_wikipedia_old.py prepare full Wikipedia 2003-05-16 ingest: cur + old (revisions) 2026-04-27 08:10:42 -04:00
test_wikipedia_xml.py add Phase IV Wikipedia XML + abstract sources 2026-04-27 17:24:28 -04:00
test_wikitext.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00