arborist/tests
russell@unturf.com 70ffc01ce4
fan-out: witness audit + 5F extractor + function-sampled demo + docs
Three small streams in one commit:

#000028 follow-up — witness divergence → 5F fixtures
=====================================================

Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.

New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).

Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.

Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).

Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.

#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================

`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.

  $ make demo-plot Q='sin(x)' PNG=/tmp/sin.png

Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.

Public docs polish (#7)
========================

- docs/_source/bench.rst: updated fixture-count narrative (~660 →
  662 default tasks + ~110 math π* fixtures); `make` quick-reference
  now lists all per-π* 5S targets (tabular, calculus-limit/series,
  linear-algebra, function-sampled) plus bench-real-shard,
  bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
  documentation + a Make-harness sub-section covering
  bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
  env-vars.

Tests
=====

- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
  write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
  determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
2026-05-09 13:19:30 -04:00
..
crawler aborist/arborist 2026-05-07 09:31:49 -04:00
fixtures verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_bench_batteries.py fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 2026-05-09 12:42:56 -04:00
test_bench_qa_sweep.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_burn.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_burn_doc.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_canonical_cache.py #000027 + #000028: canonical projections persist; STRICT-WITNESSED reachable 2026-05-09 11:37:06 -04:00
test_canonical_projection.py ticket #000030 Phases 1+2: algebra-symbolic@v1 + calculus-derivative@v1 2026-05-09 12:35:18 -04:00
test_capital.py capital: land ticket #000020 (8-capital-form cost ledger) 2026-05-07 16:41:48 -04:00
test_claim_lattice.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_claim_pack.py ticket #000029: claim-pack source for axiom/theorem JSON bundles 2026-05-09 11:44:08 -04:00
test_cli_render.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_cli_session.py tests: unit + integration + functional coverage for this session's surface 2026-05-07 21:15:14 -04:00
test_compress.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_concepts.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_dag.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_demo_plot.py fan-out: witness audit + 5F extractor + function-sampled demo + docs 2026-05-09 13:19:30 -04:00
test_directives.py #000027 + #000028: canonical projections persist; STRICT-WITNESSED reachable 2026-05-09 11:37:06 -04:00
test_distill.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_distill_recursive.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_evict.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_frame.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_grok_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_html_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_ingest.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_inspect.py fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 2026-05-09 12:42:56 -04:00
test_journal.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_keys.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_llm_context_base.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_loss_report.py loss_report: land ticket #000022 (adapter LossReport sidecar) 2026-05-07 17:58:50 -04:00
test_memory_root.py memory: land ticket #000017 (memory-root lifelong learning summary) 2026-05-07 16:46:41 -04:00
test_merkle.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_aead.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_chain.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_cli.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_cli_pull.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_wire.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_wire_e2e.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_metacognition.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_migration_audit_mode.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_pi_star.py fan-out: 5 π* graduations close the registry chapter 2026-05-09 13:04:43 -04:00
test_pi_star_algebra_symbolic.py ticket #000030 Phases 1+2: algebra-symbolic@v1 + calculus-derivative@v1 2026-05-09 12:35:18 -04:00
test_pi_star_algebra_symbolic_simplified.py ticket #000030 Phases 1b+3 + open #000031 2026-05-09 12:50:46 -04:00
test_pi_star_calculus_derivative.py ticket #000030 Phases 1+2: algebra-symbolic@v1 + calculus-derivative@v1 2026-05-09 12:35:18 -04:00
test_pi_star_calculus_integral.py ticket #000030 Phases 1b+3 + open #000031 2026-05-09 12:50:46 -04:00
test_pi_star_phase_3_to_7.py fan-out: 5 π* graduations close the registry chapter 2026-05-09 13:04:43 -04:00
test_pi_star_tabular.py fan-out: 5 π* graduations close the registry chapter 2026-05-09 13:04:43 -04:00
test_providence_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_qa.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_qa_quality_live.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_caps.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_classifier.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_phase4.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_reminder.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_query.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_reclassify.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_repair.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_resume.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_selfmodel.py selfmodel: land ticket #000014 (identity record + falsification) 2026-05-07 16:36:34 -04:00
test_session_integration.py fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 2026-05-09 12:42:56 -04:00
test_session_migrations.py tests: unit + integration + functional coverage for this session's surface 2026-05-07 21:15:14 -04:00
test_snapshot.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_soft_preflight.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_store_migration_memoization.py store: per-process migration memoization (#000026 Phase 1) 2026-05-08 12:48:48 -04:00
test_tfidf.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_v8_fork_score.py fan-out: ForkScore CLI · authorship warrant ladder · 5F Phase 1c 2026-05-09 12:14:38 -04:00
test_vcs_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_verify.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_verify_json.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_versioned_ingest.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_warrant.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_warrant_authorship.py fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 2026-05-09 12:42:56 -04:00
test_wikipedia_old.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_wikipedia_xml.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_wikitext.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_witness.py fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 2026-05-09 12:42:56 -04:00
test_witness_to_5f.py fan-out: witness audit + 5F extractor + function-sampled demo + docs 2026-05-09 13:19:30 -04:00