arborist/bench
russell@unturf.com acd1f9ca84
bench: per-row directive_compliance + 'directive coverage' summary section
Wires the seven-point program (docs/seven-point-program.md) into
the bench harness as a per-mode coverage table.

Per-row computation in _directive_compliance(answer_mode, result, err):
  D2_pointer_clauses        — answer_mode is lattice variant
  D3_cti_substrate_ready    — lattice mode + run_dag_root populated
                              (full coverage pending ticket #000002)
  D4_evidence_map_bound     — run_dag_root populated
                              (retrieval-plan binding pending #000001)
  D6_warrant_fired          — warrant ran (lattice mode today;
                              per-shape gating pending #000003)
  D7_honest_label           — audit_mode in canonical enum
                              (renderer transformation pinned in
                              test_cli_render.py)

D1 (no LLM in verifier), D5 (verifier_method enum), D8 (test-pinning
discipline) are global properties of the substrate and don't appear
per-row — they get tracked once in CLAUDE.md / the program doc.

_summarize aggregates per-row booleans into per-mode pass counts.
_render_markdown emits a 'directive coverage (seven-point program)'
section with per-mode per-directive coverage as 'count/N (pct%)'.

Discipline rationale per CLAUDE.md's five-step algorithm step 5:
the bench harness IS the automation substrate, so it must enforce
the directives before any feature gets layered on top. Bench scores
that climb without directive coverage climbing are graveyard-digging
(step 4: don't dig the grave faster).

5 new bench tests cover: per-mode pass aggregation, the markdown
section presence and content, the helper's behavior on quote mode
(D2 fail by design), lattice mode (all directives pass), and error
rows (empty dict — no signal). Full suite 669 passed.
2026-05-01 16:20:25 -04:00
..
qa_questions.txt qa: phrase-pattern retrieval route closes the reference-frame failure class 2026-05-01 13:53:03 -04:00
qa_sweep.py bench: per-row directive_compliance + 'directive coverage' summary section 2026-05-01 16:20:25 -04:00
run.sh progress reporter + structured benchmark 2026-04-27 11:37:20 -04:00