# Seven-point program — the north-star directive **Distilled 2026-05-01, Asia/Kuala_Lumpur**: > Stop making Hermes prove things. Make Hermes emit pointer clauses; > build CTI internally; bind the retrieval map and evidence map; > verify pointers deterministically; add general anchor-class warrant > before semantic NLI; rename labels honestly; and automate only after > these invariants are test-pinned. This is the architecture's north-star. Every new feature, prompt edit, schema change, and ticket walks past these seven directives before landing. The discipline frames how `bench/qa_sweep.py` and `tests/test_qa_quality_live.py` cover the substrate: the bench is the scoreboard, the live fixtures are the gates, and **this doc names what the scoreboard and gates measure for**. ## The seven directives ### D1 — Stop making Hermes prove things The verifier never calls the LLM. Faithfulness classification is a deterministic property of (answer text, evidence map, source corpus), not a model self-grade. - **Status**: ✓ enforced. - **Code anchors**: `arborist/qa/verify.py:verify_quotes`, `verify_claim_lattice`, `verify_claim_lattice_json` — pure functions with no chat-client parameter. - **Pinning tests**: `tests/test_verify*.py`, `tests/test_claim_lattice.py` — every verifier path runs without a network call. - **Smell-test**: if a future PR adds a `chat_client` argument to a verifier, D1 violated. ### D2 — Make Hermes emit pointer clauses The model's prose surface is *citation-style* prose with bracketed evidence-id tags (`Claim. [E12]`) or schema-constrained JSON (`{"claims":[{"text":"...", "evidence_ids":["E12"]}]}`). The model never types the quote string itself. - **Status**: ✓ enforced for lattice modes. - **Code anchors**: `arborist/qa/parse_claims.py:parse_pointer_claims`, `arborist/qa/verify.py:CLAIM_LATTICE_JSON_SCHEMA`, `arborist/qa/runner.py` (system prompts). - **Pinning tests**: `tests/test_claim_lattice.py`, `tests/test_verify_json.py`. - **Bench signal**: rows with `answer_mode ∈ {claim_lattice_pointer, claim_lattice}` fall under this directive; rows with `answer_mode == "quote"` predate it (legacy). ### D3 — Build CTI internally Clause-tree intelligence (frame lattice + multi-frame answer compilation) lives **on our side of the wire**. The model proposes clauses; the runtime composes them. Reference-frame queries get multiple frames acknowledged in the answer (literal vs fictional-actual vs in-universe-propaganda) rather than picking one frame and discarding the others. - **Status**: ¾ — phrase-route closed the retrieval side (commit `1b8677d`); pre-answer preflight contract (#000008 quantifier classifier + #000010 metacognition QuestionState) landed 2026-05-03. Answer-side multi-frame compilation still pending. - **Code anchors**: `arborist/qa/quantifier.py:classify_question_quantifier` (#000008), `arborist/qa/metacognition.py:preflight_question` (#000010), `arborist/qa/dag.py:preflight_node_hash` / `build_preflight_node_payload` (#000009 — five nested CTI clauses bind into the run-DAG `preflight` stage). Future: runtime-side polarity contract emission for multi-frame answers. - **Pinning tests**: `tests/test_quantifier_classifier.py` (10-rung intensity ladder, 70 tests), `tests/test_metacognition.py` (4 detectors + governance + audit-line tails, 58 tests), `tests/test_dag.py::test_preflight_*` (DAG binding; subset of the `test_dag.py` suite tagged below). Future: per-shape frame-detector tests + multi-frame answer-shape live fixtures. - **Open ticket**: [#000002 Reference-Frame Polarity Contract](ticket-000002-reference-frame-polarity-contract.md). - **Bench signal**: lattice-mode bench rows now carry `quantifier_intensity`, `quantifier_matched_token`, `scope_bound_hint`, `preflight_logical_statuses`, `preflight_question_shape`, `preflight_result`. Audit-line tails surface five metacog kinds: `· false premise`, `· contradictory`, `· stale risk`, `· out of corpus`, `· frame ambiguous`. ### D4 — Bind the retrieval map AND the evidence map Provenance binding covers both *what got retrieved* (sources) and *how retrieval got there* (the operator-influenceable inputs: keywords, top_k, over_fetch, max_context_chars, shard set). Plus the **preflight contract** governing the run (#000009 — guard / cap / reminder / reject decisions hash-bound). - **Status**: ¾ — `evidence_map_root` lives in the run-DAG (`arborist/qa/dag.py`); `retrieval_plan_hash` landed via #000001. Preflight stage (#000009) added the upstream control commitment: classifier output + answer/prompt/evidence contracts + `governance_policy_hash` / `model_profile_hash` references all fold into a single `preflight` Merkle leaf. Quote mode now 7→8 stages; pointer-mode CTI 9→10 stages; reject-broad path emits a 3-stage minimal DAG (`question → preflight → final_label`). - **Code anchors**: `arborist/qa/dag.py:build_run_dag` (legacy + preflight-extended shapes), `build_reject_run_dag` (3-stage), `preflight_node_hash` / `build_preflight_node_payload` (5-clause payload), `arborist/qa/evidence.py:evidence_map_root`. - **Pinning tests**: `tests/test_dag.py` (33 tests including preflight stage + reject path), `tests/test_evidence.py`. - **Open ticket**: [#000001 Retrieval-keywords audit gap](ticket-000001-retrieval-keywords-audit-gap.md) closed; future retrieval-side refinement still possible. - **Bench signal**: every lattice-mode row has `run_dag_root` populated. Two cache rows that share the same question + same model output + same verifier verdict but different preflight policy state now produce different `run_dag_root` values — audit replay can pin the policy decision per row. ### D5 — Verify pointers deterministically The hard verifier runs only deterministic checks (parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage, pointer-count cap, lexical anchor warrant). No NLI, no embeddings, no model self-grading. - **Status**: ✓ enforced. - **Code anchors**: `arborist/qa/verify.py:verify_claim_lattice` — seven hard checks documented inline. - **Pinning tests**: `tests/test_claim_lattice.py` (per-rule coverage), `tests/test_verify_json.py`, `tests/test_verify.py`. - **Bench signal**: every row carries `verifier_method ∈ {quote, span, entity, paraphrase, claim_lattice, none}` (CHECK constraint in `arborist/store.py`). ### D6 — General anchor-class warrant before semantic NLI The lexical anchor check (Rule 7 in `verify_claim_lattice`) catches the lazy-anchor failure where pointer / role / coverage all pass but the cited chunk never names the answer entity. Today this fires only on relation-shape questions. Generalizing the anchor-class detector to entity-list / why-cause / when-date shapes pins a broader class of failures structurally before any NLI substrate exists in the proof path. - **Status**: ½ — relation-shape only. Entity-list / why-cause / when-date shapes pass through unchecked. - **Code anchors**: `arborist/qa/verify.py:_extract_anchor_candidates`, `arborist/qa/verify.py:_question_is_relation_shape`. - **Pinning tests**: `tests/test_claim_lattice.py` covers relation-shape; entity-list / cause / date shapes have no warrant test today. - **Open ticket**: [#000003 Anchor-class warrant generalization](ticket-000003-anchor-class-warrant.md). - **Bench signal**: future column `warrant_fired: bool` (true when Rule 7 ran on the row, regardless of pass/fail). ### D7 — Rename labels honestly `STRICT` for claim-lattice modes overclaims because synthesis-heavy claims pass all hard checks without semantic entailment of the joined assertion. The renderer relabels at display time: ``` STRICT → EVIDENCE-LINKED · via claim_lattice HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer UNGROUNDED → UNGROUNDED · via claim_lattice ``` Quote / span / entity / paraphrase modes keep their original tokens (they verify against pinned spans, not synthesis). The schema column stays `STRICT`/`HYBRID`/`UNGROUNDED` so v9.8 cache_key invariants hold. - **Status**: ✓ enforced for lattice modes. - **Code anchors**: `arborist/cli.py:_render_audit_label`. - **Pinning tests**: `tests/test_cli_render.py` — `test_render_label_strict_in_claim_lattice_becomes_evidence_linked` and siblings. ### D8 — Automate only after the invariants are test-pinned The five-step algorithm in CLAUDE.md applies to automation in particular: bench harnesses, distill runners, ingest CLIs all earned their automation only after the underlying ops were simplified into single-purpose stages. **The bench harness IS the automation substrate**, so it must enforce the directives before any feature gets layered on top. - **Status**: ongoing discipline. - **Code anchors**: `bench/qa_sweep.py`, `tests/test_qa_quality_live.py`. - **Bench signal**: per-row `directive_compliance` column maps each directive to a pass/fail boolean; the markdown summary aggregates per-mode-per-directive coverage. ## How to use this doc 1. **Before opening a new ticket**, scan the seven directives. Which one does the work serve? Add the directive number to the ticket header (`Directive: D6`) so the design log threads back here. 2. **Before merging a feature**, audit it against D8: does a test pin the new invariant? If not, the feature is automation pretending to be code-level discipline. 3. **When reading a bench summary**, the per-directive coverage row tells you which directives are passing on the corpus today and which are pending implementation. Bench scores that climb without directive coverage climbing are graveyard-digging (step 4 of the five-step algorithm). 4. **When triaging a fixture failure**, identify which directive the fixture pins. A fixture that fails by name maps to a specific regression in the directive's structural invariant; an unnamed bench delta is just noise. ## Status snapshot at landing | # | Directive | Status | Tickets | |---|----------------------------------------------|------------|-------------------------------| | 1 | Stop making Hermes prove things | ✓ | #000010 (closed) | | 2 | Hermes emits pointer clauses | ✓ | | | 3 | Build CTI internally | ✓ | #000002, #000008, #000010 (closed) | | 4 | Bind retrieval map AND evidence map | ✓ | #000001, #000009 (closed) | | 5 | Verify pointers deterministically | ✓ | | | 6 | Anchor-class warrant before NLI | ✓ | #000003 (closed) | | 7 | Rename labels honestly | ✓ | #000005 (closed) | | 8 | Automate only after test-pinning | discipline | #000004 (closed) | All seven structural directives now ✓: D1, D2, D5 — were ✓ at landing. D3 closed 2026-05-02 via #000002 (frame detector + polarity preamble + renderer extension; multi-frame answers on Orwell and other reference-frame queries). D4 closed 2026-05-02 via #000001 (retrieval_plan_hash binding in the run-DAG; SQL column + audit events deferred). D6 closed 2026-05-02 via #000003 (anchor-class warrant generalization: entity-list / count / why-cause shapes). D7 closed 2026-05-02 via #000005 (four-rung ladder: POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → ENTAILMENT-VERIFIED). D8 is the meta-discipline that gates how all of the above land; the bench coverage substrate (#000004) is closed. Tests: 734 passed at the original snapshot. ### Post-landing addendum (2026-05-03 / 2026-05-04) Three additional tickets landed extending D1, D3, and D4: - **#000008** (closed `4f2b5a6`, 2026-05-03) — Broad-quantifier preflight guard. New modules `arborist/qa/quantifier.py`, `model_profiles.py`, `quantifier_reminder.py`. Seven new policy fields fold into `governance_policy_hash`; six-level disable hierarchy. Defaults flipped (after n=5 verification): reminder default-on for lattice modes, cap default operator- opt-in with JSON-only allowlist. - **#000010** (closed, 2026-05-03) — Meta-Cognition Preflight Guard. New module `arborist/qa/metacognition.py` with `QuestionState` dataclass + four pure detectors (temporal, contradiction, false-premise-lite, out-of-corpus). Six new policy fields, all default-on except `metacognition_block_on_contradiction`. Audit-line tail tokens: `· false premise`, `· contradictory`, `· stale risk`, `· out of corpus`, `· frame ambiguous`. Closes D1 by giving the runtime explicit epistemic state to refuse from instead of asking Hermes to self-regulate. - **#000009** (closed `111dda6`, 2026-05-04) — Preflight run-DAG node binding. Inserts `preflight` stage between `question` and `retrieval`. Quote mode 7→8 stages; pointer-mode CTI 9→10 stages. Reject-broad early-return path emits a 3-stage minimal DAG (`question → preflight → final_label`). Five nested CTI clauses (classifier / answer_contract / prompt_contract / evidence_contract / policy_refs) inside a single hashable leaf. Closes D4 by Merkle-binding the upstream control decisions that previously only appeared on result dicts. Test count after these landings: **993 passing** (up from 734; 259 new across the three tickets). All seven directives remain ✓. **Bench-validated post-landing** (full 75-question sweep 2026-05-03T23-30-12Z vs frozen 2026-05-02T15-07Z baseline, n=3 × 3 modes = 675 runs): - quote SR 0.54 → 0.52 (−2pp, within noise) - pointer SR 0.20 → 0.21 (+1pp, within noise) - JSON SR 0.42 → 0.44 (+2pp, within noise) All STRICT-rate deltas within the 5pp signal floor. Substrate- level wins: pointer-mode FORMAT_COLLAPSED eliminated globally (0/225), pointer-mode NO_EVIDENCE_POINTER drops from 33% (broad-only when reminder was off) to 13% (full sweep with reminder on). The default flip is strictly more honest without regressing throughput on narrow questions. See `docs/qa-modes-bench.md` Addendum 3 for the full comparison. ## Anti-regression test layer Each directive has a structural pin in `tests/test_directives.py`. A future PR that silently weakens a directive fails by name: | Directive | Pin tests in `tests/test_directives.py` | |-----------|---------------------------------------------------------------------------| | D1 | `test_d1_verify_*_signature_has_no_chat_client`, `test_d1_verifier_method_enum_excludes_llm_judges` | | D2 | `test_d2_answer_modes_include_lattice_variants`, `test_d2_pointer_parser_exists_*` | | D3 | `test_d3_runtime_owns_claim_lattice_construction`, `test_d3_evidence_map_built_*` | | D4 | `test_d4_run_dag_carries_evidence_map_root` (✓), `test_d4_retrieval_plan_binding_status` (marker — flips when #000001 lands) | | D5 | `test_d5_verify_claim_lattice_is_deterministic`, `test_d5_seven_hard_checks_are_pure_functions` | | D6 | `test_d6_warrant_*` (relation + date), `test_d6_warrant_generalization_status` (marker — flips when #000003 lands) | | D7 | `test_d7_renderer_*`, `test_d7_audit_mode_enum_canonical_set` | | D8 | `test_d8_seven_point_program_doc_exists`, `test_d8_bench_directive_compliance_helper_exists`, `test_d8_tickets_index_exists_and_pins_open_directives` | The "marker" tests for D4 and D6 document open work: today they assert that a parameter / detector is **absent**; when the corresponding ticket lands, the assertion flips to **present**. The test failing during ticket implementation is the expected signal, not a regression.