From c1dcd3142e3d1aa1254b39e79e8ded4bcab6251d Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Fri, 1 May 2026 16:19:50 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20seven-point-program=20north-star=20?= =?UTF-8?q?=E2=80=94=20directive=20matrix=20+=20status=20snapshot?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-05-01 distilled architectural directive becomes the audit lens applied before every new ticket / feature / prompt edit. Each of the seven directives gets: - status (✓ / ½ / ✗) - code anchors (verifier paths, renderer hooks) - pinning tests (per-rule coverage in tests/) - tickets covering remaining work - bench signal (which row column reports it) Status snapshot at landing: D1 (no LLM in verifier) ✓ D2 (pointer clauses) ✓ D3 (CTI internally) ½ → ticket #000002 D4 (retrieval map AND evidence map) ½ → ticket #000001 D5 (deterministic pointer verify) ✓ D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair) D7 (honest labels) ✓ D8 (test-pinning before automation) discipline CLAUDE.md docs index gains a "North-star" section pointing at the program; ticket entries gain a "Directive" column threading them back to the program. --- CLAUDE.md | 13 ++- docs/TICKETS.md | 11 +- docs/seven-point-program.md | 213 ++++++++++++++++++++++++++++++++++++ 3 files changed, 230 insertions(+), 7 deletions(-) create mode 100644 docs/seven-point-program.md diff --git a/CLAUDE.md b/CLAUDE.md index 2b456a1..0eb0a62 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -275,6 +275,12 @@ Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: ## Docs index +North-star: + +- `docs/seven-point-program.md` — the architectural directive distilled + 2026-05-01. Every new ticket / feature / prompt edit walks past this. + Bench harness reports per-mode directive coverage. + Architecture / ongoing work: - `docs/cti-architecture.md` — CTI Clause Tree Intelligence. @@ -291,9 +297,12 @@ Architecture / ongoing work: Tickets: `docs/TICKETS.md` (index + Next ID). Open tickets: - `docs/ticket-000001-retrieval-keywords-audit-gap.md` — provenance - binding for `--retrieval-keywords`. + binding for `--retrieval-keywords`. Directive D4. - `docs/ticket-000002-reference-frame-polarity-contract.md` — - Module L: multi-frame answer compilation. + Module L: multi-frame answer compilation. Directive D3. +- `docs/ticket-000003-anchor-class-warrant.md` — anchor-class + warrant generalization (entity-list / count / why-cause shapes). + Directive D6. ## Orientation protocol diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 38c86c2..9cb291d 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -53,11 +53,12 @@ implementation cost, it's a ticket. Otherwise it's reference. Newest first. Update on every open/close. -| ID | Title | Status | Opened | -|----------|------------------------------------------------|---------|------------| -| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | -| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | +| ID | Title | Status | Opened | Directive | +|----------|------------------------------------------------|---------|------------|-----------| +| #000003 | Anchor-class warrant generalization (Module H+)| open | 2026-05-01 | D6 | +| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | D3 | +| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | D4 | ## Next ID -`000003` +`000004` diff --git a/docs/seven-point-program.md b/docs/seven-point-program.md new file mode 100644 index 0000000..63c1d86 --- /dev/null +++ b/docs/seven-point-program.md @@ -0,0 +1,213 @@ +# Seven-point program — the north-star directive + +**Distilled 2026-05-01, Asia/Kuala_Lumpur**: + +> Stop making Hermes prove things. Make Hermes emit pointer clauses; +> build CTI internally; bind the retrieval map and evidence map; +> verify pointers deterministically; add general anchor-class warrant +> before semantic NLI; rename labels honestly; and automate only after +> these invariants are test-pinned. + +This is the architecture's north-star. Every new feature, prompt edit, +schema change, and ticket walks past these seven directives before +landing. The discipline frames how `bench/qa_sweep.py` and +`tests/test_qa_quality_live.py` cover the substrate: the bench is the +scoreboard, the live fixtures are the gates, and **this doc names what +the scoreboard and gates measure for**. + +## The seven directives + +### D1 — Stop making Hermes prove things + +The verifier never calls the LLM. Faithfulness classification is a +deterministic property of (answer text, evidence map, source corpus), +not a model self-grade. + +- **Status**: ✓ enforced. +- **Code anchors**: + `aborist/qa/verify.py:verify_quotes`, + `verify_claim_lattice`, `verify_claim_lattice_json` — pure functions + with no chat-client parameter. +- **Pinning tests**: `tests/test_verify*.py`, + `tests/test_claim_lattice.py` — every verifier path runs without a + network call. +- **Smell-test**: if a future PR adds a `chat_client` argument to a + verifier, D1 violated. + +### D2 — Make Hermes emit pointer clauses + +The model's prose surface is *citation-style* prose with bracketed +evidence-id tags (`Claim. [E12]`) or schema-constrained JSON +(`{"claims":[{"text":"...", "evidence_ids":["E12"]}]}`). The model +never types the quote string itself. + +- **Status**: ✓ enforced for lattice modes. +- **Code anchors**: + `aborist/qa/parse_claims.py:parse_pointer_claims`, + `aborist/qa/verify.py:CLAIM_LATTICE_JSON_SCHEMA`, + `aborist/qa/runner.py` (system prompts). +- **Pinning tests**: `tests/test_claim_lattice.py`, + `tests/test_verify_json.py`. +- **Bench signal**: rows with `answer_mode ∈ + {claim_lattice_pointer, claim_lattice}` fall under this directive; + rows with `answer_mode == "quote"` predate it (legacy). + +### D3 — Build CTI internally + +Clause-tree intelligence (frame lattice + multi-frame answer +compilation) lives **on our side of the wire**. The model proposes +clauses; the runtime composes them. Reference-frame queries get +multiple frames acknowledged in the answer (literal vs +fictional-actual vs in-universe-propaganda) rather than picking one +frame and discarding the others. + +- **Status**: ½ — phrase-route closed the **retrieval** side + (commit `1b8677d`); answer-side compilation pending. +- **Code anchors**: today, none. Future: + `aborist/qa/frame.py:detect_frame`, runtime-side polarity contract + emission. +- **Pinning tests**: today, only the phrase-route tests in + `test_query.py::test_phrase_match_surfaces_topical_doc`. Future: + per-shape frame-detector tests + multi-frame answer-shape live + fixtures. +- **Open ticket**: + [#000002 Reference-Frame Polarity Contract](ticket-000002-reference-frame-polarity-contract.md). +- **Bench signal**: today, `phrase_match_surfaced` flag on bench rows + (true when phrase route fired). Once Module L lands, additionally + `frame_kind ∈ {literal, reference, ambiguous}`. + +### D4 — Bind the retrieval map AND the evidence map + +Provenance binding covers both *what got retrieved* (sources) and +*how retrieval got there* (the operator-influenceable inputs: +keywords, top_k, over_fetch, max_context_chars, shard set). + +- **Status**: ½ — `evidence_map_root` lives in the 9-stage run-DAG + (`aborist/qa/dag.py`); retrieval-plan binding (the input side) is + the open work. +- **Code anchors**: `aborist/qa/dag.py:build_run_dag`, + `aborist/qa/evidence.py:evidence_map_root`. +- **Pinning tests**: `tests/test_qa_dag.py`, + `tests/test_evidence.py`. +- **Open ticket**: + [#000001 Retrieval-keywords audit gap](ticket-000001-retrieval-keywords-audit-gap.md). +- **Bench signal**: every lattice-mode row has `run_dag_root` + populated; future bench column will report on `retrieval_plan_hash` + presence once #000001 lands. + +### D5 — Verify pointers deterministically + +The hard verifier runs only deterministic checks (parser succeeded, +evidence_id resolves, source_role allowed, claim text non-empty, +citation coverage, pointer-count cap, lexical anchor warrant). +No NLI, no embeddings, no model self-grading. + +- **Status**: ✓ enforced. +- **Code anchors**: `aborist/qa/verify.py:verify_claim_lattice` — + seven hard checks documented inline. +- **Pinning tests**: `tests/test_claim_lattice.py` (per-rule + coverage), `tests/test_verify_json.py`, + `tests/test_verify.py`. +- **Bench signal**: every row carries `verifier_method ∈ {quote, + span, entity, paraphrase, claim_lattice, none}` (CHECK constraint + in `aborist/store.py`). + +### D6 — General anchor-class warrant before semantic NLI + +The lexical anchor check (Rule 7 in `verify_claim_lattice`) catches +the lazy-anchor failure where pointer / role / coverage all pass but +the cited chunk never names the answer entity. Today this fires +only on relation-shape questions. Generalizing the anchor-class +detector to entity-list / why-cause / when-date shapes pins a +broader class of failures structurally before any NLI substrate +exists in the proof path. + +- **Status**: ½ — relation-shape only. Entity-list / why-cause / + when-date shapes pass through unchecked. +- **Code anchors**: + `aborist/qa/verify.py:_extract_anchor_candidates`, + `aborist/qa/verify.py:_question_is_relation_shape`. +- **Pinning tests**: `tests/test_claim_lattice.py` covers + relation-shape; entity-list / cause / date shapes have no warrant + test today. +- **Open ticket**: + [#000003 Anchor-class warrant generalization](ticket-000003-anchor-class-warrant.md). +- **Bench signal**: future column `warrant_fired: bool` (true when + Rule 7 ran on the row, regardless of pass/fail). + +### D7 — Rename labels honestly + +`STRICT` for claim-lattice modes overclaims because synthesis-heavy +claims pass all hard checks without semantic entailment of the +joined assertion. The renderer relabels at display time: + +``` +STRICT → EVIDENCE-LINKED · via claim_lattice +HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer +UNGROUNDED → UNGROUNDED · via claim_lattice +``` + +Quote / span / entity / paraphrase modes keep their original tokens +(they verify against pinned spans, not synthesis). The schema column +stays `STRICT`/`HYBRID`/`UNGROUNDED` so v9.8 cache_key invariants +hold. + +- **Status**: ✓ enforced for lattice modes. +- **Code anchors**: `aborist/cli.py:_render_audit_label`. +- **Pinning tests**: `tests/test_cli_render.py` — + `test_render_label_strict_in_claim_lattice_becomes_evidence_linked` + and siblings. + +### D8 — Automate only after the invariants are test-pinned + +The five-step algorithm in CLAUDE.md applies to automation in +particular: bench harnesses, distill runners, ingest CLIs all earned +their automation only after the underlying ops were simplified into +single-purpose stages. **The bench harness IS the automation +substrate**, so it must enforce the directives before any feature +gets layered on top. + +- **Status**: ongoing discipline. +- **Code anchors**: `bench/qa_sweep.py`, + `tests/test_qa_quality_live.py`. +- **Bench signal**: per-row `directive_compliance` column maps + each directive to a pass/fail boolean; the markdown summary + aggregates per-mode-per-directive coverage. + +## How to use this doc + +1. **Before opening a new ticket**, scan the seven directives. Which + one does the work serve? Add the directive number to the ticket + header (`Directive: D6`) so the design log threads back here. + +2. **Before merging a feature**, audit it against D8: does a test + pin the new invariant? If not, the feature is automation + pretending to be code-level discipline. + +3. **When reading a bench summary**, the per-directive coverage row + tells you which directives are passing on the corpus today and + which are pending implementation. Bench scores that climb without + directive coverage climbing are graveyard-digging (step 4 of + the five-step algorithm). + +4. **When triaging a fixture failure**, identify which directive + the fixture pins. A fixture that fails by name maps to a specific + regression in the directive's structural invariant; an unnamed + bench delta is just noise. + +## Status snapshot at landing + +| # | Directive | Status | Tickets | +|---|----------------------------------------------|--------|-----------| +| 1 | Stop making Hermes prove things | ✓ | | +| 2 | Hermes emits pointer clauses | ✓ | | +| 3 | Build CTI internally | ½ | #000002 | +| 4 | Bind retrieval map AND evidence map | ½ | #000001 | +| 5 | Verify pointers deterministically | ✓ | | +| 6 | Anchor-class warrant before NLI | ½ | #000003 | +| 7 | Rename labels honestly | ✓ | | +| 8 | Automate only after test-pinning | discipline | | + +Three of seven structural directives are partial (D3, D4, D6). All +three have open tickets. D8 is the meta-discipline that gates how +all of the above land.