docs: seven-point-program north-star — directive matrix + status snapshot
The 2026-05-01 distilled architectural directive becomes the audit lens applied before every new ticket / feature / prompt edit. Each of the seven directives gets: - status (✓ / ½ / ✗) - code anchors (verifier paths, renderer hooks) - pinning tests (per-rule coverage in tests/) - tickets covering remaining work - bench signal (which row column reports it) Status snapshot at landing: D1 (no LLM in verifier) ✓ D2 (pointer clauses) ✓ D3 (CTI internally) ½ → ticket #000002 D4 (retrieval map AND evidence map) ½ → ticket #000001 D5 (deterministic pointer verify) ✓ D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair) D7 (honest labels) ✓ D8 (test-pinning before automation) discipline CLAUDE.md docs index gains a "North-star" section pointing at the program; ticket entries gain a "Directive" column threading them back to the program.
This commit is contained in:
parent
7ba1b405ee
commit
c1dcd3142e
3 changed files with 230 additions and 7 deletions
13
CLAUDE.md
13
CLAUDE.md
|
|
@ -275,6 +275,12 @@ Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:
|
|||
|
||||
## Docs index
|
||||
|
||||
North-star:
|
||||
|
||||
- `docs/seven-point-program.md` — the architectural directive distilled
|
||||
2026-05-01. Every new ticket / feature / prompt edit walks past this.
|
||||
Bench harness reports per-mode directive coverage.
|
||||
|
||||
Architecture / ongoing work:
|
||||
|
||||
- `docs/cti-architecture.md` — CTI Clause Tree Intelligence.
|
||||
|
|
@ -291,9 +297,12 @@ Architecture / ongoing work:
|
|||
Tickets: `docs/TICKETS.md` (index + Next ID). Open tickets:
|
||||
|
||||
- `docs/ticket-000001-retrieval-keywords-audit-gap.md` — provenance
|
||||
binding for `--retrieval-keywords`.
|
||||
binding for `--retrieval-keywords`. Directive D4.
|
||||
- `docs/ticket-000002-reference-frame-polarity-contract.md` —
|
||||
Module L: multi-frame answer compilation.
|
||||
Module L: multi-frame answer compilation. Directive D3.
|
||||
- `docs/ticket-000003-anchor-class-warrant.md` — anchor-class
|
||||
warrant generalization (entity-list / count / why-cause shapes).
|
||||
Directive D6.
|
||||
|
||||
## Orientation protocol
|
||||
|
||||
|
|
|
|||
|
|
@ -53,11 +53,12 @@ implementation cost, it's a ticket. Otherwise it's reference.
|
|||
|
||||
Newest first. Update on every open/close.
|
||||
|
||||
| ID | Title | Status | Opened |
|
||||
|----------|------------------------------------------------|---------|------------|
|
||||
| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 |
|
||||
| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 |
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|---------|------------|-----------|
|
||||
| #000003 | Anchor-class warrant generalization (Module H+)| open | 2026-05-01 | D6 |
|
||||
| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | D3 |
|
||||
| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | D4 |
|
||||
|
||||
## Next ID
|
||||
|
||||
`000003`
|
||||
`000004`
|
||||
|
|
|
|||
213
docs/seven-point-program.md
Normal file
213
docs/seven-point-program.md
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
# Seven-point program — the north-star directive
|
||||
|
||||
**Distilled 2026-05-01, Asia/Kuala_Lumpur**:
|
||||
|
||||
> Stop making Hermes prove things. Make Hermes emit pointer clauses;
|
||||
> build CTI internally; bind the retrieval map and evidence map;
|
||||
> verify pointers deterministically; add general anchor-class warrant
|
||||
> before semantic NLI; rename labels honestly; and automate only after
|
||||
> these invariants are test-pinned.
|
||||
|
||||
This is the architecture's north-star. Every new feature, prompt edit,
|
||||
schema change, and ticket walks past these seven directives before
|
||||
landing. The discipline frames how `bench/qa_sweep.py` and
|
||||
`tests/test_qa_quality_live.py` cover the substrate: the bench is the
|
||||
scoreboard, the live fixtures are the gates, and **this doc names what
|
||||
the scoreboard and gates measure for**.
|
||||
|
||||
## The seven directives
|
||||
|
||||
### D1 — Stop making Hermes prove things
|
||||
|
||||
The verifier never calls the LLM. Faithfulness classification is a
|
||||
deterministic property of (answer text, evidence map, source corpus),
|
||||
not a model self-grade.
|
||||
|
||||
- **Status**: ✓ enforced.
|
||||
- **Code anchors**:
|
||||
`aborist/qa/verify.py:verify_quotes`,
|
||||
`verify_claim_lattice`, `verify_claim_lattice_json` — pure functions
|
||||
with no chat-client parameter.
|
||||
- **Pinning tests**: `tests/test_verify*.py`,
|
||||
`tests/test_claim_lattice.py` — every verifier path runs without a
|
||||
network call.
|
||||
- **Smell-test**: if a future PR adds a `chat_client` argument to a
|
||||
verifier, D1 violated.
|
||||
|
||||
### D2 — Make Hermes emit pointer clauses
|
||||
|
||||
The model's prose surface is *citation-style* prose with bracketed
|
||||
evidence-id tags (`Claim. [E12]`) or schema-constrained JSON
|
||||
(`{"claims":[{"text":"...", "evidence_ids":["E12"]}]}`). The model
|
||||
never types the quote string itself.
|
||||
|
||||
- **Status**: ✓ enforced for lattice modes.
|
||||
- **Code anchors**:
|
||||
`aborist/qa/parse_claims.py:parse_pointer_claims`,
|
||||
`aborist/qa/verify.py:CLAIM_LATTICE_JSON_SCHEMA`,
|
||||
`aborist/qa/runner.py` (system prompts).
|
||||
- **Pinning tests**: `tests/test_claim_lattice.py`,
|
||||
`tests/test_verify_json.py`.
|
||||
- **Bench signal**: rows with `answer_mode ∈
|
||||
{claim_lattice_pointer, claim_lattice}` fall under this directive;
|
||||
rows with `answer_mode == "quote"` predate it (legacy).
|
||||
|
||||
### D3 — Build CTI internally
|
||||
|
||||
Clause-tree intelligence (frame lattice + multi-frame answer
|
||||
compilation) lives **on our side of the wire**. The model proposes
|
||||
clauses; the runtime composes them. Reference-frame queries get
|
||||
multiple frames acknowledged in the answer (literal vs
|
||||
fictional-actual vs in-universe-propaganda) rather than picking one
|
||||
frame and discarding the others.
|
||||
|
||||
- **Status**: ½ — phrase-route closed the **retrieval** side
|
||||
(commit `1b8677d`); answer-side compilation pending.
|
||||
- **Code anchors**: today, none. Future:
|
||||
`aborist/qa/frame.py:detect_frame`, runtime-side polarity contract
|
||||
emission.
|
||||
- **Pinning tests**: today, only the phrase-route tests in
|
||||
`test_query.py::test_phrase_match_surfaces_topical_doc`. Future:
|
||||
per-shape frame-detector tests + multi-frame answer-shape live
|
||||
fixtures.
|
||||
- **Open ticket**:
|
||||
[#000002 Reference-Frame Polarity Contract](ticket-000002-reference-frame-polarity-contract.md).
|
||||
- **Bench signal**: today, `phrase_match_surfaced` flag on bench rows
|
||||
(true when phrase route fired). Once Module L lands, additionally
|
||||
`frame_kind ∈ {literal, reference, ambiguous}`.
|
||||
|
||||
### D4 — Bind the retrieval map AND the evidence map
|
||||
|
||||
Provenance binding covers both *what got retrieved* (sources) and
|
||||
*how retrieval got there* (the operator-influenceable inputs:
|
||||
keywords, top_k, over_fetch, max_context_chars, shard set).
|
||||
|
||||
- **Status**: ½ — `evidence_map_root` lives in the 9-stage run-DAG
|
||||
(`aborist/qa/dag.py`); retrieval-plan binding (the input side) is
|
||||
the open work.
|
||||
- **Code anchors**: `aborist/qa/dag.py:build_run_dag`,
|
||||
`aborist/qa/evidence.py:evidence_map_root`.
|
||||
- **Pinning tests**: `tests/test_qa_dag.py`,
|
||||
`tests/test_evidence.py`.
|
||||
- **Open ticket**:
|
||||
[#000001 Retrieval-keywords audit gap](ticket-000001-retrieval-keywords-audit-gap.md).
|
||||
- **Bench signal**: every lattice-mode row has `run_dag_root`
|
||||
populated; future bench column will report on `retrieval_plan_hash`
|
||||
presence once #000001 lands.
|
||||
|
||||
### D5 — Verify pointers deterministically
|
||||
|
||||
The hard verifier runs only deterministic checks (parser succeeded,
|
||||
evidence_id resolves, source_role allowed, claim text non-empty,
|
||||
citation coverage, pointer-count cap, lexical anchor warrant).
|
||||
No NLI, no embeddings, no model self-grading.
|
||||
|
||||
- **Status**: ✓ enforced.
|
||||
- **Code anchors**: `aborist/qa/verify.py:verify_claim_lattice` —
|
||||
seven hard checks documented inline.
|
||||
- **Pinning tests**: `tests/test_claim_lattice.py` (per-rule
|
||||
coverage), `tests/test_verify_json.py`,
|
||||
`tests/test_verify.py`.
|
||||
- **Bench signal**: every row carries `verifier_method ∈ {quote,
|
||||
span, entity, paraphrase, claim_lattice, none}` (CHECK constraint
|
||||
in `aborist/store.py`).
|
||||
|
||||
### D6 — General anchor-class warrant before semantic NLI
|
||||
|
||||
The lexical anchor check (Rule 7 in `verify_claim_lattice`) catches
|
||||
the lazy-anchor failure where pointer / role / coverage all pass but
|
||||
the cited chunk never names the answer entity. Today this fires
|
||||
only on relation-shape questions. Generalizing the anchor-class
|
||||
detector to entity-list / why-cause / when-date shapes pins a
|
||||
broader class of failures structurally before any NLI substrate
|
||||
exists in the proof path.
|
||||
|
||||
- **Status**: ½ — relation-shape only. Entity-list / why-cause /
|
||||
when-date shapes pass through unchecked.
|
||||
- **Code anchors**:
|
||||
`aborist/qa/verify.py:_extract_anchor_candidates`,
|
||||
`aborist/qa/verify.py:_question_is_relation_shape`.
|
||||
- **Pinning tests**: `tests/test_claim_lattice.py` covers
|
||||
relation-shape; entity-list / cause / date shapes have no warrant
|
||||
test today.
|
||||
- **Open ticket**:
|
||||
[#000003 Anchor-class warrant generalization](ticket-000003-anchor-class-warrant.md).
|
||||
- **Bench signal**: future column `warrant_fired: bool` (true when
|
||||
Rule 7 ran on the row, regardless of pass/fail).
|
||||
|
||||
### D7 — Rename labels honestly
|
||||
|
||||
`STRICT` for claim-lattice modes overclaims because synthesis-heavy
|
||||
claims pass all hard checks without semantic entailment of the
|
||||
joined assertion. The renderer relabels at display time:
|
||||
|
||||
```
|
||||
STRICT → EVIDENCE-LINKED · via claim_lattice
|
||||
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer
|
||||
UNGROUNDED → UNGROUNDED · via claim_lattice
|
||||
```
|
||||
|
||||
Quote / span / entity / paraphrase modes keep their original tokens
|
||||
(they verify against pinned spans, not synthesis). The schema column
|
||||
stays `STRICT`/`HYBRID`/`UNGROUNDED` so v9.8 cache_key invariants
|
||||
hold.
|
||||
|
||||
- **Status**: ✓ enforced for lattice modes.
|
||||
- **Code anchors**: `aborist/cli.py:_render_audit_label`.
|
||||
- **Pinning tests**: `tests/test_cli_render.py` —
|
||||
`test_render_label_strict_in_claim_lattice_becomes_evidence_linked`
|
||||
and siblings.
|
||||
|
||||
### D8 — Automate only after the invariants are test-pinned
|
||||
|
||||
The five-step algorithm in CLAUDE.md applies to automation in
|
||||
particular: bench harnesses, distill runners, ingest CLIs all earned
|
||||
their automation only after the underlying ops were simplified into
|
||||
single-purpose stages. **The bench harness IS the automation
|
||||
substrate**, so it must enforce the directives before any feature
|
||||
gets layered on top.
|
||||
|
||||
- **Status**: ongoing discipline.
|
||||
- **Code anchors**: `bench/qa_sweep.py`,
|
||||
`tests/test_qa_quality_live.py`.
|
||||
- **Bench signal**: per-row `directive_compliance` column maps
|
||||
each directive to a pass/fail boolean; the markdown summary
|
||||
aggregates per-mode-per-directive coverage.
|
||||
|
||||
## How to use this doc
|
||||
|
||||
1. **Before opening a new ticket**, scan the seven directives. Which
|
||||
one does the work serve? Add the directive number to the ticket
|
||||
header (`Directive: D6`) so the design log threads back here.
|
||||
|
||||
2. **Before merging a feature**, audit it against D8: does a test
|
||||
pin the new invariant? If not, the feature is automation
|
||||
pretending to be code-level discipline.
|
||||
|
||||
3. **When reading a bench summary**, the per-directive coverage row
|
||||
tells you which directives are passing on the corpus today and
|
||||
which are pending implementation. Bench scores that climb without
|
||||
directive coverage climbing are graveyard-digging (step 4 of
|
||||
the five-step algorithm).
|
||||
|
||||
4. **When triaging a fixture failure**, identify which directive
|
||||
the fixture pins. A fixture that fails by name maps to a specific
|
||||
regression in the directive's structural invariant; an unnamed
|
||||
bench delta is just noise.
|
||||
|
||||
## Status snapshot at landing
|
||||
|
||||
| # | Directive | Status | Tickets |
|
||||
|---|----------------------------------------------|--------|-----------|
|
||||
| 1 | Stop making Hermes prove things | ✓ | |
|
||||
| 2 | Hermes emits pointer clauses | ✓ | |
|
||||
| 3 | Build CTI internally | ½ | #000002 |
|
||||
| 4 | Bind retrieval map AND evidence map | ½ | #000001 |
|
||||
| 5 | Verify pointers deterministically | ✓ | |
|
||||
| 6 | Anchor-class warrant before NLI | ½ | #000003 |
|
||||
| 7 | Rename labels honestly | ✓ | |
|
||||
| 8 | Automate only after test-pinning | discipline | |
|
||||
|
||||
Three of seven structural directives are partial (D3, D4, D6). All
|
||||
three have open tickets. D8 is the meta-discipline that gates how
|
||||
all of the above land.
|
||||
Loading…
Add table
Add a link
Reference in a new issue