russell@unturf.com
fc5ba507dc
The doc-drift pattern recurred four times today on 2026-05-10 (commits6cbbf95,14bcb99,5c21e83,30a9488). Each fix was the same shape: walk a doc, find a count that drifted from live truth during the hours after the doc was written, refresh it. Cost: ~5 min per drift × 4 = 20 min of manual catching, with no guarantee the next drift gets caught before someone external reads it. Per fox's selection: regression test that makes drift loud at test time instead of relying on visual catching. == Mechanism == `tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT tags of the form: <!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT--> Two metrics supported: - `tests` — pytest collected count for path. Batches every tagged path into one `pytest --collect-only` subprocess (~0.5s total). - `fixture-rows` — non-blank-non-comment line count in a JSONL fixture. GitHub and most markdown renderers strip HTML comments, so readers see only `N`. The tags are invisible in rendered output but make the claim machine-checkable. Three tests in the file: 1. `test_doc_autocount_claims_match_live` — the core invariant 2. `test_autocount_tags_are_well_formed` — open/close balance 3. `test_autocount_metric_names_are_documented` — fail-closed on undocumented metrics (catches typos) Failure message names the doc file, line number, and the claimed-vs-live diff. Example: `docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27` == 29 tags installed across 5 docs == While installing tags I had to read the surrounding prose, which surfaced six stale counts that had drifted same-day: `docs/soft-hash-channel-analysis.md`: - L392 14 → 23 tests for phi_alignment_probe - L417 20 → 27 tests for anchor_prg - L463 14 → 23 tests for phi_alignment_probe (status section) `docs/seven-point-program.md`: - L77 68 → 58 tests for metacognition (drift -10; the file shed tests during a refactor and the doc didn't catch up) - L78 9 tests for `test_dag.py::test_preflight_*` — removed count entirely; pytest selector subsets aren't currently supported by the AUTOCOUNT metric set (would need a `tests-matching` metric; not worth the surface for one claim). - L110 24 → 33 tests for test_dag.py `docs/calculator-test-patterns.md`: - L35 33 → 23 tests for warrant_resolver - L35 10 → 9 tests for warrant_chain - L16, L265 51 → 53 tests for t3_bound_calculator (kept initial-shipment provenance in prose) == Coverage installed == calculator-test-patterns.md 3 tagged claims soft-hash-channel-analysis.md 5 tagged claims warrant-substrate-cookbook.md 14 tagged claims seven-point-program.md 3 tagged claims tickets/ticket-000006-bench-... 4 tagged claims --- 29 tagged claims Every count that drifted today is now tagged. Future drift fires the regression test at the next pytest run instead of waiting for human catching. == Discipline pattern == Walk this pattern for any new doc that names a count: 1. Surround the number with the tag pair: `<!--AUTOCOUNT:tests:tests/test_foo.py-->N<!--/AUTOCOUNT-->` 2. Run `pytest tests/test_doc_counts.py` (~3.5s) 3. If it passes, the claim is now machine-verified Aim to tag counts on first authorship. Retrofitting is cheap but only catches drift after the fact. == Out of scope == Test counts inside source code (docstrings, CLI --help) are not scanned — would expand the test surface significantly and the drift pattern hasn't manifested there. Add `**/*.py` scope when that pattern surfaces. Alias-row counts and claim-pack-record counts could be tagged with new `db-rows:<table>` and `db-where:<sql>` metrics; deferred until the next drift on those numbers (none caught today after 30a9488's cookbook refresh). == Verification == $ .venv/bin/pytest tests/test_doc_counts.py -v 3 passed in 3.89s $ .venv/bin/pytest -q 2276 passed, 54 skipped in 153.21s No new dependencies. No schema changes. No source-code changes.
324 lines
16 KiB
Markdown
324 lines
16 KiB
Markdown
# Seven-point program — the north-star directive
|
||
|
||
**Distilled 2026-05-01, Asia/Kuala_Lumpur**:
|
||
|
||
> Stop making Hermes prove things. Make Hermes emit pointer clauses;
|
||
> build CTI internally; bind the retrieval map and evidence map;
|
||
> verify pointers deterministically; add general anchor-class warrant
|
||
> before semantic NLI; rename labels honestly; and automate only after
|
||
> these invariants are test-pinned.
|
||
|
||
This is the architecture's north-star. Every new feature, prompt edit,
|
||
schema change, and ticket walks past these seven directives before
|
||
landing. The discipline frames how `bench/qa_sweep.py` and
|
||
`tests/test_qa_quality_live.py` cover the substrate: the bench is the
|
||
scoreboard, the live fixtures are the gates, and **this doc names what
|
||
the scoreboard and gates measure for**.
|
||
|
||
## The seven directives
|
||
|
||
### D1 — Stop making Hermes prove things
|
||
|
||
The verifier never calls the LLM. Faithfulness classification is a
|
||
deterministic property of (answer text, evidence map, source corpus),
|
||
not a model self-grade.
|
||
|
||
- **Status**: ✓ enforced.
|
||
- **Code anchors**:
|
||
`arborist/qa/verify.py:verify_quotes`,
|
||
`verify_claim_lattice`, `verify_claim_lattice_json` — pure functions
|
||
with no chat-client parameter.
|
||
- **Pinning tests**: `tests/test_verify*.py`,
|
||
`tests/test_claim_lattice.py` — every verifier path runs without a
|
||
network call.
|
||
- **Smell-test**: if a future PR adds a `chat_client` argument to a
|
||
verifier, D1 violated.
|
||
|
||
### D2 — Make Hermes emit pointer clauses
|
||
|
||
The model's prose surface is *citation-style* prose with bracketed
|
||
evidence-id tags (`Claim. [E12]`) or schema-constrained JSON
|
||
(`{"claims":[{"text":"...", "evidence_ids":["E12"]}]}`). The model
|
||
never types the quote string itself.
|
||
|
||
- **Status**: ✓ enforced for lattice modes.
|
||
- **Code anchors**:
|
||
`arborist/qa/parse_claims.py:parse_pointer_claims`,
|
||
`arborist/qa/verify.py:CLAIM_LATTICE_JSON_SCHEMA`,
|
||
`arborist/qa/runner.py` (system prompts).
|
||
- **Pinning tests**: `tests/test_claim_lattice.py`,
|
||
`tests/test_verify_json.py`.
|
||
- **Bench signal**: rows with `answer_mode ∈
|
||
{claim_lattice_pointer, claim_lattice}` fall under this directive;
|
||
rows with `answer_mode == "quote"` predate it (legacy).
|
||
|
||
### D3 — Build CTI internally
|
||
|
||
Clause-tree intelligence (frame lattice + multi-frame answer
|
||
compilation) lives **on our side of the wire**. The model proposes
|
||
clauses; the runtime composes them. Reference-frame queries get
|
||
multiple frames acknowledged in the answer (literal vs
|
||
fictional-actual vs in-universe-propaganda) rather than picking one
|
||
frame and discarding the others.
|
||
|
||
- **Status**: ¾ — phrase-route closed the retrieval side
|
||
(commit `1b8677d`); pre-answer preflight contract (#000008
|
||
quantifier classifier + #000010 metacognition QuestionState)
|
||
landed 2026-05-03. Answer-side multi-frame compilation still
|
||
pending.
|
||
- **Code anchors**: `arborist/qa/quantifier.py:classify_question_quantifier`
|
||
(#000008), `arborist/qa/metacognition.py:preflight_question`
|
||
(#000010), `arborist/qa/dag.py:preflight_node_hash` /
|
||
`build_preflight_node_payload` (#000009 — five nested CTI
|
||
clauses bind into the run-DAG `preflight` stage). Future:
|
||
runtime-side polarity contract emission for multi-frame answers.
|
||
- **Pinning tests**: `tests/test_quantifier_classifier.py`
|
||
(10-rung intensity ladder, <!--AUTOCOUNT:tests:tests/test_quantifier_classifier.py-->70<!--/AUTOCOUNT--> tests), `tests/test_metacognition.py`
|
||
(4 detectors + governance + audit-line tails, <!--AUTOCOUNT:tests:tests/test_metacognition.py-->58<!--/AUTOCOUNT--> tests),
|
||
`tests/test_dag.py::test_preflight_*` (DAG binding; subset of
|
||
the `test_dag.py` suite tagged below).
|
||
Future: per-shape frame-detector tests + multi-frame answer-shape
|
||
live fixtures.
|
||
- **Open ticket**:
|
||
[#000002 Reference-Frame Polarity Contract](ticket-000002-reference-frame-polarity-contract.md).
|
||
- **Bench signal**: lattice-mode bench rows now carry
|
||
`quantifier_intensity`, `quantifier_matched_token`,
|
||
`scope_bound_hint`, `preflight_logical_statuses`,
|
||
`preflight_question_shape`, `preflight_result`. Audit-line tails
|
||
surface five metacog kinds: `· false premise`, `· contradictory`,
|
||
`· stale risk`, `· out of corpus`, `· frame ambiguous`.
|
||
|
||
### D4 — Bind the retrieval map AND the evidence map
|
||
|
||
Provenance binding covers both *what got retrieved* (sources) and
|
||
*how retrieval got there* (the operator-influenceable inputs:
|
||
keywords, top_k, over_fetch, max_context_chars, shard set). Plus
|
||
the **preflight contract** governing the run (#000009 — guard /
|
||
cap / reminder / reject decisions hash-bound).
|
||
|
||
- **Status**: ¾ — `evidence_map_root` lives in the run-DAG
|
||
(`arborist/qa/dag.py`); `retrieval_plan_hash` landed via #000001.
|
||
Preflight stage (#000009) added the upstream control commitment:
|
||
classifier output + answer/prompt/evidence contracts +
|
||
`governance_policy_hash` / `model_profile_hash` references all
|
||
fold into a single `preflight` Merkle leaf. Quote mode now 7→8
|
||
stages; pointer-mode CTI 9→10 stages; reject-broad path emits a
|
||
3-stage minimal DAG (`question → preflight → final_label`).
|
||
- **Code anchors**: `arborist/qa/dag.py:build_run_dag` (legacy +
|
||
preflight-extended shapes), `build_reject_run_dag` (3-stage),
|
||
`preflight_node_hash` / `build_preflight_node_payload` (5-clause
|
||
payload), `arborist/qa/evidence.py:evidence_map_root`.
|
||
- **Pinning tests**: `tests/test_dag.py` (<!--AUTOCOUNT:tests:tests/test_dag.py-->33<!--/AUTOCOUNT--> tests including
|
||
preflight stage + reject path), `tests/test_evidence.py`.
|
||
- **Open ticket**: [#000001 Retrieval-keywords audit gap](ticket-000001-retrieval-keywords-audit-gap.md)
|
||
closed; future retrieval-side refinement still possible.
|
||
- **Bench signal**: every lattice-mode row has `run_dag_root`
|
||
populated. Two cache rows that share the same question + same
|
||
model output + same verifier verdict but different preflight
|
||
policy state now produce different `run_dag_root` values —
|
||
audit replay can pin the policy decision per row.
|
||
|
||
### D5 — Verify pointers deterministically
|
||
|
||
The hard verifier runs only deterministic checks (parser succeeded,
|
||
evidence_id resolves, source_role allowed, claim text non-empty,
|
||
citation coverage, pointer-count cap, lexical anchor warrant).
|
||
No NLI, no embeddings, no model self-grading.
|
||
|
||
- **Status**: ✓ enforced.
|
||
- **Code anchors**: `arborist/qa/verify.py:verify_claim_lattice` —
|
||
seven hard checks documented inline.
|
||
- **Pinning tests**: `tests/test_claim_lattice.py` (per-rule
|
||
coverage), `tests/test_verify_json.py`,
|
||
`tests/test_verify.py`.
|
||
- **Bench signal**: every row carries `verifier_method ∈ {quote,
|
||
span, entity, paraphrase, claim_lattice, none}` (CHECK constraint
|
||
in `arborist/store.py`).
|
||
|
||
### D6 — General anchor-class warrant before semantic NLI
|
||
|
||
The lexical anchor check (Rule 7 in `verify_claim_lattice`) catches
|
||
the lazy-anchor failure where pointer / role / coverage all pass but
|
||
the cited chunk never names the answer entity. Today this fires
|
||
only on relation-shape questions. Generalizing the anchor-class
|
||
detector to entity-list / why-cause / when-date shapes pins a
|
||
broader class of failures structurally before any NLI substrate
|
||
exists in the proof path.
|
||
|
||
- **Status**: ½ — relation-shape only. Entity-list / why-cause /
|
||
when-date shapes pass through unchecked.
|
||
- **Code anchors**:
|
||
`arborist/qa/verify.py:_extract_anchor_candidates`,
|
||
`arborist/qa/verify.py:_question_is_relation_shape`.
|
||
- **Pinning tests**: `tests/test_claim_lattice.py` covers
|
||
relation-shape; entity-list / cause / date shapes have no warrant
|
||
test today.
|
||
- **Open ticket**:
|
||
[#000003 Anchor-class warrant generalization](ticket-000003-anchor-class-warrant.md).
|
||
- **Bench signal**: future column `warrant_fired: bool` (true when
|
||
Rule 7 ran on the row, regardless of pass/fail).
|
||
|
||
### D7 — Rename labels honestly
|
||
|
||
`STRICT` for claim-lattice modes overclaims because synthesis-heavy
|
||
claims pass all hard checks without semantic entailment of the
|
||
joined assertion. The renderer relabels at display time:
|
||
|
||
```
|
||
STRICT → EVIDENCE-LINKED · via claim_lattice
|
||
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer
|
||
UNGROUNDED → UNGROUNDED · via claim_lattice
|
||
```
|
||
|
||
Quote / span / entity / paraphrase modes keep their original tokens
|
||
(they verify against pinned spans, not synthesis). The schema column
|
||
stays `STRICT`/`HYBRID`/`UNGROUNDED` so v9.8 cache_key invariants
|
||
hold.
|
||
|
||
- **Status**: ✓ enforced for lattice modes.
|
||
- **Code anchors**: `arborist/cli.py:_render_audit_label`.
|
||
- **Pinning tests**: `tests/test_cli_render.py` —
|
||
`test_render_label_strict_in_claim_lattice_becomes_evidence_linked`
|
||
and siblings.
|
||
|
||
### D8 — Automate only after the invariants are test-pinned
|
||
|
||
The five-step algorithm in CLAUDE.md applies to automation in
|
||
particular: bench harnesses, distill runners, ingest CLIs all earned
|
||
their automation only after the underlying ops were simplified into
|
||
single-purpose stages. **The bench harness IS the automation
|
||
substrate**, so it must enforce the directives before any feature
|
||
gets layered on top.
|
||
|
||
- **Status**: ongoing discipline.
|
||
- **Code anchors**: `bench/qa_sweep.py`,
|
||
`tests/test_qa_quality_live.py`.
|
||
- **Bench signal**: per-row `directive_compliance` column maps
|
||
each directive to a pass/fail boolean; the markdown summary
|
||
aggregates per-mode-per-directive coverage.
|
||
|
||
## How to use this doc
|
||
|
||
1. **Before opening a new ticket**, scan the seven directives. Which
|
||
one does the work serve? Add the directive number to the ticket
|
||
header (`Directive: D6`) so the design log threads back here.
|
||
|
||
2. **Before merging a feature**, audit it against D8: does a test
|
||
pin the new invariant? If not, the feature is automation
|
||
pretending to be code-level discipline.
|
||
|
||
3. **When reading a bench summary**, the per-directive coverage row
|
||
tells you which directives are passing on the corpus today and
|
||
which are pending implementation. Bench scores that climb without
|
||
directive coverage climbing are graveyard-digging (step 4 of
|
||
the five-step algorithm).
|
||
|
||
4. **When triaging a fixture failure**, identify which directive
|
||
the fixture pins. A fixture that fails by name maps to a specific
|
||
regression in the directive's structural invariant; an unnamed
|
||
bench delta is just noise.
|
||
|
||
## Status snapshot at landing
|
||
|
||
| # | Directive | Status | Tickets |
|
||
|---|----------------------------------------------|------------|-------------------------------|
|
||
| 1 | Stop making Hermes prove things | ✓ | #000010 (closed) |
|
||
| 2 | Hermes emits pointer clauses | ✓ | |
|
||
| 3 | Build CTI internally | ✓ | #000002, #000008, #000010 (closed) |
|
||
| 4 | Bind retrieval map AND evidence map | ✓ | #000001, #000009 (closed) |
|
||
| 5 | Verify pointers deterministically | ✓ | |
|
||
| 6 | Anchor-class warrant before NLI | ✓ | #000003 (closed) |
|
||
| 7 | Rename labels honestly | ✓ | #000005 (closed) |
|
||
| 8 | Automate only after test-pinning | discipline | #000004 (closed) |
|
||
|
||
All seven structural directives now ✓:
|
||
|
||
D1, D2, D5 — were ✓ at landing.
|
||
D3 closed 2026-05-02 via #000002 (frame detector + polarity
|
||
preamble + renderer extension; multi-frame answers on Orwell
|
||
and other reference-frame queries).
|
||
D4 closed 2026-05-02 via #000001 (retrieval_plan_hash binding
|
||
in the run-DAG; SQL column + audit events deferred).
|
||
D6 closed 2026-05-02 via #000003 (anchor-class warrant
|
||
generalization: entity-list / count / why-cause shapes).
|
||
D7 closed 2026-05-02 via #000005 (four-rung ladder:
|
||
POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED →
|
||
ENTAILMENT-VERIFIED).
|
||
|
||
D8 is the meta-discipline that gates how all of the above land;
|
||
the bench coverage substrate (#000004) is closed. Tests: 734
|
||
passed at the original snapshot.
|
||
|
||
### Post-landing addendum (2026-05-03 / 2026-05-04)
|
||
|
||
Three additional tickets landed extending D1, D3, and D4:
|
||
|
||
- **#000008** (closed `4f2b5a6`, 2026-05-03) — Broad-quantifier
|
||
preflight guard. New modules `arborist/qa/quantifier.py`,
|
||
`model_profiles.py`, `quantifier_reminder.py`. Seven new
|
||
policy fields fold into `governance_policy_hash`; six-level
|
||
disable hierarchy. Defaults flipped (after n=5 verification):
|
||
reminder default-on for lattice modes, cap default operator-
|
||
opt-in with JSON-only allowlist.
|
||
|
||
- **#000010** (closed, 2026-05-03) — Meta-Cognition Preflight
|
||
Guard. New module `arborist/qa/metacognition.py` with
|
||
`QuestionState` dataclass + four pure detectors (temporal,
|
||
contradiction, false-premise-lite, out-of-corpus). Six new
|
||
policy fields, all default-on except
|
||
`metacognition_block_on_contradiction`. Audit-line tail
|
||
tokens: `· false premise`, `· contradictory`, `· stale risk`,
|
||
`· out of corpus`, `· frame ambiguous`. Closes D1 by giving
|
||
the runtime explicit epistemic state to refuse from instead
|
||
of asking Hermes to self-regulate.
|
||
|
||
- **#000009** (closed `111dda6`, 2026-05-04) — Preflight run-DAG
|
||
node binding. Inserts `preflight` stage between `question` and
|
||
`retrieval`. Quote mode 7→8 stages; pointer-mode CTI 9→10
|
||
stages. Reject-broad early-return path emits a 3-stage minimal
|
||
DAG (`question → preflight → final_label`). Five nested CTI
|
||
clauses (classifier / answer_contract / prompt_contract /
|
||
evidence_contract / policy_refs) inside a single hashable
|
||
leaf. Closes D4 by Merkle-binding the upstream control
|
||
decisions that previously only appeared on result dicts.
|
||
|
||
Test count after these landings: **993 passing** (up from 734;
|
||
259 new across the three tickets). All seven directives remain ✓.
|
||
|
||
**Bench-validated post-landing** (full 75-question sweep
|
||
2026-05-03T23-30-12Z vs frozen 2026-05-02T15-07Z baseline,
|
||
n=3 × 3 modes = 675 runs):
|
||
|
||
- quote SR 0.54 → 0.52 (−2pp, within noise)
|
||
- pointer SR 0.20 → 0.21 (+1pp, within noise)
|
||
- JSON SR 0.42 → 0.44 (+2pp, within noise)
|
||
|
||
All STRICT-rate deltas within the 5pp signal floor. Substrate-
|
||
level wins: pointer-mode FORMAT_COLLAPSED eliminated globally
|
||
(0/225), pointer-mode NO_EVIDENCE_POINTER drops from 33%
|
||
(broad-only when reminder was off) to 13% (full sweep with
|
||
reminder on). The default flip is strictly more honest without
|
||
regressing throughput on narrow questions. See
|
||
`docs/qa-modes-bench.md` Addendum 3 for the full comparison.
|
||
|
||
## Anti-regression test layer
|
||
|
||
Each directive has a structural pin in `tests/test_directives.py`.
|
||
A future PR that silently weakens a directive fails by name:
|
||
|
||
| Directive | Pin tests in `tests/test_directives.py` |
|
||
|-----------|---------------------------------------------------------------------------|
|
||
| D1 | `test_d1_verify_*_signature_has_no_chat_client`, `test_d1_verifier_method_enum_excludes_llm_judges` |
|
||
| D2 | `test_d2_answer_modes_include_lattice_variants`, `test_d2_pointer_parser_exists_*` |
|
||
| D3 | `test_d3_runtime_owns_claim_lattice_construction`, `test_d3_evidence_map_built_*` |
|
||
| D4 | `test_d4_run_dag_carries_evidence_map_root` (✓), `test_d4_retrieval_plan_binding_status` (marker — flips when #000001 lands) |
|
||
| D5 | `test_d5_verify_claim_lattice_is_deterministic`, `test_d5_seven_hard_checks_are_pure_functions` |
|
||
| D6 | `test_d6_warrant_*` (relation + date), `test_d6_warrant_generalization_status` (marker — flips when #000003 lands) |
|
||
| D7 | `test_d7_renderer_*`, `test_d7_audit_mode_enum_canonical_set` |
|
||
| D8 | `test_d8_seven_point_program_doc_exists`, `test_d8_bench_directive_compliance_helper_exists`, `test_d8_tickets_index_exists_and_pins_open_directives` |
|
||
|
||
The "marker" tests for D4 and D6 document open work: today they
|
||
assert that a parameter / detector is **absent**; when the
|
||
corresponding ticket lands, the assertion flips to **present**. The
|
||
test failing during ticket implementation is the expected signal,
|
||
not a regression.
|