diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 9cb291d..a4d5a8b 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -53,12 +53,14 @@ implementation cost, it's a ticket. Otherwise it's reference. Newest first. Update on every open/close. -| ID | Title | Status | Opened | Directive | -|----------|------------------------------------------------|---------|------------|-----------| -| #000003 | Anchor-class warrant generalization (Module H+)| open | 2026-05-01 | D6 | -| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | D3 | -| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | D4 | +| ID | Title | Status | Opened | Directive | +|----------|------------------------------------------------|-----------------------|------------|-----------| +| #000005 | Label ladder migration (POINTER-LINKED → …) | open | 2026-05-01 | D7 | +| #000004 | Directive coverage in bench summary | closed · `acd1f9c` | 2026-05-01 | D8 | +| #000003 | Anchor-class warrant generalization (Module H+)| open | 2026-05-01 | D6 | +| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | D3 | +| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | D4 | ## Next ID -`000004` +`000006` diff --git a/docs/seven-point-program.md b/docs/seven-point-program.md index 63c1d86..e9a59df 100644 --- a/docs/seven-point-program.md +++ b/docs/seven-point-program.md @@ -197,17 +197,43 @@ gets layered on top. ## Status snapshot at landing -| # | Directive | Status | Tickets | -|---|----------------------------------------------|--------|-----------| -| 1 | Stop making Hermes prove things | ✓ | | -| 2 | Hermes emits pointer clauses | ✓ | | -| 3 | Build CTI internally | ½ | #000002 | -| 4 | Bind retrieval map AND evidence map | ½ | #000001 | -| 5 | Verify pointers deterministically | ✓ | | -| 6 | Anchor-class warrant before NLI | ½ | #000003 | -| 7 | Rename labels honestly | ✓ | | -| 8 | Automate only after test-pinning | discipline | | +| # | Directive | Status | Tickets | +|---|----------------------------------------------|------------|-----------------| +| 1 | Stop making Hermes prove things | ✓ | | +| 2 | Hermes emits pointer clauses | ✓ | | +| 3 | Build CTI internally | ½ | #000002 | +| 4 | Bind retrieval map AND evidence map | ½ | #000001 | +| 5 | Verify pointers deterministically | ✓ | | +| 6 | Anchor-class warrant before NLI | ½ | #000003 | +| 7 | Rename labels honestly | ✓ | #000005 (ladder)| +| 8 | Automate only after test-pinning | discipline | #000004 (closed)| Three of seven structural directives are partial (D3, D4, D6). All -three have open tickets. D8 is the meta-discipline that gates how -all of the above land. +three have open tickets. D7 is shipped at the EVIDENCE-LINKED rung; +ticket #000005 proposes the four-rung ladder migration +(POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → +ENTAILMENT-VERIFIED) for stronger label discipline. D8 is the +meta-discipline that gates how all of the above land; the bench +coverage substrate (#000004) is closed. + +## Anti-regression test layer + +Each directive has a structural pin in `tests/test_directives.py`. +A future PR that silently weakens a directive fails by name: + +| Directive | Pin tests in `tests/test_directives.py` | +|-----------|---------------------------------------------------------------------------| +| D1 | `test_d1_verify_*_signature_has_no_chat_client`, `test_d1_verifier_method_enum_excludes_llm_judges` | +| D2 | `test_d2_answer_modes_include_lattice_variants`, `test_d2_pointer_parser_exists_*` | +| D3 | `test_d3_runtime_owns_claim_lattice_construction`, `test_d3_evidence_map_built_*` | +| D4 | `test_d4_run_dag_carries_evidence_map_root` (✓), `test_d4_retrieval_plan_binding_status` (marker — flips when #000001 lands) | +| D5 | `test_d5_verify_claim_lattice_is_deterministic`, `test_d5_seven_hard_checks_are_pure_functions` | +| D6 | `test_d6_warrant_*` (relation + date), `test_d6_warrant_generalization_status` (marker — flips when #000003 lands) | +| D7 | `test_d7_renderer_*`, `test_d7_audit_mode_enum_canonical_set` | +| D8 | `test_d8_seven_point_program_doc_exists`, `test_d8_bench_directive_compliance_helper_exists`, `test_d8_tickets_index_exists_and_pins_open_directives` | + +The "marker" tests for D4 and D6 document open work: today they +assert that a parameter / detector is **absent**; when the +corresponding ticket lands, the assertion flips to **present**. The +test failing during ticket implementation is the expected signal, +not a regression. diff --git a/docs/ticket-000004-directive-coverage-in-bench.md b/docs/ticket-000004-directive-coverage-in-bench.md new file mode 100644 index 0000000..263e103 --- /dev/null +++ b/docs/ticket-000004-directive-coverage-in-bench.md @@ -0,0 +1,77 @@ +# Ticket #000004 — Directive coverage in bench summary + +**Status:** closed · landed in commit `acd1f9c` (2026-05-01) +**Opened:** 2026-05-01 +**Closed:** 2026-05-01 +**Directive:** [D8](seven-point-program.md) — Automate only after the +invariants are test-pinned. Directive coverage in the bench output is +the substrate that makes D1-D7 enforcement visible per-run. +**Scope:** Wire the seven-point program into `bench/qa_sweep.py` so +each row carries a `directive_compliance` dict and the markdown +summary emits a per-mode coverage table. +**Audience:** fox + future blackops shifts. +**Hard constraint:** the bench output never edits the run-DAG, the +audit chain, or the providence_cache. Directive compliance lives +alongside the existing per-row signals (deflection, capacity, +strict-rate); never feeds back into the proof path. + +--- + +## Why this ticket exists + +The seven-point program is governance prose unless the bench +enforces it per-run. Without per-row directive compliance, an +operator can't tell whether a strict-rate climb came from real +substrate work or from accidentally weakening one of the directives. + +Per the five-step algorithm step 5 ("automate only after test- +pinning"): the bench harness IS the automation substrate, so it +must enforce the directives before any feature gets layered on top. + +## Implementation summary (landed `acd1f9c`) + +`bench/qa_sweep.py`: + +- `_directive_compliance(answer_mode, result, err)` — pure helper + computing per-row pass/fail for the directives whose pinning is + observable from a single bench row (D2, D3, D4, D6, D7). D1, D5, + D8 are global properties of the substrate; tracked once in the + program doc rather than per-row. +- Each `_run_one` result row gains a `directive_compliance: dict` + field. +- `_summarize` aggregates per-row booleans into per-mode + `directive_pass: dict[str, int]` counts. +- `_render_markdown` emits a `## directive coverage (seven-point + program)` section with a per-mode per-directive table showing + `count/N (pct%)`. + +Per-row checks: + +| ID | Check | +|----|---------------------------------------------------------------------| +| D2 | `answer_mode in ("claim_lattice_pointer", "claim_lattice")` | +| D3 | lattice mode + `run_dag_root` populated (full Module L pending #000002) | +| D4 | `run_dag_root` populated (retrieval-plan hash pending #000001) | +| D6 | warrant ran (lattice mode today; per-shape gating pending #000003) | +| D7 | `audit_mode in {STRICT, HYBRID, UNGROUNDED}` (renderer pinned in `test_cli_render.py`) | + +## Tests + +5 tests in `tests/test_bench_qa_sweep.py`: + +- `test_summarize_aggregates_directive_pass_counts_per_mode` +- `test_render_markdown_directive_coverage_section` +- `test_directive_compliance_helper_marks_quote_mode_d2_false` +- `test_directive_compliance_helper_marks_lattice_mode_d2_true` +- `test_directive_compliance_returns_empty_on_error_row` + +## Closure rationale + +Filed as a closed ticket immediately so the design log captures the +artifact alongside the work. The convention from `docs/TICKETS.md`: +"Closed tickets stay in place as the design log; do not delete +tickets — they are the design log." + +Future evolutions (per-row D1/D5 tracking, error-row directive +weighting, cross-mode comparison) get their own tickets. This one +documents the v1 substrate. diff --git a/docs/ticket-000005-label-ladder-migration.md b/docs/ticket-000005-label-ladder-migration.md new file mode 100644 index 0000000..0ad842b --- /dev/null +++ b/docs/ticket-000005-label-ladder-migration.md @@ -0,0 +1,165 @@ +# Ticket #000005 — Label ladder migration: POINTER-LINKED / ANCHOR-WARRANTED / EVIDENCE-WARRANTED + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-01 +**Directive:** [D7](seven-point-program.md) — Rename labels honestly. +**Scope:** Replace today's two-label rendered surface +(`EVIDENCE-LINKED` / `EVIDENCE-LINKED-PARTIAL`) with a four-rung +ladder that names exactly what each verifier path proved. Schema +column stays `STRICT` / `HYBRID` / `UNGROUNDED` so the v9.8 +cache_key invariant holds; only the rendered surface changes. +**Audience:** fox + future blackops shifts. +**Hard constraint:** schema column stays unchanged. Renderer-level +mapping only. The audit chain, mesh wire format, providence_cache, +and all programmatic callers continue to see the original +`audit_mode` enum. + +--- + +## 1. Problem statement + +Today's renderer relabels claim-lattice modes: + +``` +STRICT → EVIDENCE-LINKED · via claim_lattice +HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer +UNGROUNDED → UNGROUNDED · via claim_lattice +``` + +`EVIDENCE-LINKED` improves on `STRICT` (which overclaims semantic +truth) but still reads as "the evidence supports the claim." A +careful operator parses it as "pointer + role + coverage checks +passed." A casual one reads it as "the model got this right." + +The four-rung ladder fox proposed (2026-05-01) carries the +distinction explicitly: + +``` +POINTER-LINKED pointer/source/chunk relation verified +ANCHOR-WARRANTED pointer-linked AND cited evidence contains + required claim anchors (Rule 7 + future + entity/count/cause classes from #000003) +EVIDENCE-WARRANTED anchor-warranted AND no hard violations +ENTAILMENT-VERIFIED reserved for a future committed entailment + engine (NLI substrate). Not reachable today. +``` + +Each rung names a strictly stronger property. An operator reading +the rung knows what's been proved without inferring from the +verifier_method tail. + +## 2. Mapping from current state to ladder + +Today's verifier output gets re-mapped at the renderer layer: + +| Verifier method | Today's render | New render | +|------------------|-------------------------|--------------------------| +| `claim_lattice` | EVIDENCE-LINKED · via … | POINTER-LINKED (no warrant ran), ANCHOR-WARRANTED (warrant fired & passed), EVIDENCE-WARRANTED (anchor-warranted + zero soft-demote violations) | +| `claim_lattice` (HYBRID) | EVIDENCE-LINKED-PARTIAL | POINTER-LINKED-PARTIAL or ANCHOR-WARRANTED-PARTIAL depending on which checks demoted | +| `claim_lattice` (UNGROUNDED) | UNGROUNDED | UNGROUNDED (unchanged) | +| `quote` / `span` / `entity` / `paraphrase` | STRICT/HYBRID/UNGROUNDED | unchanged (these verify against pinned spans, not synthesis) | + +Mapping rule (claim-lattice modes only): + +```python +def _ladder_label(audit_mode, violations) -> str: + if audit_mode == "UNGROUNDED": + return "UNGROUNDED" + has_warrant_check = any(v["kind"] == "WARRANT_RAN" for v in violations) + has_warrant_miss = any(v["kind"] == "WARRANT_MISSING" for v in violations) + has_other_demote = any(v["kind"] in DEMOTE_KINDS for v in violations) + if not has_warrant_check: + rung = "POINTER-LINKED" + elif has_warrant_miss: + rung = "POINTER-LINKED" # warrant ran AND failed + elif has_other_demote: + rung = "ANCHOR-WARRANTED" # warrant ok but other demotes + else: + rung = "EVIDENCE-WARRANTED" # warrant ok AND no demotes + if audit_mode == "HYBRID": + rung = rung + "-PARTIAL" + return rung +``` + +## 3. New verifier signal: WARRANT_RAN + +For the renderer to know whether the warrant fired, the verifier +needs to surface that signal. Today's `warrant_check` returns +`(ok, missing)`; only `WARRANT_MISSING` enters the violations list +when `ok=False`. Add a benign `WARRANT_RAN` violation entry (or +a separate `verifier_steps_ran: list[str]` field) so the renderer +can distinguish "warrant didn't apply" from "warrant applied and +passed." + +This is a renderer-input enhancement, not a verifier semantic +change. The hard checks stay binary; we just thread the +"which checks ran" provenance up to the display layer. + +## 4. Cache implications + +None. Schema `audit_mode` enum is unchanged; the renderer mapping +runs on read, not on write. Existing providence records continue +to render under the new ladder without re-classification or cache +invalidation. + +`governance_policy_hash` does NOT include the ladder mapping — +it's a pure presentation transformation, not a behavioral change. +Two records with the same cache_key written before and after the +ladder migration render under the new ladder identically. + +## 5. Implementation sketch + +1. **Verifier enhancement** (`aborist/qa/verify.py`): both + `verify_claim_lattice` and `verify_claim_lattice_json` add a + `verifier_steps_ran: list[str]` field on the verdict dict + listing which verifier paths fired (`pointer_resolve`, + `source_role`, `coverage`, `pointer_cap`, `warrant`, + `lazy_anchor_demote`). +2. **Renderer mapping** (`aborist/cli.py:_render_audit_label`): + add the four-rung mapping function gated on + `verifier_method.startswith("claim_lattice")`. +3. **Bench update** (`bench/qa_sweep.py`): per-row + `ladder_rung` field; markdown summary gets a "ladder coverage" + table alongside the existing strict-rate breakdown. +4. **Live fixtures** (`tests/test_qa_quality_live.py`): the + existing `test_mona_lisa_strict_*` and siblings get their + assertion strings updated to the new rung names. + +## 6. Tests required + +- `[ ]` Renderer maps verifier output to POINTER-LINKED when no + warrant ran. +- `[ ]` Renderer maps to ANCHOR-WARRANTED when warrant ran & + passed but other demotes fired. +- `[ ]` Renderer maps to EVIDENCE-WARRANTED when warrant passed + & no demotes. +- `[ ]` Renderer suffix `-PARTIAL` appears for HYBRID + each rung. +- `[ ]` Renderer keeps STRICT/HYBRID/UNGROUNDED for quote-mode + verifier paths (no ladder rename). +- `[ ]` Verifier verdict dict carries `verifier_steps_ran` list. +- `[ ]` Live fixtures updated to gate on rung names. +- `[ ]` Cache lookup returns identical results before & after + migration (governance_policy_hash unchanged). + +## 7. Out of scope + +- **`ENTAILMENT-VERIFIED` rung.** Reserved for a future committed + entailment engine. Not reachable today; no code path produces it. + Naming the rung today preserves the design space without forcing + premature implementation. +- **Schema column rename.** Stays `audit_mode ∈ {STRICT, HYBRID, + UNGROUNDED}`. v9.8 cache_key invariants depend on this; never + break. +- **Mesh wire format change.** Mesh peers exchange `audit_mode`, + not the rendered rung. No wire-protocol bump required. + +## 8. Status + +**Proposal.** Forecast cost: ~2 hours (verifier signal + renderer +mapping + ~8 tests + live-fixture migration). Risk: low — pure +display layer; schema and audit chain stay unchanged. + +Forecast value: closes Directive D7 in stronger form. An operator +reading `EVIDENCE-WARRANTED` on a record knows the answer is +pointer-linked AND anchor-warranted AND no soft-demotes fired — +strictly more information than `EVIDENCE-LINKED` carried.