engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: open #000004 (closed/retroactive) + #000005; update program status

Browse source 
#000004 — Directive coverage in bench summary. Filed as closed
(landed in commit acd1f9c) for design-log traceability. Captures
the v1 substrate: per-row directive_compliance helper, _summarize
aggregation, markdown directive-coverage section, 5 unit tests.

#000005 — Label ladder migration (POINTER-LINKED / ANCHOR-WARRANTED
/ EVIDENCE-WARRANTED / ENTAILMENT-VERIFIED). Status open. Migration
from today's two-rung EVIDENCE-LINKED to a four-rung ladder where
each rung names a strictly stronger property. Schema column stays
{STRICT, HYBRID, UNGROUNDED} so v9.8 cache_key invariants hold;
renderer-level mapping only. Reserves ENTAILMENT-VERIFIED for a
future committed entailment engine without forcing premature
implementation.

TICKETS.md index gains both rows. Next ID bumps to 000006.

seven-point-program.md status snapshot updated:
  - D7 row links #000005 (ladder migration proposal).
  - D8 row links #000004 (closed substrate).
  - New "Anti-regression test layer" section maps each directive
    to its structural pin in tests/test_directives.py.

Note on fox's roadmap item #000006 (Internal CTI Expansion / frame
lattice / multi-frame answers): scope overlaps with #000002
(Reference-Frame Polarity Contract / Module L). NOT opening as
a separate ticket — Module L already covers it.
This commit is contained in:
russell@unturf.com 2026-05-01 16:31:44 -04:00
parent acd1f9ca84
commit 6db5d1dc64
No known key found for this signature in database
4 changed files with 288 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

14
docs/TICKETS.md
View file

@ -53,12 +53,14 @@ implementation cost, it's a ticket. Otherwise it's reference.
Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|---------|------------|-----------|
| #000003 | Anchor-class warrant generalization (Module H+)| open | 2026-05-01 | D6 |
| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | D3 |
| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | D4 |
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000005 | Label ladder migration (POINTER-LINKED → …) | open | 2026-05-01 | D7 |
| #000004 | Directive coverage in bench summary | closed · `acd1f9c` | 2026-05-01 | D8 |
| #000003 | Anchor-class warrant generalization (Module H+)| open | 2026-05-01 | D6 |
| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 | D3 |
| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 | D4 |
## Next ID
`000004`
`000006`

50
docs/seven-point-program.md
View file

@ -197,17 +197,43 @@ gets layered on top.
## Status snapshot at landing
| # | Directive | Status | Tickets |
|---|----------------------------------------------|--------|-----------|
| 1 | Stop making Hermes prove things | ✓ | |
| 2 | Hermes emits pointer clauses | ✓ | |
| 3 | Build CTI internally | ½ | #000002 |
| 4 | Bind retrieval map AND evidence map | ½ | #000001 |
| 5 | Verify pointers deterministically | ✓ | |
| 6 | Anchor-class warrant before NLI | ½ | #000003 |
| 7 | Rename labels honestly | ✓ | |
| 8 | Automate only after test-pinning | discipline | |
| # | Directive | Status | Tickets |
|---|----------------------------------------------|------------|-----------------|
| 1 | Stop making Hermes prove things | ✓ | |
| 2 | Hermes emits pointer clauses | ✓ | |
| 3 | Build CTI internally | ½ | #000002 |
| 4 | Bind retrieval map AND evidence map | ½ | #000001 |
| 5 | Verify pointers deterministically | ✓ | |
| 6 | Anchor-class warrant before NLI | ½ | #000003 |
| 7 | Rename labels honestly | ✓ | #000005 (ladder)|
| 8 | Automate only after test-pinning | discipline | #000004 (closed)|
Three of seven structural directives are partial (D3, D4, D6). All
three have open tickets. D8 is the meta-discipline that gates how
all of the above land.
three have open tickets. D7 is shipped at the EVIDENCE-LINKED rung;
ticket #000005 proposes the four-rung ladder migration
(POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED →
ENTAILMENT-VERIFIED) for stronger label discipline. D8 is the
meta-discipline that gates how all of the above land; the bench
coverage substrate (#000004) is closed.
## Anti-regression test layer
Each directive has a structural pin in `tests/test_directives.py`.
A future PR that silently weakens a directive fails by name:
| Directive | Pin tests in `tests/test_directives.py` |
|-----------|---------------------------------------------------------------------------|
| D1 | `test_d1_verify_*_signature_has_no_chat_client`, `test_d1_verifier_method_enum_excludes_llm_judges` |
| D2 | `test_d2_answer_modes_include_lattice_variants`, `test_d2_pointer_parser_exists_*` |
| D3 | `test_d3_runtime_owns_claim_lattice_construction`, `test_d3_evidence_map_built_*` |
| D4 | `test_d4_run_dag_carries_evidence_map_root` (✓), `test_d4_retrieval_plan_binding_status` (marker — flips when #000001 lands) |
| D5 | `test_d5_verify_claim_lattice_is_deterministic`, `test_d5_seven_hard_checks_are_pure_functions` |
| D6 | `test_d6_warrant_*` (relation + date), `test_d6_warrant_generalization_status` (marker — flips when #000003 lands) |
| D7 | `test_d7_renderer_*`, `test_d7_audit_mode_enum_canonical_set` |
| D8 | `test_d8_seven_point_program_doc_exists`, `test_d8_bench_directive_compliance_helper_exists`, `test_d8_tickets_index_exists_and_pins_open_directives` |
The "marker" tests for D4 and D6 document open work: today they
assert that a parameter / detector is **absent**; when the
corresponding ticket lands, the assertion flips to **present**. The
test failing during ticket implementation is the expected signal,
not a regression.

77
docs/ticket-000004-directive-coverage-in-bench.md Normal file
View file

@ -0,0 +1,77 @@
# Ticket #000004 — Directive coverage in bench summary
**Status:** closed · landed in commit `acd1f9c` (2026-05-01)
**Opened:** 2026-05-01
**Closed:** 2026-05-01
**Directive:** [D8](seven-point-program.md) — Automate only after the
invariants are test-pinned. Directive coverage in the bench output is
the substrate that makes D1-D7 enforcement visible per-run.
**Scope:** Wire the seven-point program into `bench/qa_sweep.py` so
each row carries a `directive_compliance` dict and the markdown
summary emits a per-mode coverage table.
**Audience:** fox + future blackops shifts.
**Hard constraint:** the bench output never edits the run-DAG, the
audit chain, or the providence_cache. Directive compliance lives
alongside the existing per-row signals (deflection, capacity,
strict-rate); never feeds back into the proof path.
---
## Why this ticket exists
The seven-point program is governance prose unless the bench
enforces it per-run. Without per-row directive compliance, an
operator can't tell whether a strict-rate climb came from real
substrate work or from accidentally weakening one of the directives.
Per the five-step algorithm step 5 ("automate only after test-
pinning"): the bench harness IS the automation substrate, so it
must enforce the directives before any feature gets layered on top.
## Implementation summary (landed `acd1f9c`)
`bench/qa_sweep.py`:
- `_directive_compliance(answer_mode, result, err)` — pure helper
computing per-row pass/fail for the directives whose pinning is
observable from a single bench row (D2, D3, D4, D6, D7). D1, D5,
D8 are global properties of the substrate; tracked once in the
program doc rather than per-row.
- Each `_run_one` result row gains a `directive_compliance: dict`
field.
- `_summarize` aggregates per-row booleans into per-mode
`directive_pass: dict[str, int]` counts.
- `_render_markdown` emits a `## directive coverage (seven-point
program)` section with a per-mode per-directive table showing
`count/N (pct%)`.
Per-row checks:
| ID | Check |
|----|---------------------------------------------------------------------|
| D2 | `answer_mode in ("claim_lattice_pointer", "claim_lattice")` |
| D3 | lattice mode + `run_dag_root` populated (full Module L pending #000002) |
| D4 | `run_dag_root` populated (retrieval-plan hash pending #000001) |
| D6 | warrant ran (lattice mode today; per-shape gating pending #000003) |
| D7 | `audit_mode in {STRICT, HYBRID, UNGROUNDED}` (renderer pinned in `test_cli_render.py`) |
## Tests
5 tests in `tests/test_bench_qa_sweep.py`:
- `test_summarize_aggregates_directive_pass_counts_per_mode`
- `test_render_markdown_directive_coverage_section`
- `test_directive_compliance_helper_marks_quote_mode_d2_false`
- `test_directive_compliance_helper_marks_lattice_mode_d2_true`
- `test_directive_compliance_returns_empty_on_error_row`
## Closure rationale
Filed as a closed ticket immediately so the design log captures the
artifact alongside the work. The convention from `docs/TICKETS.md`:
"Closed tickets stay in place as the design log; do not delete
tickets — they are the design log."
Future evolutions (per-row D1/D5 tracking, error-row directive
weighting, cross-mode comparison) get their own tickets. This one
documents the v1 substrate.

165
docs/ticket-000005-label-ladder-migration.md Normal file
View file

@ -0,0 +1,165 @@
# Ticket #000005 — Label ladder migration: POINTER-LINKED / ANCHOR-WARRANTED / EVIDENCE-WARRANTED
**Status:** open · awaiting go/no-go
**Opened:** 2026-05-01
**Directive:** [D7](seven-point-program.md) — Rename labels honestly.
**Scope:** Replace today's two-label rendered surface
(`EVIDENCE-LINKED` / `EVIDENCE-LINKED-PARTIAL`) with a four-rung
ladder that names exactly what each verifier path proved. Schema
column stays `STRICT` / `HYBRID` / `UNGROUNDED` so the v9.8
cache_key invariant holds; only the rendered surface changes.
**Audience:** fox + future blackops shifts.
**Hard constraint:** schema column stays unchanged. Renderer-level
mapping only. The audit chain, mesh wire format, providence_cache,
and all programmatic callers continue to see the original
`audit_mode` enum.
---
## 1. Problem statement
Today's renderer relabels claim-lattice modes:
```
STRICT → EVIDENCE-LINKED · via claim_lattice
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer
UNGROUNDED → UNGROUNDED · via claim_lattice
```
`EVIDENCE-LINKED` improves on `STRICT` (which overclaims semantic
truth) but still reads as "the evidence supports the claim." A
careful operator parses it as "pointer + role + coverage checks
passed." A casual one reads it as "the model got this right."
The four-rung ladder fox proposed (2026-05-01) carries the
distinction explicitly:
```
POINTER-LINKED pointer/source/chunk relation verified
ANCHOR-WARRANTED pointer-linked AND cited evidence contains
required claim anchors (Rule 7 + future
entity/count/cause classes from #000003)
EVIDENCE-WARRANTED anchor-warranted AND no hard violations
ENTAILMENT-VERIFIED reserved for a future committed entailment
engine (NLI substrate). Not reachable today.
```
Each rung names a strictly stronger property. An operator reading
the rung knows what's been proved without inferring from the
verifier_method tail.
## 2. Mapping from current state to ladder
Today's verifier output gets re-mapped at the renderer layer:
| Verifier method | Today's render | New render |
|------------------|-------------------------|--------------------------|
| `claim_lattice` | EVIDENCE-LINKED · via … | POINTER-LINKED (no warrant ran), ANCHOR-WARRANTED (warrant fired & passed), EVIDENCE-WARRANTED (anchor-warranted + zero soft-demote violations) |
| `claim_lattice` (HYBRID) | EVIDENCE-LINKED-PARTIAL | POINTER-LINKED-PARTIAL or ANCHOR-WARRANTED-PARTIAL depending on which checks demoted |
| `claim_lattice` (UNGROUNDED) | UNGROUNDED | UNGROUNDED (unchanged) |
| `quote` / `span` / `entity` / `paraphrase` | STRICT/HYBRID/UNGROUNDED | unchanged (these verify against pinned spans, not synthesis) |
Mapping rule (claim-lattice modes only):
```python
def _ladder_label(audit_mode, violations) -> str:
if audit_mode == "UNGROUNDED":
return "UNGROUNDED"
has_warrant_check = any(v["kind"] == "WARRANT_RAN" for v in violations)
has_warrant_miss = any(v["kind"] == "WARRANT_MISSING" for v in violations)
has_other_demote = any(v["kind"] in DEMOTE_KINDS for v in violations)
if not has_warrant_check:
rung = "POINTER-LINKED"
elif has_warrant_miss:
rung = "POINTER-LINKED" # warrant ran AND failed
elif has_other_demote:
rung = "ANCHOR-WARRANTED" # warrant ok but other demotes
else:
rung = "EVIDENCE-WARRANTED" # warrant ok AND no demotes
if audit_mode == "HYBRID":
rung = rung + "-PARTIAL"
return rung
```
## 3. New verifier signal: WARRANT_RAN
For the renderer to know whether the warrant fired, the verifier
needs to surface that signal. Today's `warrant_check` returns
`(ok, missing)`; only `WARRANT_MISSING` enters the violations list
when `ok=False`. Add a benign `WARRANT_RAN` violation entry (or
a separate `verifier_steps_ran: list[str]` field) so the renderer
can distinguish "warrant didn't apply" from "warrant applied and
passed."
This is a renderer-input enhancement, not a verifier semantic
change. The hard checks stay binary; we just thread the
"which checks ran" provenance up to the display layer.
## 4. Cache implications
None. Schema `audit_mode` enum is unchanged; the renderer mapping
runs on read, not on write. Existing providence records continue
to render under the new ladder without re-classification or cache
invalidation.
`governance_policy_hash` does NOT include the ladder mapping —
it's a pure presentation transformation, not a behavioral change.
Two records with the same cache_key written before and after the
ladder migration render under the new ladder identically.
## 5. Implementation sketch
1. **Verifier enhancement** (`aborist/qa/verify.py`): both
`verify_claim_lattice` and `verify_claim_lattice_json` add a
`verifier_steps_ran: list[str]` field on the verdict dict
listing which verifier paths fired (`pointer_resolve`,
`source_role`, `coverage`, `pointer_cap`, `warrant`,
`lazy_anchor_demote`).
2. **Renderer mapping** (`aborist/cli.py:_render_audit_label`):
add the four-rung mapping function gated on
`verifier_method.startswith("claim_lattice")`.
3. **Bench update** (`bench/qa_sweep.py`): per-row
`ladder_rung` field; markdown summary gets a "ladder coverage"
table alongside the existing strict-rate breakdown.
4. **Live fixtures** (`tests/test_qa_quality_live.py`): the
existing `test_mona_lisa_strict_*` and siblings get their
assertion strings updated to the new rung names.
## 6. Tests required
- `[ ]` Renderer maps verifier output to POINTER-LINKED when no
warrant ran.
- `[ ]` Renderer maps to ANCHOR-WARRANTED when warrant ran &
passed but other demotes fired.
- `[ ]` Renderer maps to EVIDENCE-WARRANTED when warrant passed
& no demotes.
- `[ ]` Renderer suffix `-PARTIAL` appears for HYBRID + each rung.
- `[ ]` Renderer keeps STRICT/HYBRID/UNGROUNDED for quote-mode
verifier paths (no ladder rename).
- `[ ]` Verifier verdict dict carries `verifier_steps_ran` list.
- `[ ]` Live fixtures updated to gate on rung names.
- `[ ]` Cache lookup returns identical results before & after
migration (governance_policy_hash unchanged).
## 7. Out of scope
- **`ENTAILMENT-VERIFIED` rung.** Reserved for a future committed
entailment engine. Not reachable today; no code path produces it.
Naming the rung today preserves the design space without forcing
premature implementation.
- **Schema column rename.** Stays `audit_mode ∈ {STRICT, HYBRID,
UNGROUNDED}`. v9.8 cache_key invariants depend on this; never
break.
- **Mesh wire format change.** Mesh peers exchange `audit_mode`,
not the rendered rung. No wire-protocol bump required.
## 8. Status
**Proposal.** Forecast cost: ~2 hours (verifier signal + renderer
mapping + ~8 tests + live-fixture migration). Risk: low — pure
display layer; schema and audit chain stay unchanged.
Forecast value: closes Directive D7 in stronger form. An operator
reading `EVIDENCE-WARRANTED` on a record knows the answer is
pointer-linked AND anchor-warranted AND no soft-demotes fired —
strictly more information than `EVIDENCE-LINKED` carried.