arborist/docs/tickets/ticket-000005-label-ladder-migration.md
russell@unturf.com 8d6961fcc1
aborist/arborist
modified:   .gitlab-ci.yml
	modified:   bench/qa_questions.txt
	modified:   bench/qa_sweep.py
	modified:   bench/run.sh
	modified:   docs/TICKETS.md
	modified:   docs/_source/README.md
	modified:   docs/_source/_ext/makefile_targets.py
	modified:   docs/_source/api/cli.rst
	modified:   docs/_source/api/distill.rst
	modified:   docs/_source/api/mesh.rst
	modified:   docs/_source/api/qa.rst
	modified:   docs/_source/api/retrieval.rst
	modified:   docs/_source/api/storage.rst
	modified:   docs/_source/api/substrate.rst
	modified:   docs/_source/concepts.rst
	modified:   docs/_source/conf.py
	modified:   docs/_source/cookbook.rst
	modified:   docs/_source/index.rst
	modified:   docs/_source/license.rst
	modified:   docs/_source/quickstart.rst
	modified:   docs/bench-maxing.md
	modified:   docs/benchmarks.md
	modified:   docs/cti-architecture.md
	modified:   docs/diagrams/aborist-modules.dot
	modified:   docs/diagrams/aborist-modules.svg
	modified:   docs/diagrams/mesh-data-flow.dot
	modified:   docs/diagrams/mesh-epoch-lifecycle.dot
	modified:   docs/diagrams/mesh-epoch-lifecycle.svg
	modified:   docs/diagrams/mesh-group-decisions.dot
	modified:   docs/diagrams/mesh-group-decisions.svg
	modified:   docs/diagrams/mesh-identity-stack.dot
	modified:   docs/diagrams/mesh-secret-envelope.dot
	modified:   docs/mesh.md
	modified:   docs/qa-modes-bench.md
	modified:   docs/seven-point-program.md
	modified:   docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md
	modified:   docs/tickets/ticket-000002-reference-frame-polarity-contract.md
	modified:   docs/tickets/ticket-000003-anchor-class-warrant.md
	modified:   docs/tickets/ticket-000005-label-ladder-migration.md
	modified:   docs/tickets/ticket-000006-bench-emergent-findings.md
	modified:   docs/tickets/ticket-000007-query-layer-hyphen-fold.md
	modified:   docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md
	modified:   docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md
	modified:   docs/tickets/ticket-000010-metacognition-preflight-guard.md
	modified:   docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md
	modified:   scripts/backfill_concepts.py
	modified:   scripts/bench_emergent.py
	modified:   tests/crawler/test_async_web_fetcher.py
	modified:   tests/crawler/test_bridge.py
	modified:   tests/crawler/test_web_fetch.py
	modified:   tests/test_bench_qa_sweep.py
	modified:   tests/test_burn.py
	modified:   tests/test_burn_doc.py
	modified:   tests/test_claim_lattice.py
	modified:   tests/test_cli_render.py
	modified:   tests/test_compress.py
	modified:   tests/test_concepts.py
	modified:   tests/test_dag.py
	modified:   tests/test_directives.py
	modified:   tests/test_distill.py
	modified:   tests/test_distill_recursive.py
	modified:   tests/test_evict.py
	modified:   tests/test_frame.py
	modified:   tests/test_grok_source.py
	modified:   tests/test_html_source.py
	modified:   tests/test_ingest.py
	modified:   tests/test_inspect.py
	modified:   tests/test_journal.py
	modified:   tests/test_keys.py
	modified:   tests/test_llm_context_base.py
	modified:   tests/test_merkle.py
	modified:   tests/test_mesh.py
	modified:   tests/test_mesh_aead.py
	modified:   tests/test_mesh_chain.py
	modified:   tests/test_mesh_cli.py
	modified:   tests/test_mesh_cli_pull.py
	modified:   tests/test_mesh_wire.py
	modified:   tests/test_mesh_wire_e2e.py
	modified:   tests/test_metacognition.py
	modified:   tests/test_migration_audit_mode.py
	modified:   tests/test_providence_source.py
	modified:   tests/test_qa.py
	modified:   tests/test_qa_quality_live.py
	modified:   tests/test_quantifier_caps.py
	modified:   tests/test_quantifier_classifier.py
	modified:   tests/test_quantifier_phase4.py
	modified:   tests/test_quantifier_reminder.py
	modified:   tests/test_query.py
	modified:   tests/test_reclassify.py
	modified:   tests/test_repair.py
	modified:   tests/test_resume.py
	modified:   tests/test_snapshot.py
	modified:   tests/test_soft_preflight.py
	modified:   tests/test_tfidf.py
	modified:   tests/test_vcs_source.py
	modified:   tests/test_verify.py
	modified:   tests/test_verify_json.py
	modified:   tests/test_versioned_ingest.py
	modified:   tests/test_warrant.py
	modified:   tests/test_wikipedia_old.py
	modified:   tests/test_wikipedia_xml.py
	modified:   tests/test_wikitext.py
2026-05-07 09:31:49 -04:00

8.4 KiB

Ticket #000005 — Label ladder migration: POINTER-LINKED / ANCHOR-WARRANTED / EVIDENCE-WARRANTED

Status: closed · landed 2026-05-02 Opened: 2026-05-01 Closed: 2026-05-02 Directive: D7 — Rename labels honestly. Scope: Replace today's two-label rendered surface (EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL) with a four-rung ladder that names exactly what each verifier path proved. Schema column stays STRICT / HYBRID / UNGROUNDED so the v9.8 cache_key invariant holds; only the rendered surface changes. Audience: fox + future blackops shifts. Hard constraint: schema column stays unchanged. Renderer-level mapping only. The audit chain, mesh wire format, providence_cache, and all programmatic callers continue to see the original audit_mode enum.


1. Problem statement

Today's renderer relabels claim-lattice modes:

STRICT     → EVIDENCE-LINKED · via claim_lattice
HYBRID     → EVIDENCE-LINKED-PARTIAL · via claim_lattice_pointer
UNGROUNDED → UNGROUNDED · via claim_lattice

EVIDENCE-LINKED improves on STRICT (which overclaims semantic truth) but still reads as "the evidence supports the claim." A careful operator parses it as "pointer + role + coverage checks passed." A casual one reads it as "the model got this right."

The four-rung ladder fox proposed (2026-05-01) carries the distinction explicitly:

POINTER-LINKED       pointer/source/chunk relation verified
ANCHOR-WARRANTED     pointer-linked AND cited evidence contains
                     required claim anchors (Rule 7 + future
                     entity/count/cause classes from #000003)
EVIDENCE-WARRANTED   anchor-warranted AND no hard violations
ENTAILMENT-VERIFIED  reserved for a future committed entailment
                     engine (NLI substrate). Not reachable today.

Each rung names a strictly stronger property. An operator reading the rung knows what's been proved without inferring from the verifier_method tail.

2. Mapping from current state to ladder

Today's verifier output gets re-mapped at the renderer layer:

Verifier method Today's render New render
claim_lattice EVIDENCE-LINKED · via … POINTER-LINKED (no warrant ran), ANCHOR-WARRANTED (warrant fired & passed), EVIDENCE-WARRANTED (anchor-warranted + zero soft-demote violations)
claim_lattice (HYBRID) EVIDENCE-LINKED-PARTIAL POINTER-LINKED-PARTIAL or ANCHOR-WARRANTED-PARTIAL depending on which checks demoted
claim_lattice (UNGROUNDED) UNGROUNDED UNGROUNDED (unchanged)
quote / span / entity / paraphrase STRICT/HYBRID/UNGROUNDED unchanged (these verify against pinned spans, not synthesis)

Mapping rule (claim-lattice modes only):

def _ladder_label(audit_mode, violations) -> str:
    if audit_mode == "UNGROUNDED":
        return "UNGROUNDED"
    has_warrant_check = any(v["kind"] == "WARRANT_RAN" for v in violations)
    has_warrant_miss  = any(v["kind"] == "WARRANT_MISSING" for v in violations)
    has_other_demote  = any(v["kind"] in DEMOTE_KINDS for v in violations)
    if not has_warrant_check:
        rung = "POINTER-LINKED"
    elif has_warrant_miss:
        rung = "POINTER-LINKED"          # warrant ran AND failed
    elif has_other_demote:
        rung = "ANCHOR-WARRANTED"        # warrant ok but other demotes
    else:
        rung = "EVIDENCE-WARRANTED"      # warrant ok AND no demotes
    if audit_mode == "HYBRID":
        rung = rung + "-PARTIAL"
    return rung

3. New verifier signal: WARRANT_RAN

For the renderer to know whether the warrant fired, the verifier needs to surface that signal. Today's warrant_check returns (ok, missing); only WARRANT_MISSING enters the violations list when ok=False. Add a benign WARRANT_RAN violation entry (or a separate verifier_steps_ran: list[str] field) so the renderer can distinguish "warrant didn't apply" from "warrant applied and passed."

This is a renderer-input enhancement, not a verifier semantic change. The hard checks stay binary; we just thread the "which checks ran" provenance up to the display layer.

4. Cache implications

None. Schema audit_mode enum is unchanged; the renderer mapping runs on read, not on write. Existing providence records continue to render under the new ladder without re-classification or cache invalidation.

governance_policy_hash does NOT include the ladder mapping — it's a pure presentation transformation, not a behavioral change. Two records with the same cache_key written before and after the ladder migration render under the new ladder identically.

5. Implementation sketch

  1. Verifier enhancement (arborist/qa/verify.py): both verify_claim_lattice and verify_claim_lattice_json add a verifier_steps_ran: list[str] field on the verdict dict listing which verifier paths fired (pointer_resolve, source_role, coverage, pointer_cap, warrant, lazy_anchor_demote).
  2. Renderer mapping (arborist/cli.py:_render_audit_label): add the four-rung mapping function gated on verifier_method.startswith("claim_lattice").
  3. Bench update (bench/qa_sweep.py): per-row ladder_rung field; markdown summary gets a "ladder coverage" table alongside the existing strict-rate breakdown.
  4. Live fixtures (tests/test_qa_quality_live.py): the existing test_mona_lisa_strict_* and siblings get their assertion strings updated to the new rung names.

6. Tests required

  • [ ] Renderer maps verifier output to POINTER-LINKED when no warrant ran.
  • [ ] Renderer maps to ANCHOR-WARRANTED when warrant ran & passed but other demotes fired.
  • [ ] Renderer maps to EVIDENCE-WARRANTED when warrant passed & no demotes.
  • [ ] Renderer suffix -PARTIAL appears for HYBRID + each rung.
  • [ ] Renderer keeps STRICT/HYBRID/UNGROUNDED for quote-mode verifier paths (no ladder rename).
  • [ ] Verifier verdict dict carries verifier_steps_ran list.
  • [ ] Live fixtures updated to gate on rung names.
  • [ ] Cache lookup returns identical results before & after migration (governance_policy_hash unchanged).

7. Out of scope

  • ENTAILMENT-VERIFIED rung. Reserved for a future committed entailment engine. Not reachable today; no code path produces it. Naming the rung today preserves the design space without forcing premature implementation.
  • Schema column rename. Stays audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}. v9.8 cache_key invariants depend on this; never break.
  • Mesh wire format change. Mesh peers exchange audit_mode, not the rendered rung. No wire-protocol bump required.

8. Status

Closed 2026-05-02. Landed via:

  • _render_audit_label(audit_mode, verifier_method, violations) in arborist/cli.py — three-arg signature; violations defaults to None for backward-compat with callers that don't have the list yet.
  • _ladder_rung_for_lattice helper computes the rung from (audit_mode, violations) using the existing violation kinds (no new verifier output field needed):
    • WARRANT_MISSING in violations → POINTER-LINKED
    • any of LAZY_ANCHOR_DEMOTED / POINTER_OVERFLOW_TRIMMED / TOO_MANY_CLAIMS / BARE_NAME_CLAIMANCHOR-WARRANTED
    • else → EVIDENCE-WARRANTED
    • HYBRID adds -PARTIAL suffix.
  • Quote / span / entity / paraphrase methods unchanged (their STRICT verifies pinned spans, not synthesis).
  • Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED} — pure renderer transformation.

Design simplification vs §3: the proposed verifier_steps_ran: list[str] field on the verdict dict was NOT needed. The existing violations list carries enough signal to discriminate the rungs (presence of WARRANT_MISSING discriminates POINTER-LINKED from ANCHOR-WARRANTED; presence of any soft-demote kind discriminates ANCHOR-WARRANTED from EVIDENCE-WARRANTED). Per the five-step algorithm step 2: don't add fields you don't need.

5 new renderer tests in tests/test_cli_render.py covering each rung mapping. D7 directive test in tests/test_directives.py updated to gate on the ladder labels. Full suite: 711 passed.

Cache implications: none. Schema stays unchanged; existing records render under the new ladder without re-classification. governance_policy_hash does NOT include the rendered label — pure presentation transformation.

Mesh wire format: unchanged. Peers exchange audit_mode not the rendered rung.