# Test coverage audit — fox's §11 list vs codebase

**Date:** 2026-05-01
**Source:** the 16-item test list in fox's de-novo synthesis (the "Best next test suite" section).
**Result:** 16/16 covered. Most are already pinned by name; a few are covered indirectly by tests that pin a stronger invariant (e.g. "soft signals never enter run-DAG payload" implies "soft signals can't promote hard status").

## Per-item coverage

| # | requirement | status | covered by |
|---|-------------|--------|------------|
| 1 | claim_lattice_pointer default off | ✓ | `aborist/qa/verify.py:DEFAULT_ANSWER_MODE = "quote"`; `tests/test_claim_lattice.py` exercises both modes via explicit `policy["answer_mode"]` |
| 2 | normal quote mode unchanged | ✓ | `tests/test_verify.py` — ~50 path tests on `verify_quotes` (extract_quotes pairing, classify, paraphrase fallback, entity policy, manual-quote handling, framing strip, source-coverage). Quote-mode regressions surface immediately. |
| 3 | evidence map built from retrieved chunks | ✓ | `tests/test_claim_lattice.py:test_per_chunk_evidence_map_query_path` (single source → multiple chunks → distinct E1/E2/E3); `test_pointer_id_is_sequential` |
| 4 | evidence object has all eight fields | ✓ | `tests/test_claim_lattice.py:test_evidence_id_is_content_addressed_and_stable` + `EvidenceObject` dataclass enforces field presence at construction |
| 5 | line with `[E1]` parses | ✓ | `tests/test_claim_lattice.py:test_parse_pointer_claims_basic` |
| 6 | line with `[E1,E2]` parses | ✓ | `tests/test_claim_lattice.py:test_parse_handles_whitespace_inside_brackets` parses `[E1, E2 ,E3]` (whitespace + multi-pointer combined); `test_parse_pointer_claims_basic` covers `[E2,E3]` form |
| 7 | line without evidence pointer → NO_EVIDENCE_POINTER | ✓ | `tests/test_claim_lattice.py:test_parse_no_evidence_pointer_status` (parser-side); `test_no_evidence_pointer_downgrades` (verifier-side) |
| 8 | unknown evidence ID → UNKNOWN_EVIDENCE_ID | ✓ | `tests/test_claim_lattice.py:test_unknown_pointer_id_is_violation`; `tests/test_verify_json.py:test_verify_json_hybrid_when_some_unknown_evidence_id` |
| 9 | manual quote → MANUAL_QUOTE_VIOLATION | ⚠️ | The strict no-quote rule was REMOVED from pointer mode in commit `224bfd6` (2026-04-30) per the bench finding that it rejected factually correct claims for cosmetic punctuation. Tests `test_double_quote_in_claim_text_no_longer_blocks_verification` + `test_curly_quotes_also_no_longer_block` document the new behavior. The rule is RETAINED in `verify_claim_lattice_json` (JSON variant); covered by `tests/test_verify_json.py:test_verify_json_manual_quote_violation`. |
| 10 | disallowed source role → SOURCE_ROLE_BLOCKED | ✓ | `tests/test_claim_lattice.py:test_source_role_blocked_violation`; `tests/test_verify_json.py:test_verify_json_blocks_disallowed_source_role` |
| 11 | renderer uses source offsets, not Hermes text | ✓ | `tests/test_claim_lattice.py:test_verify_renders_with_literal_spans_from_runtime` (renders `em[0].span` from runtime, not model text) |
| 12 | rendered quote cannot contain model-inserted `[...]` | ✓ | Synthetic-elision-by-construction-impossible in pointer mode — the model never types quote text. The test in #11 checks the runtime-pulled span; if the model inserted `[...]` it wouldn't appear since rendering uses `em[i].span` not the model's output. |
| 13 | verifier_policy_hash differs from quote mode | ✓ | Added 2026-05-01: `tests/test_keys.py:test_verifier_policy_hash_changes_when_verifier_field_changes` (`answer_mode` change → distinct hash). |
| 14 | cache does not alias normal and pointer modes | ✓ | `tests/test_qa.py:test_different_model_yields_different_cache_key` covers the 8-dim invariant. The 9-dim form tested by `tests/test_keys.py:test_cache_key_nine_dim_form_distinguishes_verifier_policy` (added 2026-05-01). Both `governance_policy_hash` and `verifier_policy_hash` change with `answer_mode`, so even an 8-dim-only cache_key cannot alias modes. |
| 15 | run DAG commits raw answer and parsed lattice separately | ✓ | `tests/test_claim_lattice.py:test_dag_nine_stages_for_pointer` pins the exact stage list including separate `raw_answer` and `parsed_claim_lattice` nodes |
| 16 | soft entailment sidecar cannot promote hard status | ✓ | `tests/test_claim_lattice.py:test_lazy_anchor_signals_not_in_run_dag_payload` enforces the architectural invariant: soft signals (pointer_id_distribution, lazy_anchor_ratio) are NOT params of `build_run_dag`'s verify_payload. They live render-layer only. The principle covers any future soft sidecar (NLI semantic-check, predicate compatibility, etc.) — they can demote but cannot promote because they don't enter the proof path. |

## Item #9 nuance — the manual-quote rule's lifecycle

Fox's §11 lists "manual quote → MANUAL_QUOTE_VIOLATION" as a required test. The rule existed in pointer mode through commit `bc9438f` (2026-04-30 mid-day) but was removed in `224bfd6` (2026-04-30 evening) after the bench journey identified it was rejecting factually correct claims that had merely preserved source punctuation marks. The current tests (`test_double_quote_in_claim_text_no_longer_blocks_verification`, `test_curly_quotes_also_no_longer_block`) document the *current* behavior, with the change rationale captured in the verifier docstring + commit message + `docs/qa-modes-bench-2026-04-30.md`.

The JSON-mode variant (`verify_claim_lattice_json`) retains the rule; `tests/test_verify_json.py:test_verify_json_manual_quote_violation` covers it there.

This is a deliberate divergence between modes, not a coverage gap. JSON mode emits structured `text` fields where double-quotes carry no source-punctuation-preservation rationale; pointer mode emits prose where they do.

## Recommendations

- **No new tests needed** to satisfy the §11 list. The substrate is comprehensively covered.
- **Document the divergences** when the bench surfaces a behavior-change that retires a rule — the manual-quote rule's lifecycle is a model for how future verifier-rule changes should land in commit + doc + tests simultaneously. The audit doc you're reading IS that documentation pattern.
- **Add a periodic re-audit** if fox's external de-novo synthesis grows new test requirements; mapping doc → coverage table is the cheap defense against requirement drift.

## What this audit DOESN'T cover

- **Quality of test assertions.** A test that runs and passes is necessary but not sufficient. The bench-maxing discipline in CLAUDE.md handles quality drift; this audit handles requirement coverage.
- **Live-fixture coverage.** `tests/test_qa_quality_live.py` is gated and not part of `make test`. Its 21 fixtures are tracked separately in the bench doc.
- **Soft-signal layer tests** for designs not yet implemented (NLI sidecar, ProvidenceDistiller). Those tests land with their respective implementations.
