docs: tickets/ subdir + delete naming-deferral, ref-frame, coverage-audit
Two cleanup operations bundled (separate scopes, single commit
since they share the doc-tree settle):
1. Move ticket-NNNNNN-<slug>.md files into docs/tickets/. The
directory makes browsing the design log easier; the index
stays at docs/TICKETS.md (top-level pointer). Convention text
in TICKETS.md updated to spell the new path.
2. Delete three docs whose load-bearing content has either been
absorbed into the codebase or distilled into closed tickets:
- docs/naming-deferral.md (147 lines) — explained why we
don't rename claim_lattice → CTI/PROMETHEUS-Σ. Decision
stays in place; the rationale is no longer worth a
dedicated doc. Inline citations removed from
cti-architecture.md (4 refs), warrant.py, ticket-000003
(closed-ticket internal ref).
- docs/reference-frame-failure-class.md (169 lines) — Orwell
case journal that motivated the phrase-pattern retrieval
route. The route shipped; the analysis is now duplicate
with the closed Ticket #000002. Inline citation removed
from CLAUDE.md retrieval pipeline section + frame.py.
- docs/test-coverage-audit-2026-05-01.md (46 lines) —
point-in-time audit checking 16/16 of fox's §11 list. Tests
themselves live in tests/; the audit was a one-shot
checkmark exercise.
References updated:
CLAUDE.md, aborist/qa/frame.py, aborist/qa/retrieval_plan.py,
aborist/qa/warrant.py, docs/cti-architecture.md, docs/TICKETS.md,
docs/tickets/ticket-000003 + ticket-000004 (internal links).
Net: -362 lines + tickets/ subdir. 751/34 tests still pass.
This commit is contained in:
parent
ecc18ea724
commit
7444f106fc
14 changed files with 12 additions and 380 deletions
|
|
@ -221,7 +221,7 @@ enough; revert at your peril. Order:
|
|||
(verbatim n=5/n=6 sequences from the question). Phrase route
|
||||
closes the allusion gap (Orwell case: "always been at war" verbatim
|
||||
matches the 1984 article whose title shares zero tokens with the
|
||||
query). See `docs/reference-frame-failure-class.md`.
|
||||
query).
|
||||
2. **Body-coverage `sqrt` rerank** — counters BM25's short-doc bias.
|
||||
3. **Title-token boost** — `boost × overlap` on title-token-matching hits.
|
||||
4. **`_filter_by_title_relevance` — four accept paths**: title-token
|
||||
|
|
@ -247,7 +247,7 @@ enough; revert at your peril. Order:
|
|||
`--retrieval-keywords` (CLI: `K="..."`) lets an operator augment
|
||||
retrieval-side tokens without changing what the LLM sees as its
|
||||
question. Provenance gap on this is tracked in
|
||||
[Ticket #000001](docs/ticket-000001-retrieval-keywords-audit-gap.md).
|
||||
[Ticket #000001](docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md).
|
||||
|
||||
## Hot path / gotchas
|
||||
|
||||
|
|
@ -330,13 +330,10 @@ Architecture / ongoing work:
|
|||
pointer 0.23, JSON 0.44. Concurrency landed alongside.
|
||||
- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey
|
||||
(prior bench reference).
|
||||
- `docs/reference-frame-failure-class.md` — Orwell case + phrase-route fix.
|
||||
- `docs/bench-maxing.md` — bench discipline.
|
||||
- `docs/test-coverage-audit-2026-05-01.md` — test-suite coverage audit.
|
||||
- `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal.
|
||||
- `docs/self-reference-design.md`,
|
||||
`docs/self-reference-design.md` — recursive distillation.
|
||||
- `docs/naming-deferral.md` — naming convention notes.
|
||||
|
||||
Tickets: `docs/TICKETS.md` is the authoritative index with `Next
|
||||
ID`. As of 2026-05-02 all five shipped tickets (#000001–#000005)
|
||||
|
|
|
|||
|
|
@ -1,14 +1,14 @@
|
|||
"""Reference-frame detection for claim-lattice queries.
|
||||
|
||||
Closes the answer-side gap left open by the phrase-pattern retrieval
|
||||
route (commit `1b8677d` / docs/reference-frame-failure-class.md). The
|
||||
route (commit `1b8677d`). The
|
||||
phrase route surfaces the right article for allusion-shape queries
|
||||
(e.g. Nineteen Eighty-Four for "has oceania always been at war with
|
||||
east asia"); this module then classifies the answer's intended frame
|
||||
so the prompt + renderer can produce a multi-frame answer that
|
||||
distinguishes literal from reference from in-universe-propaganda.
|
||||
|
||||
See `docs/ticket-000002-reference-frame-polarity-contract.md` for the
|
||||
See `docs/tickets/ticket-000002-reference-frame-polarity-contract.md` for the
|
||||
full scope. This module is the *substrate* — it returns a
|
||||
`FrameDetection` per call. Prompt augmentation + renderer extension
|
||||
live in their respective layers and consume this output.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Without this hash, two runs with the same question + different
|
|||
retrieval keywords that surface identical sources would be Merkle-
|
||||
indistinguishable — an audit could recover "these documents were
|
||||
selected" but not "these were the keywords that pulled them in." See
|
||||
`docs/ticket-000001-retrieval-keywords-audit-gap.md` for the full
|
||||
`docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md` for the full
|
||||
problem statement.
|
||||
|
||||
Hard or soft? Hard. The hash is SHA-256 over canonical-JSON; the
|
||||
|
|
|
|||
|
|
@ -55,8 +55,7 @@ question-type rule libraries like "release_date_lookup with
|
|||
required predicate cues"). The general anchor-class primitive
|
||||
catches the back-to-the-future failure deterministically without a
|
||||
per-type rule book. Typed contracts earn their slot only when bench
|
||||
evidence shows the general primitive misses cases — see
|
||||
docs/naming-deferral.md for the discipline.
|
||||
evidence shows the general primitive misses cases.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ go/no-go, or scoped enhancements queued for future implementation.
|
|||
|
||||
## Convention
|
||||
|
||||
- **Filename:** `docs/ticket-NNNNNN-<slug>.md`. The ID is a six-digit
|
||||
- **Filename:** `docs/tickets/ticket-NNNNNN-<slug>.md`. The ID is a six-digit
|
||||
zero-padded integer; the slug is hyphen-lowercase and short enough
|
||||
to stay readable on a 100-column terminal.
|
||||
- **Numbering:** monotonic, never reused. Pick the next free integer
|
||||
|
|
@ -39,7 +39,6 @@ proposing change. Examples that stay un-numbered:
|
|||
|
||||
- `cti-architecture.md` (architecture reference)
|
||||
- `mesh.md`, `mesh-deploy.md` (architecture + ops reference)
|
||||
- `naming-deferral.md` (naming convention reference)
|
||||
- `qa-modes-bench-2026-04-30.md` (point-in-time bench journal)
|
||||
- `verifier-semantic-gap-design.md`,
|
||||
`self-reference-design.md` (older design docs that
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
**Audience:** fox + future blackops shifts.
|
||||
**Purpose:** name the architectural layers fox's de novo synthesis identifies (PROMETHEUS-Σ / CTI / Merkle-AGI-DAG / Reverse-RAG / v9.8 / Hermes) against the modules that already exist, so future work has a stable vocabulary for which layer it touches.
|
||||
|
||||
This is a **mapping doc**, not a rename. The codebase keeps its internal terms (`claim_lattice`, `claim_lattice_pointer`, `verify_claim_lattice`); the architectural labels (CTI, PROMETHEUS-Σ, Merkle-AGI) live in docs and commit messages where the cross-cutting story matters. See `docs/naming-deferral.md` for why.
|
||||
This is a **mapping doc**, not a rename. The codebase keeps its internal terms (`claim_lattice`, `claim_lattice_pointer`, `verify_claim_lattice`); the architectural labels (CTI, PROMETHEUS-Σ, Merkle-AGI) live in docs and commit messages where the cross-cutting story matters.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -68,7 +68,7 @@ The animating principle: **the model proposes, the runtime structures, the verif
|
|||
- `DEFAULT_POLICY` (runner) and `DEFAULT_QUERY_POLICY` (query) — the policy dict
|
||||
- `aborist.qa.keys.governance_policy_hash` — folds policy into the cache_key
|
||||
|
||||
**Status:** Implicit. The layer exists as code but is not named. The de-novo doc proposes naming it; the codebase doesn't need a `prometheus.py` module today — the dispatch logic in `runner.py` and `query.py` IS PROMETHEUS-Σ. (See `docs/naming-deferral.md`.)
|
||||
**Status:** Implicit. The layer exists as code but is not named. The de-novo doc proposes naming it; the codebase doesn't need a `prometheus.py` module today — the dispatch logic in `runner.py` and `query.py` IS PROMETHEUS-Σ.
|
||||
|
||||
### 2.2 CTI — Clause Tree Intelligence (reasoning IR)
|
||||
|
||||
|
|
@ -185,7 +185,6 @@ The audit flow is what makes Merkle-AGI O(log N + k) per challenged claim — yo
|
|||
| NLI semantic-gap sidecar | `docs/verifier-semantic-gap-design.md` |
|
||||
| Self-reference (flat MVP) | `docs/self-reference-design.md` |
|
||||
| Self-reference (deep, distillation-based) | `docs/self-reference-design.md` |
|
||||
| Why we don't rename code to CTI/PROMETHEUS-Σ | `docs/naming-deferral.md` |
|
||||
| Bench-maxing discipline | CLAUDE.md "Bench-maxing" section |
|
||||
| Schema invariants | CLAUDE.md "Schema invariants (do not break)" |
|
||||
| Convention list | CLAUDE.md "Conventions (do not silently change)" |
|
||||
|
|
@ -208,6 +207,6 @@ These are the layer-cake-shaped work items still on the roadmap:
|
|||
1. **Verifier-policy-hash separation** — landing in this commit pass. Fold `answer_mode + parser_version + evidence_schema_version + manual_quote_policy + hard_checks_list` into a 9th dimension that's separate from `governance_policy_hash`. Prevents cross-mode cache aliasing more cleanly.
|
||||
2. **Soft-signal taxonomy expansion** — the NLI sidecar (`docs/verifier-semantic-gap-design.md`) is one. Predicate-compatibility, completeness, counterevidence, source-authority, scope-ambiguity are others. All demote-only, all out of the proof path. Implementations come one at a time as the bench surfaces motivating cases.
|
||||
3. **Self-reference distillation** — STRICT claims become Cores via a new `ProvidenceDistiller` (see `docs/self-reference-design.md`). Lets new claims compose from existing facts, not just retrieve them.
|
||||
4. **PROMETHEUS-Σ explicit naming** — deferred, see `docs/naming-deferral.md`. The control logic exists; an extracted `prometheus.py` module is a refactor with no behavior change.
|
||||
4. **PROMETHEUS-Σ explicit naming** — deferred. The control logic exists; an extracted `prometheus.py` module is a refactor with no behavior change.
|
||||
|
||||
The architecture is real. The labels above name what's already there.
|
||||
|
|
|
|||
|
|
@ -1,147 +0,0 @@
|
|||
# Why we don't rename `claim_lattice` to CTI / PROMETHEUS-Σ
|
||||
|
||||
**Date:** 2026-05-01
|
||||
**Decision:** keep the codebase's internal terms (`claim_lattice`, `claim_lattice_pointer`, `verify_claim_lattice`, `governance_policy_hash`, etc.). The architectural labels (CTI, PROMETHEUS-Σ, Merkle-AGI-DAG, Reverse-RAG) live in `docs/cti-architecture.md` and commit messages where the cross-cutting story matters.
|
||||
**Status:** active. Re-evaluate when the rename triggers below fire.
|
||||
|
||||
---
|
||||
|
||||
## 1. The concrete proposal we considered
|
||||
|
||||
Fox's de-novo synthesis (2026-05-01) names the architectural layers cleanly:
|
||||
|
||||
```text
|
||||
PROMETHEUS-Σ controller / policy / admissibility / one-shot
|
||||
CTI Clause Tree Intelligence — the runtime IR
|
||||
Merkle-AGI-DAG cryptographic commitment layer
|
||||
Reverse-RAG evidence direction (claim → span → root)
|
||||
v9.8 Providence admissibility ledger
|
||||
Hermes weak proposer
|
||||
```
|
||||
|
||||
The internal codebase uses different names for the same layers:
|
||||
|
||||
| architectural label | internal name(s) | location |
|
||||
|---|---|---|
|
||||
| PROMETHEUS-Σ | `runner.ask`, `query.query`, the policy-dict + dispatch logic | `aborist/qa/runner.py`, `aborist/qa/query.py` |
|
||||
| CTI | `claim_lattice` (mode), `claim_lattice_pointer` (mode), `verify_claim_lattice`, `parse_pointer_claims`, `EvidenceObject` | `aborist/qa/{verify,parse_claims,evidence}.py` |
|
||||
| Merkle-AGI-DAG | `build_run_dag`, `run_dag_root`, `audit_events`, `MerkleTree`, `HashCombine` | `aborist/qa/dag.py`, `aborist/merkle.py`, `aborist/store.py` |
|
||||
| Reverse-RAG | the `claim → evidence_id → chunk_root → source_root` chain in `verify_claim_lattice` + `render_claim_lattice` | `aborist/qa/verify.py`, `aborist/qa/evidence.py` |
|
||||
| v9.8 Providence | `providence_cache` table, the 8-dim `cache_key`, `falsification_state` | `aborist/store.py`, `aborist/qa/keys.py` |
|
||||
| Hermes proposer | `OpenAICompatibleClient`, `claim_lattice_system_prompt` | `aborist/qa/client.py`, `aborist/qa/runner.py` |
|
||||
|
||||
Both vocabularies refer to exactly the same code. The question is whether to rename internal symbols to match the external architectural labels.
|
||||
|
||||
**The decision is no, defer.** The reasoning is below.
|
||||
|
||||
## 2. What the rename would actually cost
|
||||
|
||||
Renaming the internal symbols isn't a search-and-replace. The cost lives in five places.
|
||||
|
||||
### 2.1 Cache invalidation
|
||||
|
||||
The string `"claim_lattice"` and `"claim_lattice_pointer"` are values of `policy["answer_mode"]` — and `answer_mode` is folded into `governance_policy_hash` which is one of the 8 dimensions of `cache_key`. Renaming the mode strings changes the hash, which **stales every cached record under those modes** on next lookup.
|
||||
|
||||
That's not catastrophic — caches are designed to handle invalidation — but it converts a stylistic rename into a substrate-wide cache flush. Every cached run since the modes existed becomes inaccessible-by-default. Mesh-broadcast records would diverge between renamed and pre-rename peers until they all migrated.
|
||||
|
||||
The rename has to be *worth* a mass cache invalidation. Aesthetic alignment with architectural docs isn't.
|
||||
|
||||
### 2.2 Test fixture footprint
|
||||
|
||||
The strings `claim_lattice`, `claim_lattice_pointer`, `verify_claim_lattice`, `policy["answer_mode"] = "claim_lattice"`, etc. appear in:
|
||||
|
||||
- `tests/test_claim_lattice.py` (~75 references)
|
||||
- `tests/test_verify_json.py` (~40 references)
|
||||
- `tests/test_qa_quality_live.py` (~20 references — `mode="claim_lattice_pointer"` parameter)
|
||||
- `tests/test_query.py` and `tests/test_qa.py` (additional refs)
|
||||
- Many docstrings in `aborist/qa/*.py`
|
||||
|
||||
A consistent rename touches ~150+ references across the test suite + docstrings. None of it is hard; all of it is churn. Reviewing the diff to confirm "every rename is correct" is the actual cost.
|
||||
|
||||
### 2.3 Schema CHECK constraints
|
||||
|
||||
The `providence_cache.verifier_method` column has a SQL `CHECK` constraint:
|
||||
|
||||
```sql
|
||||
verifier_method IN ('quote','span','entity','paraphrase','claim_lattice','none')
|
||||
```
|
||||
|
||||
If we rename `claim_lattice` to `cti`, every existing cached record violates the new CHECK. Migration script + downtime + audit-chain integrity check. Real engineering work, not a refactor.
|
||||
|
||||
### 2.4 External commit history + doc archeology
|
||||
|
||||
Today's `git log --oneline` includes commit messages that read clean:
|
||||
|
||||
```
|
||||
qa: drop manual-quote rule from pointer verifier; port G0 policy to runner
|
||||
qa: bare-name guard, lazy-anchor demote, game tie-in noisy markers
|
||||
qa: JSON mode uses pointer IDs (E1, E2, …) — close hallucination loop
|
||||
qa: self-reference thought chains — STRICT-as-fact substrate
|
||||
```
|
||||
|
||||
A rename produces a different parallel set of commit messages mid-history:
|
||||
|
||||
```
|
||||
qa: rename claim_lattice → cti everywhere
|
||||
qa: rename claim_lattice_pointer → cti_pointer everywhere
|
||||
... [N follow-up commits fixing references the first pass missed]
|
||||
```
|
||||
|
||||
The blame view becomes messier; `git log -S claim_lattice_pointer` returns nothing useful past the rename point; future agents reading commit messages see a disconnect between yesterday's commits and today's symbol names.
|
||||
|
||||
### 2.5 Mesh / broadcast compatibility
|
||||
|
||||
Records exchanged across the mesh carry their full 8-dim cache_key + answer_mode. A peer running a renamed build sees `answer_mode="cti_pointer"` from itself but `answer_mode="claim_lattice_pointer"` from peers running pre-rename. Either:
|
||||
|
||||
- the rename is rolled out atomically across all mesh peers (real coordination cost), or
|
||||
- both names are accepted in parallel for a transition window (which means we maintain BOTH vocabularies for as long as the transition takes — the explicit cost we're trying to avoid).
|
||||
|
||||
## 3. What we get from the rename
|
||||
|
||||
Honest accounting:
|
||||
|
||||
- **Symbol-level alignment with architectural docs.** Reading `verify_cti` instead of `verify_claim_lattice` matches `docs/cti-architecture.md` directly.
|
||||
- **Onboarding clarity for new contributors.** "What is `claim_lattice`?" requires a doc lookup; "what is `cti`?" matches the prose docs more obviously.
|
||||
- **Marketing / external-doc consistency.** If the project ever publishes externally, "CTI" reads as a coherent architecture name; "claim lattice" reads as one implementation detail.
|
||||
|
||||
These are real wins. They're also small relative to the cost in §2.
|
||||
|
||||
## 4. The core argument: an internal vocabulary IS a feature
|
||||
|
||||
Every codebase that survives long enough develops its own internal terminology that doesn't match the architectural prose. Linux kernel "vmas" for "virtual memory areas." Postgres "tuples" for "rows." Git "blobs" for "file contents." That terminology survives because:
|
||||
|
||||
1. **It's what the maintainers mutter about while debugging at 3am.** Naming continuity matters when you're scanning for the symbol that changed three commits ago.
|
||||
2. **It carries history.** Code symbols are historical artifacts as much as design decisions. Renaming them is an act of erasure that disconnects future-you from past-you's reasoning.
|
||||
3. **The architectural label is the right shape for prose; the internal label is the right shape for grep.** They're different ergonomics with different audiences.
|
||||
|
||||
The codebase has its own vocabulary. The architectural doc has its own vocabulary. Both are correct in their domain. The mapping doc (`docs/cti-architecture.md`) is the bridge — read it once, then read code in code's vocabulary and prose in prose's vocabulary.
|
||||
|
||||
## 5. Triggers that would change this decision
|
||||
|
||||
We rename when:
|
||||
|
||||
1. **The cost ratio inverts.** If the codebase grows substantially and onboarding cost (people stuck on "what is claim_lattice?") exceeds the rename's cache-flush + test-churn cost, do the rename.
|
||||
2. **A SCHEMA / cache invalidation is happening anyway.** A schema_version or chunking_version bump already stales all cached records — adding the rename to that migration is free incremental cost.
|
||||
3. **An external publication forces the architectural names.** If a paper / blog / disclosure is published using "CTI" terminology and the codebase needs to match for credibility, do the rename.
|
||||
4. **A PROMETHEUS-Σ refactor extracts an explicit controller module.** If someone needs to write a `prometheus.py` that holds the policy/admissibility/one-shot dispatch logic, the rest of the renames could ride along in one consolidated commit.
|
||||
|
||||
None of those are the case today. So we hold.
|
||||
|
||||
## 6. What we DO commit to
|
||||
|
||||
Independent of the rename decision:
|
||||
|
||||
- **`docs/cti-architecture.md`** — the mapping doc, kept current as the architecture evolves.
|
||||
- **Commit messages use both vocabularies when relevant.** "qa: pointer mode hardening (CTI runtime IR)" is fine; "qa: claim_lattice tweak" is fine; mixing them when the cross-cutting story matters is fine.
|
||||
- **Docstrings reference the architectural labels** where the layer-cake context matters. `verify_claim_lattice`'s docstring should mention "the deterministic verifier in the CTI layer" so a reader following the architectural label can find the symbol.
|
||||
- **CLAUDE.md authority.** When the architectural labels and the internal labels disagree, CLAUDE.md and the docs in `docs/` are authoritative for the layer story; the symbol names are authoritative for the implementation. Neither is wrong.
|
||||
|
||||
## 7. Why this matters for fox's framing
|
||||
|
||||
Fox's de-novo synthesis is correct: CTI / PROMETHEUS-Σ / Merkle-AGI-DAG / Reverse-RAG / v9.8 / Hermes is the right architectural decomposition. Calling that out clearly in `docs/cti-architecture.md` is the right move.
|
||||
|
||||
But the architectural decomposition is a **layered reading** of code that already exists. The code already does CTI; the code already does PROMETHEUS-Σ; the code already does Reverse-RAG. Renaming the symbols doesn't make the architecture more real — it just changes which words name the same machinery.
|
||||
|
||||
The decision to defer is the decision to spend our limited rename-budget when it actually buys something. Today the buy is small and the cost is real. So we name the architecture in docs, and let the code keep its history-rich internal vocabulary.
|
||||
|
||||
If the cost ratio inverts, we revisit. The mapping table in §1 of this doc is the deferred rename plan, ready to execute when one of the §5 triggers fires.
|
||||
|
|
@ -1,169 +0,0 @@
|
|||
# Reference-frame failure class — the Orwell case
|
||||
|
||||
**Date:** 2026-05-01
|
||||
**Scope:** Worked example documenting a failure mode in retrieval +
|
||||
phrase-pattern boost as the response.
|
||||
**Audience:** fox + future blackops shifts.
|
||||
**Status:** investigation log; the phrase-pattern route landed in
|
||||
the same commit window. Not a ticket — this is a journal entry.
|
||||
|
||||
---
|
||||
|
||||
## What happened
|
||||
|
||||
Run 1 (no augmentation):
|
||||
|
||||
```
|
||||
make query Q="has oceania always been at war with east asia"
|
||||
HYBRID 2/16 via claim_lattice
|
||||
sources: Oceania · Asia · Outline of Oceania · Far East · ...
|
||||
```
|
||||
|
||||
The system selected the **literal geography frame** — answered as
|
||||
if asked about real-world Oceania and East Asia regions. The
|
||||
Nineteen Eighty-Four article was nowhere in the top-K despite
|
||||
existing in shard `003.db`.
|
||||
|
||||
Run 2 (manual augmentation — `K=` flag):
|
||||
|
||||
```
|
||||
make query Q="has oceania always been at war with east asia? do you understand what this reference"
|
||||
STRICT 1/1 via claim_lattice
|
||||
sources: Nineteen Eighty-Four (top result)
|
||||
```
|
||||
|
||||
Adding the literal word "reference" to the question pushed the
|
||||
retrieval into the right map. The user solved it themselves with
|
||||
operator hints — but a high-quality system shouldn't need those
|
||||
hints for diagnostic phrases.
|
||||
|
||||
## The diagnosis
|
||||
|
||||
The cause was retrieval-shape, not model behavior:
|
||||
|
||||
```
|
||||
Q tokens (FTS5-tokenized, no stem): {has, oceania, always, been, at, war, with, east, asia}
|
||||
Q content tokens (stopword-stripped): {oceania, war, east, asia}
|
||||
```
|
||||
|
||||
For these content tokens:
|
||||
|
||||
| Article | spin | glass | modeling | tensors | distinct match |
|
||||
|----------------------------|------|-------|----------|---------|----------------|
|
||||
| Nineteen Eighty-Four | 0 | - | - | - | (no overlap on title-tokens at all) |
|
||||
| Oceania | many | - | - | - | 1 token in title (oceania) |
|
||||
| Foreign relations of Axis | - | - | - | - | 0 |
|
||||
|
||||
The **diagnostic signal** wasn't in the content-token overlap. It
|
||||
was in the verbatim 5-token phrase `oceania always been at war`
|
||||
which appears in the 1984 article's body but nowhere else.
|
||||
|
||||
Pre-2026-05-01 retrieval had three routes (body BM25, title-LIKE,
|
||||
core-keyword) — none of them captured "find articles whose body
|
||||
contains a verbatim multi-token sequence from the question." So
|
||||
the 1984 article was effectively invisible to the search.
|
||||
|
||||
## The fix — phrase-pattern retrieval route
|
||||
|
||||
`aborist/qa/query.py:_search_phrases` adds a fourth retrieval
|
||||
route that runs an FTS5 quoted-phrase MATCH for each n-gram
|
||||
extracted from the question:
|
||||
|
||||
```python
|
||||
_question_phrases("has oceania always been at war with east asia", n=5)
|
||||
# → ["has oceania always been at",
|
||||
# "oceania always been at war", ← diagnostic Orwell signal
|
||||
# "always been at war with",
|
||||
# "been at war with east",
|
||||
# "at war with east asia"]
|
||||
```
|
||||
|
||||
Two pass-throughs — n=6 (specificity 100) and n=5 (specificity 90).
|
||||
4-grams were tried and rejected: "always been at war" matches generic
|
||||
war-history articles too noisily for the rerank pipeline to separate.
|
||||
5+ tokens trade recall for precision; most allusions survive at
|
||||
length 5.
|
||||
|
||||
**`_filter_by_title_relevance` accept-path 4** lets phrase-route
|
||||
hits bypass the title-token-overlap gate. The 1984 article's title
|
||||
shares zero tokens with the question — without accept-path 4,
|
||||
phrase-route candidates would be retrieved and immediately filtered
|
||||
out before they could rerank into top-K.
|
||||
|
||||
## Result
|
||||
|
||||
After the fix:
|
||||
|
||||
```
|
||||
make query Q="has oceania always been at war with east asia"
|
||||
EVIDENCE-LINKED · via claim_lattice 1/1 11.5s (fresh)
|
||||
- The text does not directly state that Oceania has always been at
|
||||
war with East Asia. The passage describes a change in alliances,
|
||||
where Oceania switched from being allies with Eastasia to being
|
||||
allies with Eurasia, and the public was manipulated to accept
|
||||
this change without realizing it.
|
||||
[E13 | Nineteen Eighty-Four | 682f0a11: "...To hide such
|
||||
contradictions, history is re-written to explain that the (new)
|
||||
alliance always was so..."]
|
||||
```
|
||||
|
||||
No operator augmentation needed. The Orwell frame surfaces from
|
||||
phrase-route alone.
|
||||
|
||||
## Why not a "Reference Frame Router"?
|
||||
|
||||
An earlier sketch (in fox's review, 2026-05-01 Asia/Kuala_Lumpur)
|
||||
proposed a hand-rolled `REFERENCE_PATTERNS` table mapping query
|
||||
shapes ("has X always been at war with Y") to known references
|
||||
(Orwell). That approach was rejected because:
|
||||
|
||||
- **Scaling**: allusions are long-tail. `winter is coming`,
|
||||
`the cake is a lie`, `may the force be with you`, `to be or
|
||||
not to be` — the catalog is open-ended.
|
||||
- **Maintenance**: per-pattern code rots; new allusions need new
|
||||
rules.
|
||||
- **The corpus already knows**: 1984 article exists in the corpus.
|
||||
The defect was retrieval not surfacing it. Fix retrieval, not
|
||||
add a new pre-retrieval stage.
|
||||
|
||||
Phrase-pattern boost generalizes: any verbatim 5+ token sequence
|
||||
from the question that appears in an article body lifts that
|
||||
article into consideration, regardless of whether it's an Orwell
|
||||
reference, a Star Wars quote, a Hamlet line, or a meme.
|
||||
|
||||
## Bench coverage
|
||||
|
||||
Added to `bench/qa_questions.txt` under `# allusion / reference
|
||||
frame`:
|
||||
|
||||
```
|
||||
has oceania always been at war with east asia?
|
||||
who said may the force be with you?
|
||||
what does winter is coming mean?
|
||||
what is the meaning of rosebud?
|
||||
who said to be or not to be that is the question?
|
||||
what does the cake is a lie reference?
|
||||
```
|
||||
|
||||
A future bench sweep will reveal whether phrase-pattern boost
|
||||
generalizes across these allusions or if some need different
|
||||
treatment.
|
||||
|
||||
## Related concerns NOT addressed here
|
||||
|
||||
- **Source-role taxonomy** — proposed roles like
|
||||
`primary_reference_source` (cited reference work) vs
|
||||
`literal_world_background` (geography article that's tangentially
|
||||
related). Currently all retrieved sources land as
|
||||
`background_source` in the rendered output; finer-grained roles
|
||||
could improve answer framing. Out of scope for this fix.
|
||||
- **Frame-assumption verifier field** — orthogonal to STRICT/HYBRID/
|
||||
UNGROUNDED labels. Records "answered as Orwell reference" vs
|
||||
"answered as geography." Useful for audit but requires verifier
|
||||
schema work; deferred.
|
||||
- **Reference-aware prompting** — system prompt could acknowledge
|
||||
reference-frame ambiguity ("if the query reads as an allusion,
|
||||
prefer the reference frame when retrieved evidence supports it").
|
||||
Per fox's "less prompt engineering, more code-level discipline"
|
||||
preference, deferred until phrase-pattern boost's empirical
|
||||
performance is known.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
# Test coverage audit — fox's §11 list vs codebase
|
||||
|
||||
**Date:** 2026-05-01
|
||||
**Source:** the 16-item test list in fox's de-novo synthesis (the "Best next test suite" section).
|
||||
**Result:** 16/16 covered. Most are already pinned by name; a few are covered indirectly by tests that pin a stronger invariant (e.g. "soft signals never enter run-DAG payload" implies "soft signals can't promote hard status").
|
||||
|
||||
## Per-item coverage
|
||||
|
||||
| # | requirement | status | covered by |
|
||||
|---|-------------|--------|------------|
|
||||
| 1 | claim_lattice_pointer default off | ✓ | `aborist/qa/verify.py:DEFAULT_ANSWER_MODE = "quote"`; `tests/test_claim_lattice.py` exercises both modes via explicit `policy["answer_mode"]` |
|
||||
| 2 | normal quote mode unchanged | ✓ | `tests/test_verify.py` — ~50 path tests on `verify_quotes` (extract_quotes pairing, classify, paraphrase fallback, entity policy, manual-quote handling, framing strip, source-coverage). Quote-mode regressions surface immediately. |
|
||||
| 3 | evidence map built from retrieved chunks | ✓ | `tests/test_claim_lattice.py:test_per_chunk_evidence_map_query_path` (single source → multiple chunks → distinct E1/E2/E3); `test_pointer_id_is_sequential` |
|
||||
| 4 | evidence object has all eight fields | ✓ | `tests/test_claim_lattice.py:test_evidence_id_is_content_addressed_and_stable` + `EvidenceObject` dataclass enforces field presence at construction |
|
||||
| 5 | line with `[E1]` parses | ✓ | `tests/test_claim_lattice.py:test_parse_pointer_claims_basic` |
|
||||
| 6 | line with `[E1,E2]` parses | ✓ | `tests/test_claim_lattice.py:test_parse_handles_whitespace_inside_brackets` parses `[E1, E2 ,E3]` (whitespace + multi-pointer combined); `test_parse_pointer_claims_basic` covers `[E2,E3]` form |
|
||||
| 7 | line without evidence pointer → NO_EVIDENCE_POINTER | ✓ | `tests/test_claim_lattice.py:test_parse_no_evidence_pointer_status` (parser-side); `test_no_evidence_pointer_downgrades` (verifier-side) |
|
||||
| 8 | unknown evidence ID → UNKNOWN_EVIDENCE_ID | ✓ | `tests/test_claim_lattice.py:test_unknown_pointer_id_is_violation`; `tests/test_verify_json.py:test_verify_json_hybrid_when_some_unknown_evidence_id` |
|
||||
| 9 | manual quote → MANUAL_QUOTE_VIOLATION | ⚠️ | The strict no-quote rule was REMOVED from pointer mode in commit `224bfd6` (2026-04-30) per the bench finding that it rejected factually correct claims for cosmetic punctuation. Tests `test_double_quote_in_claim_text_no_longer_blocks_verification` + `test_curly_quotes_also_no_longer_block` document the new behavior. The rule is RETAINED in `verify_claim_lattice_json` (JSON variant); covered by `tests/test_verify_json.py:test_verify_json_manual_quote_violation`. |
|
||||
| 10 | disallowed source role → SOURCE_ROLE_BLOCKED | ✓ | `tests/test_claim_lattice.py:test_source_role_blocked_violation`; `tests/test_verify_json.py:test_verify_json_blocks_disallowed_source_role` |
|
||||
| 11 | renderer uses source offsets, not Hermes text | ✓ | `tests/test_claim_lattice.py:test_verify_renders_with_literal_spans_from_runtime` (renders `em[0].span` from runtime, not model text) |
|
||||
| 12 | rendered quote cannot contain model-inserted `[...]` | ✓ | Synthetic-elision-by-construction-impossible in pointer mode — the model never types quote text. The test in #11 checks the runtime-pulled span; if the model inserted `[...]` it wouldn't appear since rendering uses `em[i].span` not the model's output. |
|
||||
| 13 | verifier_policy_hash differs from quote mode | ✓ | Added 2026-05-01: `tests/test_keys.py:test_verifier_policy_hash_changes_when_verifier_field_changes` (`answer_mode` change → distinct hash). |
|
||||
| 14 | cache does not alias normal and pointer modes | ✓ | `tests/test_qa.py:test_different_model_yields_different_cache_key` covers the 8-dim invariant. The 9-dim form tested by `tests/test_keys.py:test_cache_key_nine_dim_form_distinguishes_verifier_policy` (added 2026-05-01). Both `governance_policy_hash` and `verifier_policy_hash` change with `answer_mode`, so even an 8-dim-only cache_key cannot alias modes. |
|
||||
| 15 | run DAG commits raw answer and parsed lattice separately | ✓ | `tests/test_claim_lattice.py:test_dag_nine_stages_for_pointer` pins the exact stage list including separate `raw_answer` and `parsed_claim_lattice` nodes |
|
||||
| 16 | soft entailment sidecar cannot promote hard status | ✓ | `tests/test_claim_lattice.py:test_lazy_anchor_signals_not_in_run_dag_payload` enforces the architectural invariant: soft signals (pointer_id_distribution, lazy_anchor_ratio) are NOT params of `build_run_dag`'s verify_payload. They live render-layer only. The principle covers any future soft sidecar (NLI semantic-check, predicate compatibility, etc.) — they can demote but cannot promote because they don't enter the proof path. |
|
||||
|
||||
## Item #9 nuance — the manual-quote rule's lifecycle
|
||||
|
||||
Fox's §11 lists "manual quote → MANUAL_QUOTE_VIOLATION" as a required test. The rule existed in pointer mode through commit `bc9438f` (2026-04-30 mid-day) but was removed in `224bfd6` (2026-04-30 evening) after the bench journey identified it was rejecting factually correct claims that had merely preserved source punctuation marks. The current tests (`test_double_quote_in_claim_text_no_longer_blocks_verification`, `test_curly_quotes_also_no_longer_block`) document the *current* behavior, with the change rationale captured in the verifier docstring + commit message + `docs/qa-modes-bench-2026-04-30.md`.
|
||||
|
||||
The JSON-mode variant (`verify_claim_lattice_json`) retains the rule; `tests/test_verify_json.py:test_verify_json_manual_quote_violation` covers it there.
|
||||
|
||||
This is a deliberate divergence between modes, not a coverage gap. JSON mode emits structured `text` fields where double-quotes carry no source-punctuation-preservation rationale; pointer mode emits prose where they do.
|
||||
|
||||
## Recommendations
|
||||
|
||||
- **No new tests needed** to satisfy the §11 list. The substrate is comprehensively covered.
|
||||
- **Document the divergences** when the bench surfaces a behavior-change that retires a rule — the manual-quote rule's lifecycle is a model for how future verifier-rule changes should land in commit + doc + tests simultaneously. The audit doc you're reading IS that documentation pattern.
|
||||
- **Add a periodic re-audit** if fox's external de-novo synthesis grows new test requirements; mapping doc → coverage table is the cheap defense against requirement drift.
|
||||
|
||||
## What this audit DOESN'T cover
|
||||
|
||||
- **Quality of test assertions.** A test that runs and passes is necessary but not sufficient. The bench-maxing discipline in CLAUDE.md handles quality drift; this audit handles requirement coverage.
|
||||
- **Live-fixture coverage.** `tests/test_qa_quality_live.py` is gated and not part of `make test`. Its 21 fixtures are tracked separately in the bench doc.
|
||||
- **Soft-signal layer tests** for designs not yet implemented (NLI sidecar, ProvidenceDistiller). Those tests land with their respective implementations.
|
||||
|
|
@ -239,7 +239,7 @@ To add when this ticket lands:
|
|||
without a per-type rule book. Typed contracts (e.g.
|
||||
`release_date_lookup with required predicate cues`) earn their
|
||||
slot only when bench evidence shows the general primitive misses
|
||||
cases. See `docs/naming-deferral.md`.
|
||||
cases.
|
||||
- **NLI-grade entailment.** Soft signal, separate ticket if/when
|
||||
the verifier-semantic-gap design (`docs/verifier-semantic-gap-design.md`)
|
||||
lands.
|
||||
|
|
@ -68,7 +68,7 @@ Per-row checks:
|
|||
## Closure rationale
|
||||
|
||||
Filed as a closed ticket immediately so the design log captures the
|
||||
artifact alongside the work. The convention from `docs/TICKETS.md`:
|
||||
artifact alongside the work. The convention from `../TICKETS.md`:
|
||||
"Closed tickets stay in place as the design log; do not delete
|
||||
tickets — they are the design log."
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue