# Ticket #000040 — Phase 5 resolver fix (content-token strategy)

**Status:** closed · cascade landed 2026-05-09; expected lift blocked by terminology mismatch (Hilbert 1902 says "connection", claim-pack says "incidence") — see §6; follow-up tracked as #000042 term-aliases
**Opened:** 2026-05-09
**Scope:** Fix `arborist.qa.warrant_resolver` for the failure
mode noted in `#000038` §6: 7 of the 18 Hilbert pillar-IV axiom
records fail to resolve because their discriminating tokens
("Incidence", "Plane", "Line") are too common in the cited
textbook for BM25 to pick a winning chunk. Add a phrase-first
+ content-token-fallback strategy so the resolver finds
per-axiom chunks even when the title's individual tokens are
ubiquitous.
**Audience:** future blackops shifts running warrant promotion
on more textbooks.
**Hard constraint:** **same fail-closed honesty as #000031.**
A failed match still produces no derivations row. The fix
expands what counts as a SUCCESSFUL match; it doesn't lower
the threshold for false positives.

---

## 1. Problem statement

After `#000031` Phase 1+2+3 landed, 11 of 18 Hilbert pillar-IV
axiom records resolved cleanly. The 7 misses are:

  Axiom of Line Incidence
  Axiom of Plane Incidence
  Axiom of Point-Line Incidence
  Axiom of Point-Plane Incidence
  Axiom of Non-Triviality
  Axiom of Side-Angle-Side (SAS)
  Euclidean Parallel Postulate

The current resolver (commit `69e0a95`) extracts discriminating
tokens from the theorem name (drops categorical "axiom" /
"theorem" / "principle"), AND-joins them as the FTS5 query.
For "Axiom of Line Incidence" → tokens `Line` + `Incidence`.

Both tokens occur on virtually every page of Hilbert's
*Foundations of Geometry* — the entire Group I chapter is
"Axioms of Incidence" naming straight lines and points.
FTS5 returns dozens of candidate chunks; BM25 ranks by IDF
weight; no clear winner emerges; the matcher backs off and
returns no match.

This is the false-negative case. The textbook IS in the shard.
The axiom IS in the textbook. The resolver just can't find the
right chunk because its query is too generic.

## 2. Design choices

### 2.1 Strategy options

**A. Phrase-match first.** Many axiom names have a literal
discriminating phrase: "Line Incidence", "Plane Incidence",
"Point-Line Incidence". FTS5 phrase syntax `"line incidence"`
ranks chunks that contain those exact two words adjacently.
Fast, cheap, deterministic.

  Limitation: "Side-Angle-Side" and "Euclidean Parallel
  Postulate" might not appear verbatim either. Need a fallback.

**B. Use claim-pack chunk content as the query.** Each claim-
pack record's chunk has the Δ formula + ∇verbose prose, which
collectively contain WAY more discriminating language than the
title. Extract the most-distinctive 5-10 tokens from that
content (TF-IDF against a corpus baseline, or just
length-based + uniqueness heuristics) and use those as the
FTS5 query.

  Limitation: requires reading the claim-pack chunk content
  per resolver call. Cheap in practice (1 SQL query per record).

**C. TF-IDF over BM25 ranking.** Replace BM25 with TF-IDF that
weights tokens by their rarity within the textbook itself, not
just within the global corpus. "Incidence" is rare even within
Hilbert's text once you consider it discriminates Group I
chapters from Group V chapters.

  Limitation: requires a per-shard token-frequency index.
  More machinery than warranted; defer.

**D. Match on Δ-formula tokens.** The Δ field of an axiom is
a LaTeX formula like `A \to B \to A`. Most chunks won't
contain `\to` at FTS5 token level (LaTeX backslash + `to`).
This is too narrow — the cited textbook prose typically
DOES NOT have the same LaTeX formulas as the curriculum-shaped
claim-pack record.

→ Recommended: **A then B** as a layered cascade. Phrase match
first; if no result, content-token match. Skip C and D for
this iteration.

### 2.2 What "discriminating" means in the content-token path

For option B, "discriminating" is heuristic. Reasonable rules:

- Token length ≥ 5 (drops short common words).
- Drop generic theorem terms (already in `_GENERIC_THEOREM_TERMS`).
- Drop top-50 most common English words (we can hardcode a
  small list).
- Prefer proper nouns (capitalized in original; lowercased for
  FTS5).
- Cap to top-N tokens by some uniqueness score; AND-join.

A cheap uniqueness signal: token's frequency within the chunk
content vs. its frequency across all claim-pack chunks. Tokens
that appear in ONE claim-pack record but not in others are
discriminating per-record. Pre-compute this.

### 2.3 What this doesn't change

- The author-last-name + title-token shard match
  (`_shard_matches_citation`) stays as-is; this ticket only
  changes the per-record FTS5 query inside resolved shards.
- The fail-closed allow-list and threshold logic stays.
- The proof-blob writer stays.

## 3. Recommendation

Implement **A then B** in a new helper `_build_record_query`
that takes the claim-pack record's title AND chunk content
(plus the citation), tries phrase-match on the most
discriminating 2-3-word phrase, falls back to content-token
AND-join when phrase yields zero.

Re-run `arborist warrant-resolve --shards-dir ~/.arborist/shards
--write` after the fix. Expected: 7 more Hilbert records
resolve → 18 of 92 chains.

## 4. Implementation sketch

```python
def _phrase_for_axiom(theorem_name: str) -> str:
    """Pick a 2-3 word discriminating phrase from the theorem name.

    "Axiom of Line Incidence" → "line incidence"
    "Axiom of Plane Incidence" → "plane incidence"
    "Pasch's Axiom" → "" (no multi-word phrase; falls through to
                          single-token AND)
    """
    # Strip leading "Axiom of " / "Theorem " / etc.
    # Take what's left; lowercase + drop generic terms.
    ...


def _content_tokens(chunk_content: str, max_n: int = 8) -> list[str]:
    """Pull discriminating tokens from a claim-pack chunk content.

    Uses (a) lengths >= 5, (b) drop generic + stopword lists,
    (c) prefer proper nouns. Returns top-N uniquely-ranked.
    """
    ...


def _build_record_query(
    citation: Citation,
    record_title: str,
    record_content: str,
) -> list[str]:
    """Returns a list of FTS5 queries to try in order.
    First non-empty match wins.

    Order:
      1. Phrase from title (e.g., '"line incidence"')
      2. Discriminating tokens from chunk content (AND-joined)
      3. Existing discriminating tokens from title (AND-joined)
      4. Title tokens OR-joined (existing fallback)
    """
    ...
```

Resolver loop tries each query; first to yield a chunk match
wins.

## 5. Hard constraints (re-stated)

1. Failed match still produces no derivations row.
2. Phrase match doesn't introduce false positives — phrase
   syntax is more selective, not less.
3. Content-token strategy uses the SAME data path
   (claim-pack chunks already in shard) — no schema change.
4. Idempotent: same input → same output → re-running is
   no-op at DB layer (PK on derivations).

## 6. Status

**Cascade landed 2026-05-09 in commit `b9e5bbd`; zero impact on
Hilbert pillar IV records — root cause is terminology, not
ranking.** The cascade itself is correct and ready for
textbooks that share vocabulary with the claim-pack record's
modern naming.

The 7 still-unresolved Hilbert records are blocked by a
different problem:

**Terminology mismatch.** Claim-pack records (g4-generated
2025) use modern post-1950s names: "Axiom of Line Incidence",
"Axioms of Incidence" (Group I). Hilbert's 1902 Townsend
translation uses the older "Verknüpfung" / "axioms of
connection" — Hilbert's own original term, before the field
adopted "incidence" as the standard rendering.

Verified empirically: the literal token "incidence" appears
**zero times** in the ingested Hilbert TeX surface;
"connection" is the relevant synonym.

Phase 5's cascade (phrase → content-tokens → discriminating-
tokens → OR-fallback) tries each strategy in order, but
every strategy looks for tokens that the TEXTBOOK doesn't
contain. No matter how clever the query, you can't find a
word that isn't there.

**Two paths forward (out of this ticket's scope):**

1. **Add a more recent Hilbert translation** that uses
   modern "incidence" terminology. Robert Bernhard's 1971
   edition does; in copyright. Goes in `#000038` Phase 4
   license decisions.
2. **Term-alias layer** — `arborist term_aliases` table
   mapping `("incidence", "geometry") → ("connection",
   "geometry")` AND'd with the existing query at warrant-
   resolve time. Sibling design to the citation-alias
   proposal in `#000041`. File as `#000042` once Phase 4
   surfaces concrete need-cases.

The Phase 5 cascade still pulls its weight for any future
textbook where cited vocabulary matches textbook prose —
Stanley, Brualdi, Knuth, etc. (modern works using modern
terminology). The implementation is preserved for that
future landing.

Estimated-size predictions held: ~150 LOC implementation +
the helpers + filter lists. Future-ticket #000041 (term-
alias) is the next dependency on this chain.
