# `aborist.search`

The retrieval primitive. Today's only backend is FTS5 over the
chunks table; the `SearchBackend` ABC is in place so additional
backends (BM25 over titles, embedding-based vector search) can be
added without touching the rest of the QA pipeline.

## `SearchBackend` ABC

```python
from aborist.search import SearchBackend, AuditMode, Hit

class SearchBackend(ABC):
    @abstractmethod
    def search(self, query: str, limit: int = 20) -> list[Hit]: ...
```

Each `Hit` carries `(document_root, document_uri, chunk_idx,
snippet, score, audit_mode, title)`. `audit_mode` is the sticky
provenance label that tracks how the chunk made it into the index;
FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify
anything — that's the QA pipeline's job).

## `FTS5Backend`

Wraps the contentless `chunks_fts` virtual table. Two-mode query:

- **AND-mode (strict, primary):** every content token must appear
  in the doc. Keeps unrelated docs out of the context window.
- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR
  but **capped to top-5 longest tokens** (proxy for rarity). Long
  topical synonyms fed via `extra_or_tokens` join the pool —
  `neurotechnology` (15 chars) outranks `thoughts` (8) by length
  and surfaces brain-tech titles for vocabulary-mismatch queries.

```python
from aborist.search import FTS5Backend
backend = FTS5Backend(conn)

# Plain search
hits = backend.search("permacomputer", limit=32)

# Search with synonym pool injection (used by qa.query._search_corpus)
hits = backend.search(
    long_query,
    limit=32,
    extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
)
```

## Stopword & stopword-cap discipline

`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`)
+ generic connectors (`one`, `some`, `another`, `without`, `soon`,
`currently`) before AND/OR construction. Two principles:

- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token
  filtered at retrieval time but kept at title-relevance check (or
  vice versa) creates ranking incoherence.
- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without
  this, a 19-token OR clause matches millions of docs and forces
  BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.

## Snippet building

FTS5 contentless mode means SQLite's built-in `snippet()` and
`highlight()` return empty. Aborist builds snippets in Python by
joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the
chunk content, and locating query tokens locally
(`_build_snippet`).

## Source

[`aborist/search/fts5.py`](../../aborist/search/fts5.py) ·
[`aborist/search/__init__.py`](../../aborist/search/__init__.py)