russell@unturf.com
5fd458aa41
Phase 2 of the concepts/ layer. The Apr 27 commit (c6182ae) shipped hand-curated frozensets in aborist/qa/concepts.py with a TODO to "derive from Wikipedia's category graph or 'See also' sections" — that's this commit. Fox's 2026-05-01 critique landed it: a 7-entry list of arbitrary frozensets won't scale to a 3.47M-doc corpus, & adding domains shouldn't require a Python edit + commit + redeploy. Architecture: (1) Per-shard concept_relations SQLite table. Append-only, with UNIQUE (source_root, relation_kind, token, target, evidence_kind) so re-derivation is idempotent. Lives next to documents in each shard so mesh sync moves relations alongside the docs that derived them. A SECONDARY index — writes here NEVER affect document_root / chunk_root / cache_key, so backfilling is safe across the entire corpus. (2) aborist/concepts/ package: - store.py: add_concept_relation, concept_relations_for_token, purge_by_evidence_kind, list_evidence_kinds - query.py: cross-shard synonym_expand, rivalry_excluded; union-find collapse on synonym edges so partial pairs build full equivalence classes; mtime-keyed per-process LRU so retrieval doesn't re-walk shards on hot loops - seed.py: one-shot migration of legacy frozensets (8 groups incl. brain-tech) to evidence_kind='manual_legacy' rows under source_root='__legacy__concepts__' - extract.py: pluggable extractor registry. Built-in: link_reciprocity_synonym — for any reciprocal edge pair (A→B AND B→A) in the existing edges table, emit synonym edges between the docs' title-tokens. Works for Wikipedia (See-also bidirectional), HTML site internal-link clusters (russell.ballestrini.net pattern), or any link graph the corpus already encodes — no new crawler needed; the html_page parser already populates `edges` rows on ingest. (3) aborist/qa/concepts.py rewritten as a backwards-compat shim — same public API (synonym_expand, rivalry_excluded, has_compare_phrasing) so query.py call sites unchanged. shards_dir threaded through _filter_by_title_relevance & _search_corpus' synonym_expand calls. Without shards_dir (legacy 2-arg call shape), helpers degenerate to no-op — matches the behavior the frozenset code had when no group hit. (4) Cross-shard UNION view: concept_relations added to _SHARDABLE_TABLES in store.py so connect_query() exposes a unified view across all shards (same pattern as documents, chunks, providence_cache, etc.). Live-verified on the 4-shard 3.47M-doc corpus + the crawl_russell_ballestrini_net.db shard: Q: what technology are currently or soon available which may enable one person to reconstruct and understand some or a portion of another persons thoughts or ideas without speaking or sign language? Result: same as thebde1bd6in-memory frozenset version — Telepathy E4 cited alongside Videoconferencing E1/E2, POINTER-LINKED-PARTIAL 6/10. The DB-backed lookup reproduces the frozenset behavior byte-for-byte. Tests: 633 passed (was 624). 9 new concept tests covering DB-backed synonym/rivalry lookup, shards_dir=None degenerate behavior, brain-tech group seed, cache invalidation. Old tests that called helpers directly without shards_dir kept as no-op assertions (synonym_expand({"athlon"}) without shards_dir returns {"athlon"} unchanged). Backfill mechanics: existing live shards needed a one-shot `connect()` to auto-create the new concept_relations table (SCHEMA_SQL has CREATE TABLE IF NOT EXISTS). Re-derivation never mutates the Merkle tree — concept_relations is fully orthogonal to document_root / chunk_root. Cache_key dimensions are unaffected. Deferred (follow-on): - CLI commands: aborist concepts {seed,list,add,derive,purge} (currently fox runs the helpers via python -c) - Wikipedia See-also extractor (requires parsing wikitext sections beyond what's already in edges) - Wikipedia category extractor (requires reading Category: links from chunked wikitext)
68 lines
2.3 KiB
Python
68 lines
2.3 KiB
Python
"""Backwards-compat shim — public API delegates to ``aborist.concepts``.
|
|
|
|
The actual data lived as hand-curated frozensets in this module
|
|
through April 2026 (commit c6182ae) and one entry was added in
|
|
2026-05-01 (commit bde1bd6 — mind/brain-tech group). 2026-05-01 the
|
|
data layer moved to a per-shard ``concept_relations`` SQLite table
|
|
(see aborist/concepts/__init__.py for the rationale).
|
|
|
|
This shim preserves the call-site signatures `query.py` already uses
|
|
(``synonym_expand(qtokens)`` & ``rivalry_excluded(qtokens, compare_phrasing=...)``)
|
|
so the migration is a pure-implementation swap with no API change.
|
|
|
|
The new implementation needs a ``shards_dir`` to know which shards to
|
|
walk. The runtime threads it through query() already (kwarg passes
|
|
through), so the bridging happens here: callers that don't pass
|
|
``shards_dir`` get the empty-set degenerate behavior, which matches
|
|
the legacy code path when no synonym/rivalry was applicable.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
from aborist.concepts.query import (
|
|
has_compare_phrasing,
|
|
rivalry_excluded as _rivalry_excluded_impl,
|
|
synonym_expand as _synonym_expand_impl,
|
|
)
|
|
|
|
__all__ = ["has_compare_phrasing", "rivalry_excluded", "synonym_expand"]
|
|
|
|
|
|
def synonym_expand(
|
|
tokens: set[str],
|
|
*,
|
|
shards_dir: Path | str | None = None,
|
|
) -> set[str]:
|
|
"""Add all synonym-group members for any token that hits a group.
|
|
|
|
When ``shards_dir`` is None (the legacy two-arg call shape), returns
|
|
the input tokens unchanged. The runtime threads ``shards_dir``
|
|
through retrieval — call sites that pass it get the corpus-derived
|
|
expansion; tests that don't get a no-op.
|
|
"""
|
|
if shards_dir is None:
|
|
return set(tokens)
|
|
return _synonym_expand_impl(tokens, shards_dir=shards_dir)
|
|
|
|
|
|
def rivalry_excluded(
|
|
tokens: set[str],
|
|
compare_phrasing: bool = False,
|
|
*,
|
|
shards_dir: Path | str | None = None,
|
|
) -> set[str]:
|
|
"""Tokens whose presence in a doc title means EXCLUDE that doc.
|
|
|
|
Same call shape as the legacy positional (tokens, compare_phrasing)
|
|
so existing callers don't break. New calls pass ``shards_dir``
|
|
via kwarg to enable corpus-derived rivalries.
|
|
"""
|
|
if shards_dir is None:
|
|
return set()
|
|
return _rivalry_excluded_impl(
|
|
tokens,
|
|
shards_dir=shards_dir,
|
|
compare_phrasing=compare_phrasing,
|
|
)
|