arborist/tests
russell@unturf.com c6182ae54e
concept overlay: synonym expansion + rivalry exclusion
A small knowledge-graph layer that does two distinct jobs:

1. SYNONYM_GROUPS broaden retrieval. A query mentioning "Athlon"
   now also matches AMD-titled docs because Athlon IS an AMD product.
   Groups currently cover AMD-family, Intel-family, HTTP family,
   FTP, Mac, Windows, Linux. Easy to extend.

2. RIVALRIES narrow retrieval. The pair (AMD-group, Intel-group)
   means: if the query mentions one side and not the other, drop
   docs whose titles contain the OTHER side's tokens. So a "fastest
   AMD CPU" question never gets Pentium_4 in the context — even if
   FTS5 BM25 ranks it high — because Pentium is in the Intel group
   and Intel isn't in the query.

   COMPARE_WORDS ("vs", "versus", "compare", "between", ...) suppress
   the exclusion. "compare AMD vs Intel" keeps both sides. "what is
   the fastest AMD CPU?" does not.

Demos against the 128k Wikipedia 2003-05-16 cur shards:

  Q "fastest AMD CPU"   → 8 sources, all AMD/CPU titled, NO Intel
                          Answer: "Athlon XP 3200+" (real AMD chip,
                          grounded in the AMD article)

  Q "compare AMD vs Intel"
                        → 8 sources, mix of Intel_8028x, Intel_8048x,
                          AMD_Duron — both sides preserved

  Q "Athlon processor"  → AMD, AMD_Duron, AMD_5x86, Athlon all
                          surfaced via synonym expansion

Phase 1 implementation hand-curates the groups; Phase 2 idea is to
derive them from Wikipedia's link/category graph (dense bidirectional
clusters → synonym groups; same-category-without-cross-links →
rivalry candidates).

66 tests passing.
2026-04-27 12:34:00 -04:00
..
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_concepts.py concept overlay: synonym expansion + rivalry exclusion 2026-04-27 12:34:00 -04:00
test_distill.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill_recursive.py recursive distillation: core -> depth+1 core 2026-04-27 07:54:30 -04:00
test_evict.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_html_source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_ingest.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_qa.py add Q&A layer: v9.8 providence_cache writes with Merkle-bound proofs 2026-04-27 08:01:23 -04:00
test_query.py multi-source corpus query: pose a question, the tree pulls related cached docs 2026-04-27 11:43:57 -04:00
test_resume.py resumable ingest + per-shard audit chain integrity 2026-04-27 11:29:27 -04:00
test_tfidf.py add TF-IDF keyword distiller 2026-04-27 07:55:53 -04:00
test_versioned_ingest.py versioned re-ingest: same URI + changed content writes a 'supersedes' edge 2026-04-27 07:56:49 -04:00
test_wikipedia_old.py prepare full Wikipedia 2003-05-16 ingest: cur + old (revisions) 2026-04-27 08:10:42 -04:00