arborist/tests
russell@unturf.com 082143e158
add git and mercurial repo sources for self-play
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).

MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.

Makefile adds:
  make ingest-self                          (this repo -> aborist-self.db)
  make ingest-git GIT_REPO=/path/to/repo    (arbitrary git clone)
  make ingest-hg  HG_REPO=/path/to/repo     (mercurial)

Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
2026-04-27 18:17:39 -04:00
..
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
test_concepts.py concept overlay: synonym expansion + rivalry exclusion 2026-04-27 12:34:00 -04:00
test_distill.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill_recursive.py recursive distillation: core -> depth+1 core 2026-04-27 07:54:30 -04:00
test_evict.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_grok_source.py add Grok export source: conversations + media posts 2026-04-27 13:49:42 -04:00
test_html_source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_ingest.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_qa.py add Q&A layer: v9.8 providence_cache writes with Merkle-bound proofs 2026-04-27 08:01:23 -04:00
test_query.py multi-source corpus query: pose a question, the tree pulls related cached docs 2026-04-27 11:43:57 -04:00
test_resume.py resumable ingest + per-shard audit chain integrity 2026-04-27 11:29:27 -04:00
test_tfidf.py add TF-IDF keyword distiller 2026-04-27 07:55:53 -04:00
test_vcs_source.py add git and mercurial repo sources for self-play 2026-04-27 18:17:39 -04:00
test_versioned_ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
test_wikipedia_old.py prepare full Wikipedia 2003-05-16 ingest: cur + old (revisions) 2026-04-27 08:10:42 -04:00
test_wikipedia_xml.py add Phase IV Wikipedia XML + abstract sources 2026-04-27 17:24:28 -04:00