russell@unturf.com
082143e158
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`, yielding one Document per text file. URI shape `git://<repo>/file/<path>` intentionally omits the commit hash — re-ingest after new commits produces fresh document_roots that aborist's prior-doc detection chains via `supersedes` edges, so the audit trail and Merkle tree grow as the repo grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files filtered out by default. Commit hash + timestamp + subject ride along in `extra` (informational only; not part of the Merkle commitment). MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes semantics, same shape. Makefile adds: make ingest-self (this repo -> aborist-self.db) make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone) make ingest-hg HG_REPO=/path/to/repo (mercurial) Each lands in its own shard file alongside existing shards/grok.db, keeping per-shard write paths independent of the wikipedia 4-way ingest's WAL writer lock. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| grok.py | ||
| html_page.py | ||
| vcs.py | ||
| wikipedia.py | ||
| wikipedia_xml.py | ||