Commit graph

1 commit

Author SHA1 Message Date
082143e158
add git and mercurial repo sources for self-play
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).

MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.

Makefile adds:
  make ingest-self                          (this repo -> aborist-self.db)
  make ingest-git GIT_REPO=/path/to/repo    (arbitrary git clone)
  make ingest-hg  HG_REPO=/path/to/repo     (mercurial)

Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
2026-04-27 18:17:39 -04:00