russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
80 lines
2.7 KiB
Markdown
80 lines
2.7 KiB
Markdown
# `aborist.sources`
|
||
|
||
Corpus producers. Each is a `Source` ABC implementation that yields
|
||
`Document` instances; the standard `ingest.ingest_source(source, db)`
|
||
pipeline takes them from there.
|
||
|
||
The `Source` ABC lives in [`aborist/source.py`](../../aborist/source.py):
|
||
|
||
```python
|
||
class Source(ABC):
|
||
@abstractmethod
|
||
def iter_documents(self) -> Iterator[Document]: ...
|
||
```
|
||
|
||
## Built-in sources
|
||
|
||
### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)
|
||
|
||
Streams the Wikipedia 2003 `cur` (current revisions) and `old`
|
||
(revision history) SQL dumps. Hand-rolled escape-aware parser
|
||
(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
|
||
char-by-char loops via `str.find` + slicing. cProfile any change.
|
||
|
||
Default Wikipedia 2003-05-16 dump source:
|
||
`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
|
||
robots.txt returned 404 → no rules.
|
||
|
||
[`aborist/sources/wikipedia.py`](../../aborist/sources/wikipedia.py)
|
||
|
||
### `wikipedia_xml.py` — Phase IV XML dumps
|
||
|
||
Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
|
||
`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
|
||
to stream-parse without loading the whole tree.
|
||
|
||
[`aborist/sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py)
|
||
|
||
### `html_page.py` — single-URL or URL-list HTML ingest
|
||
|
||
Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
|
||
HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
|
||
the main body text + every `<a href>` as an `Edge` row.
|
||
|
||
The `edges` rows are what the corpus-derived synonym extractor
|
||
later reads — no separate crawler needed for site-internal link
|
||
graphs.
|
||
|
||
Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.
|
||
|
||
[`aborist/sources/html_page.py`](../../aborist/sources/html_page.py)
|
||
|
||
### `crawler/` — async BFS web crawl
|
||
|
||
Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
|
||
bridge. BFS-discovers same-domain URLs from a seed, respecting
|
||
`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
|
||
into `document_http_meta` so a future recrawl can send conditional
|
||
HEAD requests.
|
||
|
||
[`aborist/sources/crawler/bridge.py`](../../aborist/sources/crawler/bridge.py)
|
||
|
||
### `grok.py` — xAI Grok export
|
||
|
||
Reads the `xAI-conversations.json` data export shape. Each
|
||
conversation becomes one `Document`; media prompts are kept
|
||
inline.
|
||
|
||
[`aborist/sources/grok.py`](../../aborist/sources/grok.py)
|
||
|
||
### `vcs.py` — git + Mercurial repositories
|
||
|
||
HEAD walk. Each commit becomes a `Document` (commit message + diff
|
||
stat). The supersedes chain captures commit ancestry as edges.
|
||
|
||
[`aborist/sources/vcs.py`](../../aborist/sources/vcs.py)
|
||
|
||
## Source
|
||
|
||
[`aborist/sources/`](../../aborist/sources/) ·
|
||
[`aborist/source.py`](../../aborist/source.py) (ABC)
|