Per fox: a query for "who is Russell Ballestrini" classified STRICT by
grounding 6/6 quotes against `feeds/all.atom.xml` — a 230-chunk dump
of post metadata. The verifier was technically correct (every quoted
string IS in the feed) but the result was hollow: feeds list URLs,
they don't carry knowledge. The ACTUAL bios live at the linked posts.
Two-pass filter in _CrawledHtmlSource.iter_documents:
1. Pre-fetch URL-pattern check (_looks_like_feed_url):
- suffix matches: .atom, .atom.xml, .rss, .rss.xml, .rdf,
/feed.xml, /atom.xml, /rss.xml, /rss2.xml, wp-rss2.xml,
wp-atom.xml, wp-rdf.xml, wp-rss.xml
- substring matches in path: /feed/, /feeds/, /atom, /rss, /sitemap
2. Post-fetch Content-Type check (_looks_like_feed_response):
- rejects: application/atom+xml, application/rss+xml,
application/rdf+xml, application/xml, text/xml
- keeps: application/xhtml+xml (xhtml IS html, just stricter syntax)
Defense in depth — a feed served at a non-feed path (e.g.
/index.html returning text/xml) still gets dropped on Content-Type.
What stays out of the corpus:
/feeds/all.atom.xml (Atom feeds)
/sitemap.xml (XML sitemaps)
/wp-rss2.xml (WordPress RSS)
/feed/ (any feed alias)
What's still allowed:
/post-name/ (real prose pages)
/index.html (HTML)
/page.xhtml (xhtml)
Tests: 12 new parametric cases (8 path patterns + 6 Content-Types +
xhtml positive case). 36 bridge tests + 273 default suite, all
passing.
For fox: the existing feed entry already in the crawl shard was burned
manually via `aborist burn --kind document --root eaf7c8d5...` — 230
chunks gone. Re-running `make crawl-ingest` won't re-introduce it.
The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:
- timeouts: 5s vs 15-60s
- parallel page workers: cpu_count() * 3 vs 1
- ignores robots.txt 'crawl-delay' directive
- robots.txt 'Disallow' is STILL honored (only the delay drops)
Surface:
aborist crawl --seed-url ... --fast
make crawl-ingest URL=... FAST=1
Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
A real BFS hits sitemap.xml / RSS feeds via the verbatim crawler's
generic page handler. bs4 warns "you're using an HTML parser on XML"
on each — benign (parsing still works) but spams stderr during a
deep crawl, drowning the progress heartbeat we just wired in.
Filter at aborist/sources/crawler/__init__.py so the lifted source
stays untouched. The filter applies whenever any caller imports from
this subpackage, including the CLI's `from ... bridge import ...`
(which runs __init__.py first per Python import semantics).
A no-cap crawl could take many seconds with no feedback. Wires the
existing aborist.progress.Progress into both phases so stderr shows
heartbeats every 2s (Progress's default interval).
Discovery phase: prints prefix='crawl', counts discovered URLs,
shows queue depth as the secondary number ('inserted' slot in the
Progress format — works fine, semantically "still to do").
Ingest phase: prefix='ingest', total_estimate=len(urls) so the user
sees percent + ETA. Threads through ingest_source's existing
progress= parameter.
Plus three banner lines to stderr at phase boundaries (start crawl,
end discovery, start ingest) so even sub-2s crawls show signs of
life. All flushed via Progress's flush=True path; stderr is
line-buffered by default so this works without PYTHONUNBUFFERED.
Tests stay green (Progress goes to stderr, pytest captures only the
stdout summary). 19 bridge tests + 273 default suite, all passing.
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:
aborist crawl --depth 2 # no page cap
aborist crawl --depth 2 --max-pages 50 # opt-in cap when needed
make crawl-ingest URL=... DEPTH=2 # no page cap
make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap
Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.
Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
The verbatim AsyncWebFetcher.fetch_webpage defaults extract_links=False
and returns (html, []) when omitted. Bridge's BFS therefore terminated
at the seed — a real-world crawl of russell.ballestrini.net discovered
exactly 1 URL (the seed). One-line fix; the kwarg was always there in
the lifted API.
Added test_crawl_seed_passes_extract_links_true that records every
fetch_webpage call and asserts the kwarg is True. Pin so this can't
silently regress when bridge internals get reorganized.
Updated the stub fetcher signature in the existing tests to accept
the new kwarg without behavior change.
Glues the verbatim crawler lift to aborist's existing ingest pipeline,
plus adds a freshness probe that asks "does this need recrawling?"
without paying for a body fetch when the answer is no.
Bridge (aborist/sources/crawler/bridge.py):
- crawl_seed(seed, depth, max_pages): BFS-discovers same-domain URLs
via AsyncWebFetcher. Exact-hostname filter so subdomain crossover is
off by default. Caps at max_pages. URL dedup, fragment-strip
canonicalization. Sync wrapper around an async core so the CLI
remains a one-liner.
- ingest_crawled(conn, urls): runs the URL list through the standard
ingest_source() path so chunking_version / canonicalization_version
/ Merkle commitment all stay consistent. Captures ETag +
Last-Modified per response and writes one row per document_root
into the new document_http_meta table.
Schema (idempotent migration, mirrors _migrate_mesh_peer_chains):
document_http_meta (
document_root TEXT PRIMARY KEY,
etag TEXT,
last_modified TEXT,
last_fetched_at INTEGER NOT NULL,
last_status INTEGER,
last_checked_at INTEGER,
FK(document_root) -> documents(document_root) ON DELETE CASCADE
)
Recrawl-check (recrawl_check):
- Sends conditional HEAD with If-None-Match + If-Modified-Since.
304 -> fresh (no body transfer needed). 200 -> stale (server says
the body changed). 404/410 -> gone. Other -> unreachable.
- Updates last_status + last_checked_at so consecutive runs visit
oldest-first (NULLS FIRST in ORDER BY).
- Optional --domain filter so operators can probe one site at a time.
CLI:
aborist crawl --seed-url URL [--depth 2] [--max-pages 20] [--ingest]
aborist crawler recrawl-check [--domain DOMAIN] [--limit 100]
Makefile:
crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]
recrawl-check [DOMAIN=example.com LIMIT=100]
Both targets gate on bootstrap-crawler so [crawler] extras land first.
Tests: 17 new in tests/crawler/test_bridge.py — same-domain filter,
BFS bounds (depth + max_pages), dedup, fragment normalization, ETag
capture, content-type filter, HEAD classification (304/200/404),
domain filter, last_checked_at update. All stub the network via a
fake httpx.Client; default suite still 273 passed, 1 skipped.
End-to-end test path:
make bootstrap-crawler
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 MAX=20
make query Q="who is russell ballestrini?"
make recrawl-check DOMAIN=russell.ballestrini.net
Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:
core/async_web_fetcher.py -> aborist/sources/crawler/async_web_fetcher.py
core/web_fetch.py -> aborist/sources/crawler/web_fetch.py
Two source-side changes during the lift:
1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
`import aborist.sources.crawler` raises ImportError with the install
hint instead of leaking AttributeErrors deep in user code.
2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
`has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
replaced with NotImplementedError stubs. Aborist has no chat
surface — fetch intent is detected at the application layer. The
two test classes that exercised these triggers are
`@pytest.mark.skip`'d with the same rationale.
Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.
Off by default:
- `[crawler]` extras section in pyproject.toml carries the heavy
deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.
Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2
and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus
pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields
pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly
via iterparse with bounded memory; same shard / resume contract as the SQL
source. Falls back to title-prefix namespace filtering when older export
schemas omit the per-page <ns> element (e.g. enwiki 20101011).
Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR /
WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot.
aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
The optimized parser silently dropped articles whose body contained an
HTML decimal entity (&#NNNNN;) — affecting roughly any Wikipedia article
referencing a Japanese/Korean/Chinese interwiki link. ~6,000 docs missing
from the 128k cur ingest. The HTTP article was one casualty.
Root cause: when the scanner ran out of pending data while inside a
quoted string, flush_complete() returned with in_string=True. The outer
read-loop trimmed pending and called flush_complete() again, which RESET
i=0 and searched for the next "'" from the start of the buffer. The
first quote found was the title-field open quote of the article we were
already deep inside; bs=0 (no preceding backslash) so the parser
incorrectly flipped in_string=False and treated the next ';' (typically
inside an &#NNNNN; entity) as a statement terminator.
Fix: persist scan_pos across read() chunks. After a trim, remap
scan_pos = scan_pos - yield_from. The next call resumes scanning at the
correct position rather than restarting at 0.
Verified: optimized parser now yields identical statements to the
canonical char-by-char reference on the first 60 MB of the dump. HTTP
article correctly emitted at doc #7722 with full 2375-char body.
Cost: ~50% slower on a 5000-doc benchmark (11.8s vs broken 7.8s) due
to the correct cross-chunk state tracking. Worth the trade — a corpus
that randomly drops articles is worse than a slightly slower complete
one. Still ~3x faster than the original char-by-char parser.
The existing 122k cur shard set is missing the same ~6k articles. To
recover: rm -rf ~/.aborist/shards && make ingest-cur-attached SHARDS=4
takes ~3-4 min wall.
57 tests still passing.
Two issues surfaced by the user query
"tell me about back to the future plot?":
1. Mojibake in titles like 'La_Jet\\ufffde'.
The 2003 MediaWiki dumps mix Latin-1 raw bytes (e.g., 0xE9 for é)
with HTML entities. Opening the bz2 as utf-8 with errors="replace"
substitutes U+FFFD for any high byte. Latin-1 is lossless on every
byte and gives the right code point for the raw-byte cases. New
default for WikipediaSqlDump; modern dumps can pass encoding="utf-8"
explicitly.
The existing 122k cur shards keep their slightly-mangled titles
on a handful of articles — but a re-ingest from scratch (now ~2.5
min thanks to the parallel attached path) recovers them, and
--resume protects against starting over from zero.
2. Tell_(poker) outranking Back_to_the_Future for "tell me about
back to the future plot?" — BM25 on body text alone favors
short docs with rare body tokens. The title is a much stronger
topical signal.
New _rerank_by_title in qa/query.py: hits whose title contains
query tokens get +8 per match. After rerank on this query:
1. Back_to_the_Future 31.74 (was 15.74, was rank 3)
2. Back_to_the_Future_Part_III 27.83
3. Back_to_the_Future_Part_II 27.63
4. Tell_(poker) 23.41 (was 15.41, was rank 6)
5. Gunpowder_Plot 19.57
Hermes now produces a real plot summary because the canonical
Back_to_the_Future article is the lead source.
57 tests still passing.
Resume (rsync-style)
Each shard DB gains a `meta` table. After every successful batch
flush, ingest_source persists `source_high_water:<source_type>` ->
the largest row id seen (cur_id or old_id). On --resume, the source
reads it back and skips rows whose id is <= the mark, so an ingest
killed at any point can be re-run cheaply: already-cached docs are
fast-forwarded past without re-hashing or DB writes.
WikipediaSqlDump now exposes `start_id` (skip threshold) and
`last_id` (running max). cur_id is surfaced in Document.extra
alongside old_id so both tables behave the same.
CLI: aborist ingest --resume
Demo on shared DB (cur):
Round 1: --limit 3000 --resume high_water = 5714
Round 2: --limit 5000 --resume skips 1..5714, picks up at 5715
Round 3: --limit 100 --resume skips 1..15362, ingests 100 more
Always idempotent on re-run; no dups, no missing docs.
Per-shard audit chain integrity for sharded analyze
Cross-shard analyze previously reported nonsense breaks counts —
each shard owns its own audit chain (genesis -> ... -> latest), and
the UNION view interleaves them so cross-shard transitions look
like break events. The fix: when --shards-dir is set, open each
shard's DB directly and run _check_audit_chain on it, then
aggregate.
Output now reads:
"audit_chain": {
"events": <total>,
"breaks": 0,
"shards": [{"shard": "000.db", "events": N, "breaks": 0}, ...]
}
53 tests passing. Tests cover: high-water write, skip-on-resume,
idempotency across two resume runs, kill-and-resume continuity.
Workers each parse the full SQL stream but only emit rows whose index
satisfies (idx % total == rank). N processes cooperate against a
single shared SQLite file; WAL+synchronous=NORMAL handles writer
contention via BEGIN IMMEDIATE serialization.
Implementation chose OS-level processes over in-process mp.Queue: the
gain ceiling is identical (writer is the bottleneck either way) and
the surface area is one new flag plus stride-filter logic instead of
a queue-based pipeline with picklable workers.
Makefile gains:
ingest-cur-parallel SHARDS=4
ingest-old-parallel SHARDS=4
Benchmark on 20k cur articles:
serial 55.3 s 361 docs/s 1.0x CPU usage
4-way parallel 43.1 s 464 docs/s ~3.0x CPU usage 1.28x wall
Both runs landed 19,999 unique documents — correctness verified.
The write-lock is now the bottleneck; further gains require Phase 2
attach-forever sharding (each shard owns its own SQLite file, queries
fan out via ATTACH DATABASE, zero merge cost).
Profile of the prior implementation (cProfile on 2000-doc ingest):
total 23.6s
parser 20.1s (85% of time, 27.7M list.append calls)
DB writes 2.9s
Three changes compound:
1. PRAGMA pack in store.connect():
synchronous=NORMAL skip per-commit fsync (durable under WAL replay)
cache_size=-65536 64 MB page cache
temp_store=MEMORY temp tables in RAM
mmap_size=256 MB page-cache reads bypass read() syscall
2. Batched transactions in ingest_source and distill_existing:
200 docs/tx default, --batch-size CLI flag.
One BEGIN/COMMIT per batch instead of per doc.
Audit chain SELECT-then-INSERT still chains correctly because
SQLite reads its own uncommitted writes within a transaction.
Idempotent on collision (INSERT OR IGNORE / pre-existence check)
so partial-batch re-runs are safe.
3. Parser rewrite (the actual bottleneck): replaced char-by-char
list-append loops with str.find / string slicing.
_decode_mysql_string fast-path: if no '\\\\' in input, return as-is.
_split_values_tuples uses _find_string_end (str.find with
backslash-parity check) instead of per-char scanning.
_iter_insert_statements buffered scan (str.find for ';' and "'")
crossing read-chunk boundaries via in_string flag.
~13x fewer Python function calls, semantics identical.
_upsert_edges now batches via executemany.
After: 2000 docs in 6.0s; 5000 old revisions in 12.8s wall (was 32.3s).
Throughput 85 -> 333 docs/sec. 49 tests still passing.
Cheaper to fail = cheaper to experiment.
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
