russell@unturf.com
52071df8a2
Two issues surfaced by the user query
"tell me about back to the future plot?":
1. Mojibake in titles like 'La_Jet\\ufffde'.
The 2003 MediaWiki dumps mix Latin-1 raw bytes (e.g., 0xE9 for é)
with HTML entities. Opening the bz2 as utf-8 with errors="replace"
substitutes U+FFFD for any high byte. Latin-1 is lossless on every
byte and gives the right code point for the raw-byte cases. New
default for WikipediaSqlDump; modern dumps can pass encoding="utf-8"
explicitly.
The existing 122k cur shards keep their slightly-mangled titles
on a handful of articles — but a re-ingest from scratch (now ~2.5
min thanks to the parallel attached path) recovers them, and
--resume protects against starting over from zero.
2. Tell_(poker) outranking Back_to_the_Future for "tell me about
back to the future plot?" — BM25 on body text alone favors
short docs with rare body tokens. The title is a much stronger
topical signal.
New _rerank_by_title in qa/query.py: hits whose title contains
query tokens get +8 per match. After rerank on this query:
1. Back_to_the_Future 31.74 (was 15.74, was rank 3)
2. Back_to_the_Future_Part_III 27.83
3. Back_to_the_Future_Part_II 27.63
4. Tell_(poker) 23.41 (was 15.41, was rank 6)
5. Gunpowder_Plot 19.57
Hermes now produces a real plot summary because the canonical
Back_to_the_Future article is the lead source.
57 tests still passing.
|
||
|---|---|---|
| aborist | ||
| bench | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
aborist
An arborist for trees and forests of cross-linked information.
Aborist ingests documents into a content-addressed, Merkle-committed store. Every
document carries a URI for backtrack and cross-link. Search hooks return hits with
declared audit modes (STRICT / HYBRID / VISUAL) so callers never overclaim
what a result actually proves.
Design
Aborist implements the runtime spec from Merkle-AGI v9.8 / Merkle Providence
Reverse RAG. The 8-dim admissibility key (source_root, question_hash,
model_profile_hash, conversation_hash, governance_policy_hash,
schema_version, canonicalization_version, chunking_version) plus
falsification state ensures cached records are never reused under drift.
Two layered document kinds:
- surface — diverse ingested content (full chunks, FTS-indexed)
- core — distilled records (haiku/equation/snippet) Merkle-signed back to
source surface roots via
derivations
Reversible eviction via chunk tier ∈ {hot, warm, cold}. Cold = leaf hash
- URI only; rehydratable from URI, identity verifiable via leaf hash.
Quick start
make all # bootstrap + fetch + ingest + verify + stats
make search Q='…'
License
AGPL-3.0-only.