russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
50 lines
1.7 KiB
Markdown
50 lines
1.7 KiB
Markdown
# `aborist.evict`
|
|
|
|
Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia
|
|
docs); not every chunk fits in working memory. `evict.py` is the
|
|
mechanism that moves rarely-touched chunks to a cold tier (still
|
|
indexed, just stored separately) and rehydrates them on demand
|
|
from the original source.
|
|
|
|
## API surface
|
|
|
|
```python
|
|
from aborist.evict import evict_to_cold, rehydrate
|
|
|
|
# Move chunks unused for >threshold days to cold tier
|
|
evict_to_cold(conn, max_age_days=90, max_evictions=10000)
|
|
|
|
# Pull a cold chunk back to hot from its original source
|
|
rehydrate(conn, document_root="abc123...")
|
|
```
|
|
|
|
## Invariant: cores never evict
|
|
|
|
`evict_to_cold` filters `WHERE kind='surface'`. Cores are always
|
|
hot — they're the long-tail-friendly compression layer that justifies
|
|
evicting their underlying surfaces. Evicting cores would defeat the
|
|
purpose.
|
|
|
|
## v9.8 falsification on drift
|
|
|
|
When `rehydrate()` re-fetches a document and the recomputed
|
|
`document_root` differs from the stored one, the source has
|
|
changed since ingest (Wikipedia article was edited, HTML page was
|
|
republished, etc.). The cache record's `falsification_state` flips
|
|
from `live` to `stale` — every providence record keyed on that
|
|
`source_root` is no longer admissible to lookups.
|
|
|
|
This is the **drift-detection-as-falsification** discipline: cache
|
|
hits don't blindly trust historical answers; they trust answers
|
|
that the SAME source still grounds.
|
|
|
|
## Tier values
|
|
|
|
`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`;
|
|
cold chunks live with NULL `content` and a `cold_uri` pointing at
|
|
the source. The QA pipeline's chunk-fetch path checks `tier`; on
|
|
'cold', it triggers `rehydrate` before continuing.
|
|
|
|
## Source
|
|
|
|
[`aborist/evict.py`](../../aborist/evict.py)
|