russell@unturf.com
856b3116d7
A content-addressed, Merkle-committed document store implementing the runtime spec from Merkle Providence Reverse RAG (April 2026 whitepaper) and Merkle-AGI v9.8 admissibility ledger. Ports proxy.unturf.com Go merkle conventions to Python: non-commutative HashCombine with 0x03 prefix, explicit IsLeft per sibling, self-duplicate odd elements. What's in: - merkle.py — proof generation/verification, JSON serialization - store.py — v9.8 SQLite schema: 8-dim providence_cache key, falsification_state, append-only audit chain, surface/core kind, hot/warm/cold tier, derivations, edges - ingest.py — Source -> normalize -> chunk -> merkle -> upsert, idempotent on document_root collision - search/ — SearchBackend ABC with explicit AuditMode (STRICT/HYBRID/ VISUAL), FTS5 backend returning VISUAL hits - sources/ — wikipedia.py (streaming bz2/MySQL extended-INSERT parser for 2003-era cur dumps); html_page.py (selectolax + httpx, robots.txt honored automatically) - distill/ — Distiller ABC + first-sentence-v1 stub. Runner generates per-contributing-chunk Merkle proofs binding cores back to source document_root. - evict.py — hot->cold demote (NULLs content, drops FTS row, retains leaf_hash). rehydrate() refetches via source pipeline; matching root restores content, mismatching root marks providence stale and writes rehydrate_drift event. Cores never evict. - cli.py — ingest / search / verify / stats / distill / evict / rehydrate - 31 tests covering merkle round-trip, ingest+audit, chunker version binding, html parse, distillation proof verification, evict+ rehydrate including drift detection. Smoke: 503 Wikipedia 2003-05-16 + 3 fox-owned HTML pages ingested, 478 cores produced (24 surface->core merkle dedups), 7 chunks evicted to cold and round-tripped via rehydrate, 987 audit events chained 0 breaks.
83 lines
2.7 KiB
Python
83 lines
2.7 KiB
Python
"""Pure-parse tests for HtmlPageSource. No network."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
# Skip if optional extras are not installed.
|
|
selectolax = pytest.importorskip("selectolax")
|
|
|
|
from aborist.sources.html_page import parse_html
|
|
|
|
|
|
SAMPLE_HTML = """<!doctype html>
|
|
<html>
|
|
<head>
|
|
<title>Eight Forms of Capital</title>
|
|
</head>
|
|
<body>
|
|
<header><h1>Site nav we want stripped</h1></header>
|
|
<nav>nav links also stripped</nav>
|
|
<main>
|
|
<h1>Eight Forms of Capital</h1>
|
|
<p>Living, Material, Financial, Intellectual.</p>
|
|
<p>Experiential, Social, Cultural, Spiritual.</p>
|
|
<p>See also <a href="/eight-forms-of-capital/">this page</a> and
|
|
<a href="https://example.org/external#section">an external link</a>.</p>
|
|
<a href="javascript:void(0)">js link</a>
|
|
<a href="mailto:foo@example.com">mail link</a>
|
|
</main>
|
|
<footer>strip me too</footer>
|
|
<script>console.log('strip me');</script>
|
|
<style>body { color: red }</style>
|
|
</body>
|
|
</html>"""
|
|
|
|
|
|
def test_parse_extracts_title_and_body():
|
|
doc = parse_html("https://example.com/page", SAMPLE_HTML)
|
|
assert doc is not None
|
|
assert doc.title == "Eight Forms of Capital"
|
|
assert "Living" in doc.content
|
|
assert "Spiritual" in doc.content
|
|
|
|
|
|
def test_parse_strips_noise():
|
|
doc = parse_html("https://example.com/page", SAMPLE_HTML)
|
|
assert doc is not None
|
|
assert "console.log" not in doc.content
|
|
assert "color: red" not in doc.content
|
|
assert "Site nav we want stripped" not in doc.content
|
|
assert "strip me too" not in doc.content
|
|
assert "nav links also stripped" not in doc.content
|
|
|
|
|
|
def test_parse_extracts_edges_and_resolves_relative():
|
|
doc = parse_html("https://example.com/page", SAMPLE_HTML)
|
|
assert doc is not None
|
|
uris = {e.dst_uri for e in doc.edges}
|
|
# Relative href resolved against the page URL.
|
|
assert "https://example.com/eight-forms-of-capital/" in uris
|
|
# External link kept; fragment moved to anchor.
|
|
assert "https://example.org/external" in uris
|
|
# javascript: / mailto: / tel: / hash-only anchors must be skipped.
|
|
assert all("javascript" not in u for u in uris)
|
|
assert all("mailto" not in u for u in uris)
|
|
|
|
|
|
def test_parse_anchor_split():
|
|
doc = parse_html("https://example.com/page", SAMPLE_HTML)
|
|
assert doc is not None
|
|
external = next(e for e in doc.edges if e.dst_uri == "https://example.org/external")
|
|
assert external.anchor == "section"
|
|
|
|
|
|
def test_parse_empty_body_returns_none():
|
|
doc = parse_html("https://example.com/empty", "<html><body></body></html>")
|
|
assert doc is None
|
|
|
|
|
|
def test_parse_no_html_returns_none_or_empty():
|
|
# Selectolax tolerates non-HTML; we want no Document for empty content.
|
|
doc = parse_html("https://example.com/x", "")
|
|
assert doc is None
|