A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
New docs/crawler.md covering the crawl path we actually run: BFS
same-host discovery, robots/feed/sitemap handling, polite vs --fast,
the shared-session + no-HEAD + crawl-delay-fix speedups, and the
content-addressed payoff. Leads on the two store-derived diagnostics:
duplicate detection (group by document_root — body, not URI) and
partial-overlap (shared chunk leaves), plus orphan finding
(sitemap − BFS-reached) and the planned crawl-report webmaster tools.
Honest pros/cons: orphans invisible to crawl by design, single-host,
no JS execution, --fast is anti-social off your own turf.
Full --fast crawl of russell.ballestrini.net (242 URIs): 26s -> ~5s.
Three changes, biggest first:
1. fast_mode now actually ignores crawl-delay (the ~5x). The delay was
only zeroed on the robots-200 path; a site with no robots.txt (404)
or a robots fetch error fell back to default_crawl_delay (2s). Under
--fast that made every concurrent fetch wave sleep ~2s — ~10 waves
x 2s dominated the wall time. _enforce_crawl_delay now short-circuits
when fast_mode, matching the documented "ignore crawl-delay"
contract regardless of robots status. Disallow is still honored
(separate path).
2. One shared ClientSession for the fetcher's lifetime (keepalive TCP
connector sized to page-worker width) instead of a fresh session per
fetch — ~3x on a 24-page wave. Lazily built in-loop via _get_session;
the bridge closes it in a finally (guarded on owning the fetcher).
3. Drop the per-page preflight HEAD. aiohttp exposes response headers
before the body is read, so the existing content-type binary guard
skips images/video/audio without downloading them — the HEAD was a
redundant round trip that doubled per-page latency.
Diverges arborist's AsyncWebFetcher from the agents.ai.unturf.com/core
verbatim lift (fox-approved); candidate to upstream. Regression tests
pin fast=no-delay / polite=delay, shared-session lifecycle, and bridge
session teardown (owned vs injected).
Two crawler-discovery changes surfaced while chasing fast-crawl wall
time on russell.ballestrini.net:
1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs
(a multi-MB atom.xml among them) only for ingest_crawled to discard
them. Gate enqueue on the existing _looks_like_feed_url so we never
fetch crawl-infrastructure URLs — less wasted work and one fewer
slow wave straggler.
2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup
html.parser closures (fresh fetch + 2 cache paths) collapse into one
module-level extract_page_links() backed by lxml.html (C parser,
releases the GIL so to_thread actually parallelises) with a BS4
fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s.
Honest scope: neither moves full-crawl wall time much — measurement
showed the dominant cost is the per-page HEAD+GET double round-trip on
a per-call ClientSession, not parsing. These are correct-and-cleaner
on their own; the wall-time lever (shared session + drop redundant
HEAD) is a separate change. lxml extraction is regression-pinned
against the BS4 fallback for parity.
The bridge BFS fetched pages one-at-a-time, so --fast only dropped the
crawl-delay (sequential, zero-wait). Fast_mode's CPU*3 page-worker
budget never reached the path operators actually run.
Replace the popleft loop with a wave loop: each iteration pulls up to
`fetcher.max_page_workers` URLs off the queue front and fetches them
with asyncio.gather. Width is CPU*3 under fast_mode, 1 otherwise, so
the polite path stays byte-for-byte sequential and the per-page
crawl-delay still serialises same-domain fetches. Wave size is capped
to the remaining max_pages budget; dedup moves from pop-time to
enqueue-time so a URL linked from two parents in one wave is fetched
exactly once.
Measured on russell.ballestrini.net (own host, robots 404): same
12-page work 23.1s polite -> 4.0s fast (5.7x); full 243-page crawl
~25s vs the ~486s polite floor (19x). Disallow still honored; only
the rate limit is lifted.
Tests: peak-in-flight pins (>1 fast, ==1 polite) plus all existing
BFS bound / dedup / depth / max-pages cases on the width=1 path.
When a re-crawl detects a real content delta (a just-ingested root that
supersedes a prior version — content hash changed, not redeploy/ETag
noise the idempotent ingest already no-op'd), the pipeline now surfaces
the page's document chain over time instead of just 'something changed'.
bridge.py: version_chain(conn, uri) walks a URI's documents by ingest_ts
(each content change = new content-addressed doc + supersedes edge);
delta_report() adds the word-level similarity of the latest change;
render_delta_report() prints it. ingest_crawled() detects superseding
roots, emits the lineage report to stderr per changed page, and returns
'deltas' in its summary. Validated on the live russell.ballestrini.net
re-crawl: 223 pages, full redeploy, exactly 1 content change (/about/),
rendered as a 2-version chain (90% similar to prior). 2 tests; suite
2551 passed.
- '.. class:: center' parsed as a Python class named 'center' (duplicate
across one-pager + two-pager) -> '.. rst-class:: center' (styling, no
object). Clears the duplicate-object-description warnings.
- two v8-consensus section underlines were shorter than their titles
('underline too short') -> extended to title length.
Pager + v8 pages now build with zero warnings.
The page was a markdown+LaTeX paper dumped into .rst — 499 errors/warnings,
big chunks rendering broken. Mechanical, content-preserving fixes:
- markdown code fences (```lang) -> RST .. code-block:: (48 blocks; the
dominant error source — RST read each ``` as an unclosed inline literal)
- display math [ ... ] -> literal blocks (34)
- code-block:: json -> text where bodies are schemas with <...>/[m,n]
placeholders, not valid JSON (10; same as the v8 fix)
- escape inline | in prose (math like |Z|, |pred(v)|) read as RST
substitution refs (37 lines; no pipe-tables in the file, so safe)
Residual 62 are scattered indented-math derivation lines RST treats as
block quotes — they still render, just warn; diminishing returns on an
advanced/optional theory page.
The simple-table (=== separators) had a wrapped multi-line cell that
docutils rejected as malformed -> the floor table rendered broken on the
live site. Converted to a list-table (alignment-proof). Also switched the
proposer-submission block from 'code-block:: json' to 'text' — it's a
schema with <...> placeholders, not valid JSON, so the json lexer warned.
Page now builds with no errors/warnings on these blocks.
The Substrate group is Dav1d's formal Merkle-AGI research that drove the
design — valuable, but it sat third in the nav and read like required
reading. IA-only fix (no content touched):
- re-caption 'Substrate' -> 'Substrate theory (advanced)' + a preamble
marking it optional (nothing in Getting started / API depends on it);
- reorder it BELOW the practical sections (now: Summary -> Getting started
-> API -> Substrate theory -> Project);
- pull 'bench' out of Substrate into Getting started (it's the practical
benchmark surface, not theory).
Practitioner path comes first; theory stays intact + credited, just last.
- Frame the solution as a Reverse RAG (Merkle Providence Reverse RAG) with
a link to the whitepaper (unfirehose.com/merkle-providence-reverse-rag-
whitepaper — note: published on unfirehose, not uncloseai).
- New differentiator: NO vector embeddings — retrieval is lexical-first
(FTS5 BM25 + Merkle), dense-vector optional + off by default; embedding
10M docs costs 10-100x more/doc + a vector index to store/maintain. A
big part of why COGS is low.
- COGS framed per 1,000 answers, labeled @ $0.33/kWh (intro + diagram cost
node). Cost node clarified: no embeddings, NO reasoning (reasoning is the
thing that would cost 4-6x, which we skip). Retrieval node + mapping
table updated to lexical-first / no vector index.
ranksep=1.2 over-stretched it vertically (1193x1483, lots of whitespace).
Drop to 0.4 + nodesep 0.25: 1202x1063, ~28% less area, no vertical sprawl,
stays roughly square. Width is floored by the widest node label; click-to-
zoom (zoom.js) covers reading detail.
Graphviz renders SVG as <object>, which swallows clicks and can't be
lightboxed. Vendored, dependency-free zoom.js converts each graphviz
<object> to a clickable <img> (crisp vector) and adds a fullscreen
overlay: click to open, scroll to zoom toward the cursor, drag to pan,
Esc / dbl-click / background-click to close. Works for any img.zoomable,
so future charts get it free. Degrades gracefully (no JS -> inline image
still renders). Wired via html_css_files / html_js_files.
TB alone left it landscape (1202x1101, h/w 0.92). Width is floored by the
widest node label (can't shrink without trimming labels), so bump
ranksep to stretch vertically: 1193x1483, h/w 1.24 — clearly top-down/
vertical for the docs column.
rankdir LR -> TB so the diagram reads top-down and fits the docs column
(roughly square ~1200x1100, fox: 'square is good') instead of a wide
horizontal strip.
Graphviz DOT diagram of the arborist pipeline as the answer to the Google
L5 system-design prompt 'design a RAG pipeline for 10M docs with zero
hallucination'. Maps our components onto the canonical 10-box RAG design
and shows the three extensions that buy zero-hallucination + near-zero
cost: deterministic verifier (not a model confidence score) -> honest
UNGROUNDED; Merkle-bound cache that skips the GPU; measured energy COGS
(~$0.07-0.16/1k answers, non-reasoning). Includes the mapping table +
the 3.47M->10M scaling math (+77GB, sourcing/storage not redesign).
Enables sphinx.ext.graphviz (SVG output) + graphviz apt package on the
RTD build; adds the page to the Summary-pages toctree. Builds clean
(page renders, DOT validates via dot -Tsvg).
Main tweet + follow-up (Merkle cache hit skips GPU, doesn't increment the
per-1k). Numbers are the measured claim_lattice figures from the energy
report; note attached to hold the arbitrage/forcing-function framing until
the value side is hardened (higher N + blinded judge).
$0.10/1k-q overstated the qwen-27B case. Honest range: ~$0.07-0.16 per
1,000 queries of GPU electricity — hermes-8B $0.07-0.09 (under a dime),
qwen-27B $0.12-0.16 (over a dime; claim_lattice dearer than quote from
more prefilled context). Fixes the §5.4 'either rig' claim.
The report is now cost AND value (quality-per-dollar), not cost-only.
§5.5 quality delta: substrate-vs-solo (code judge, n=30) on numeral +
stale fixtures, hermes-8B vs qwen-27B-nothink. Substrate lifts both
2-13x and they nearly converge (hermes 18/21, qwen 19/26 grounded-
correct) — grounding comes from retrieval+verification, not parametric
size, so the base model matters far less. Bare-model failure styles
diverge (hermes abstains, qwen fabricates). Quality-per-dollar: ~18-26/30
grounded for $0.085-0.158/1k-q.
Honest caveats recorded: CORRECT=grounded-in-2010-corpus (not current);
the stale fixture is no longer post-corpus; the qwen thinking bug (39c040c)
that voided the first run and the judge strengthening (2d31866). §1
reframed; §8 repro + §9 next updated (post-2010 fixture, SOTA judge for
residue).
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.
New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
Root cause of 'arborist abstains on everything with qwen' (fox 2026-05-21):
Qwen3 thinking-on default burns the entire token budget on hidden <think>
reasoning over a 20K RAG context and returns EMPTY message.content
(measured: 768/768 completion tokens, content '') -> every arborist answer
UNGROUNDED. Bench harnesses passed enable_thinking=False via the MODELS
dict, but the CLI + control_ab did not, so the quality bench was measuring
a thinking-budget-exhaustion artifact, not abstention.
OpenAICompatibleClient now defaults Qwen3 to enable_thinking=False unless a
caller set it explicitly (reasoning-variant path passes True, preserved).
Verified: same France query goes empty/UNGROUNDED -> STRICT 'Nicolas
Sarkozy' with the flag. Fixes every caller (CLI, control_ab). 5 tests;
full suite 2547 passed. Today's qwen QUALITY numbers are void and need
re-running; energy numbers stand (real inference happened regardless).
Was a stale hardcoded 'same Hermes; judge=Opus hermetic' label that
misreported any run with --model/--judge overrides (e.g. qwen + code
judge). Now reflects the real config — honest header for the artifact.
fox 2026-05-21: two corpora exist (2003-05-16 + 2010); the 2010 one is
loaded in ~/.arborist/shards (verified: 866K docs/shard, has Barack
Obama / YouTube articles a 2003 dump can't). CLAUDE.md Live-endpoints and
the energy-COGS report now state 2010; fabrication-bait fixtures target
post-2010 events. Historical closed-ticket prose left as point-in-time
records.
- §2 now a two-rig table: 4090/Qwen-27B-Q4_K_XL/llama.cpp (isolated) vs
3090/Hermes-8B-FP8-Dynamic/vLLM (live/public). Spells out that the
cross-model rate gap confounds FOUR variables (params, quant/precision,
engine, GPU) — not '8B vs 27B' alone.
- §5.4 hermes results under live traffic. Method finding: the watt_bench
window integral is unusable on a contended card, but the slope
calibration survives clean (cancels the variable shared-load baseline)
— so on shared/public cards use watt_calibrate, not the window integral.
Rates: hermes 0.109/4.40 J/tok vs qwen 0.175/6.16; decode 35-40x prefill.
- Per-query DOLLARS both rigs @$0.33/kWh: a grounded substrate answer is
<$0.10 per 1000 queries (hermes $0.085, qwen $0.158); hermes ~half qwen.
- §5.2 budget confound corrected to flag Hermes-tuning honestly (was
rationalized). §9: fixed-budget apples-to-apples re-run + per-model
budget tuning added as next steps.
Full report for Dav1d: qwen-nothink on the dedicated 4090. Headline —
prefill 0.175 J/input-tok ($0.016/M) vs decode 6.157 J/output-tok
($0.564/M), decode 35x dearer per token. The substrate prefills ~6.6k
input tok/query (vs solo ~52, 127x): ~67% of its GPU energy is reading
the retrieved context, not generating. Per-query 1719 J (substrate-CL)
vs 95 J (solo) = $0.16 vs $0.009 per 1k queries; calibration predicts
measured within ~5%. Real cost is CPU latency, not GPU watts.
Documents the full methodology + the corrections that got here
(contamination -> isolation; blended mean-W -> measured states; gross
-> not-attributable; per-token -> separated input/output; len//4 ->
real usage), threats to validity, reproducibility (commits, config hash,
persisted samples), and next rigs (3090+Hermes, reasoning, prefill-cache
study).
fox 2026-05-21: account wattage for input and output separately. Prefill
(process all prompt tokens, parallel/compute-bound) and decode (generate
output, autoregressive/bandwidth-bound) are different GPU ops with
different J/token — a single per-token number can't represent both.
Slope calibration (no sub-request power alignment): sweep prompt length
at tiny max_tokens -> prefill J/input-tok (fixed overhead cancels in the
slope); fix a tiny prompt and sweep forced output length (ignore_eos) ->
decode J/output-tok. Prefill kept COLD (unique filler so cached_tokens=0).
Reuses watt_bench probes. Bad points (context overflow) skip, not abort.
Measured qwen-nothink/4090 @$0.33/kWh: prefill 0.175 J/tok
($0.016/M-input-tok), decode 6.16 J/tok ($0.564/M-output-tok) — decode
35x dearer per token. Predicts measured substrate J/q within ~5%. 14
tests (+ slope). Validated live.
fox 2026-05-21: (1) use REAL API token usage, not len//4; (2) the
substrate prefills a large retrieved CONTEXT as INPUT while solo feeds
~nothing, so per-completion-token over-charges the substrate — and per-
TOTAL-token UNDER-charges it (its mix is ~98% cheap prefill tokens).
Measured n=30 qwen-nothink/4090: substrate prefills ~6.6k input tok/query
(claim_lattice) vs solo ~52 — ~127x. Neither single per-token denominator
is honest; prefill (parallel, cheap/tok) and decode (autoregressive,
dear/tok) must be costed separately.
- OpenAICompatibleClient stashes data['usage'] as .last_usage (non-
invasive; return type unchanged).
- watt_bench captures real prompt_tokens + completion_tokens per call
(both arms), aggregates per cell, and energy_cogs reports gross +
marginal per BOTH 1k-total-tok and 1k-completion-tok plus the context
size. Prints the prompt/completion split.
- 12 tests incl. the prompt-context artifact (per-total cheap, per-
completion dear). Full suite 2540 passed.
The clean per-input-tok / per-output-tok split rides bench/watt_calibrate
(slope calibration; separate commit once validated live).
fox 2026-05-21: 'we dont touch card during retrieval'. Retrieval + verify
are CPU/SQLite on the orchestrator; the GPU is idle (at the shared, always-
on model-resident floor) during them. So the substrate does NOT 'hold the
card' through its wall-clock — the gross window integral over-counts by
charging that always-on floor for the seconds we spend retrieving (energy
that exists regardless of the query; the card serves other traffic then).
Correct attribution: the GPU cost of a query is its GENERATION energy only
(the marginal — burst above the serving floor). Relabel: marginal is the
headline GPU COGS; gross is demoted to a reference 'window total, not
query-attributable'. Print + energy_cogs docstring + stock-v1-config doc
updated. No math change (marginal was already right) — this corrects the
narration. Measured qwen-nothink/4090: substrate GPU COGS is LOW
(claim_lattice $0.21/M-tok, quote $0.83) — its real overhead is latency
(CPU retrieval), not GPU watts.
fox 2026-05-21: 'gen 200W' was a bug — joules/window blends the ~400W
generation bursts with the sub-100W gaps (retrieval/verify/network) into
a power state the card never sits at. A card occupies DISTINCT states
(idle / middle-idle = resident-between-requests / generation), differing
per card×model×server.
watt_probe.classify_power_bands(): largest-gap split of the window
samples into a low band (serving floor) and high band (generation draw)
+ duty cycle. Data-derived, never hardcoded — tested at two scales. The
worker emits the decomposition + raw samples; RemoteProbe/LocalProbe
expose band_stats() uniformly.
energy_cogs: marginal now taken against the measured SERVING FLOOR (the
standing cost of being ready), not deep idle; the blend is kept but
labelled window_mean_w. Reports idle/serving-floor/gen-draw/duty.
Cache-miss certainty (fox's question): the arborist arm runs
burn_existing=True (force-deletes any live providence row before
inference) and asserts cache_hits==0 with a loud warning + real_inference
flag — so we time real generation, never a SQLite lookup. Solo has no
cache path. 11 tests (energy math + band split). Validated live on the
isolated 4090: solo gen 308W/70%-duty vs substrate 396W/8.6%-duty —
substrate marginal/tok is LOWER, gross/tok higher (it holds the card
longer for retrieval).
fox 2026-05-21: compute cost-of-goods-sold by kWh vs tokens, with the
three power states (idle / warm-idle / generation) MEASURED per
card×model×server — never hardcoded (his 40/127/380 W were illustrative
of one 3090). The only operator input is --price-per-kwh (default 0.33
USD/kWh, a configurable site rate).
energy_cogs() (pure, unit-tested) decomposes measured generation energy
against the measured warm-idle baseline:
* gross — all measured joules over the window (all-in, includes the
warm-idle cost of keeping the model hot, amortized).
* marginal — joules ABOVE warm-idle: what one more request's burst
actually costs (clamped >=0).
kWh = J/3.6e6; $/1k-tok is the unit that compares to API pricing. Both
surface per cell + a COGS print line.
watt_bench's arborist arm now loads the frozen bench.stock_v1 policy
(--answer-mode, drift-guarded on non-reasoning) so cost is measured for
the SAME substrate the campaign grades. Cells record
window_start/end_unix so a post-hoc load_monitor queue-depth cross-ref
can flag organic-traffic contamination on the non-isolated single-slot
endpoints. 6 COGS tests; full suite 2534 passed.
fox 2026-05-21: characterize substrate-ON under BOTH answer shapes, so
answer_mode is a swept axis, not a single pinned value.
stock_v1.py now exposes STOCK_V1_POLICIES{quote,claim_lattice} +
STOCK_V1_GOVERNANCE_HASHES (quote 5b6ca4c5..., claim_lattice 036a4c79...),
policy_for(mode), and assert_not_drifted(mode). Shared pins (crosslang
OFF, repair OFF, quantifier dry-run, metacognition label-only,
soft-preflight OFF, claim cap 12, v2-acronym-aware) are frozen
identically across modes.
Wire the treatment arms to the pin (the consumer-side step that makes
the freeze real):
* control_ab --answer-mode {quote,claim_lattice}
* control_sweep --arborist-answer-mode {quote,claim_lattice}
Both default claim_lattice (prior behavior), call assert_not_drifted on
non-reasoning runs (halts the sweep if DEFAULT_QUERY_POLICY drifts), and
load the frozen policy_for(mode) instead of an inline
dict(DEFAULT_QUERY_POLICY, ...). Reasoning refs (phase 3) keep their
documented JSON overrides and skip the assert by design (different hash).
jaggedness is left standalone — it is a mode-agnostic retrieval
instrument, coupling it to the answer-policy freeze adds friction with no
correctness gain. Full suite 2528 passed.
Before the multi-day campaign (hermes 3090/4090 -> qwen 3090/4090 ->
reasoning variants) the substrate-ON treatment arm must NOT drift. It
previously inherited DEFAULT_QUERY_POLICY implicitly, so any mid-run
edit would silently change what 'substrate-ON' means.
bench/stock_v1.py snapshots DEFAULT_QUERY_POLICY + re-asserts the
load-bearing pins (answer_mode=quote, crosslang OFF, repair OFF,
quantifier caps dry-run, metacognition label-only, soft-preflight OFF,
claim ceiling 12, v2-acronym-aware), then hashes the whole effective
dict. assert_not_drifted() fails loudly if that hash ever changes —
re-pinning is a deliberate fox-gated V.2 bump, never silent. The whole
campaign is identified by one governance_policy_hash
(5b6ca4c5...aade4e). Non-reasoning + non-distributed are harness axes
(reasoning -> phase 3, mesh -> later fork), not policy fields.
docs/stock-v1-config.md documents V.1, substrate-OFF (control_ab arm A),
the campaign matrix, and the energy-COGS companion (#000057) — whose
power states (idle / warm-idle / generation) are MEASURED per
card+model+inference-server at runtime, never hardcoded; only $/kWh is
an operator flag.
Answers fox's 2026-05-21 question — 'are we being swamped because we're
open to internet?' — before the multi-day GPU bench, where uncontrolled
internet traffic on the single-slot hermes (3090/vLLM) and qwen
(4090/llama.cpp) endpoints would contaminate wattage + throughput.
Three stdlib subcommands (urllib + sqlite3 + hand-rolled SVG, nothing to
install):
poll — scrape each endpoint's Prometheus /metrics on an interval into
SQLite; queue depth (num_requests_waiting / requests_deferred)
is the swamp signal a single GPU slot exposes. Prunes past
--retention-days each cycle (bounded store, no cancer growth).
graph — multi-panel SVG: queue depth, running, req/s, tok/s, e2e latency.
access — parse Caddy/nginx access log for the real client IPs the backend
can't see behind the proxy hop; top talkers + per-IP rate SVG.
Backend /metrics = HOW MUCH; proxy log = WHO. make monitor-poll /
monitor-graph / monitor-access.
v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.
Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).
make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
fox: a graveyard that keeps every wrong answer forever grows unbounded
like cancer and is counter to constraint-optimization. Bake in the
bound:
- earn-to-enter (recurrence-gated) — a failure earns a tombstone only
when its canonical claim-family re-fails; one-offs dropped.
- fingerprints not transcripts (UTXO-set analogy) — store
canonical_claim_family|falsifier_root|failure_class, prune bulk text.
- decay/compact to steady-state — evicts like a surface, never a core.
- off the hot path — dedicated graveyard shard, bloom-filter burden
check only.
Gossip-group falsifier admission inherits the same discipline:
difficulty-adjusted stable-rate ingestion (BTC block-rate target) +
per-window budget (#000036), enforced in #000012/mesh. BTC lesson is
bounded self-regulating ingestion, not store-everything. If it cannot
be bounded to a steady state, the graveyard is not built.
Three tickets from the 2026-05-20 Dav1dPrometheus "Protocol-Layer AGI"
working report (held outside the repo; referenced not committed):
- #000058 cache_key_9 verifier-policy mandatory-vs-legible decision +
doc reconcile. Records the five-step-#1 correction: verifier fields
already fold into governance_policy_hash, so the 9th dim is audit
legibility not a correctness gap.
- #000059 admission discipline: claim-graveyard burden-shift +
self-providence quarantine (guards the existing ingest-self-providence
self-confirmation loop).
- #000060 H-ABCDEFG same-model substrate-delta harness (jaggedness +
curvature); curvature-aware ForkScore folded into #000012 Phase 2,
not spawned as a sibling.
Reconciles CLAUDE.md cache_key invariant (8-dim -> 8 + optional 9th).
Next ID 000058 -> 000061.
Implements fox's 2026-05-20 architecture: don't ship shards to the
tight-on-disk GPU boxes; run the benchmark FROM the laptop (local
shards + retrieval + judge + workload loop, driving the worker's LLM
endpoint over the network) and have the worker boxes REPORT their own
power.
Adds a probe abstraction with two implementations behind one
start/stop/gpu_stats/cpu_stats interface:
- LocalProbe: wraps PowerSampler + CpuSampler (watt_bench runs ON the
GPU box — the original mode)
- RemoteProbe: orchestrates bench/watt_probe.py on a remote worker
over SSH (scp the stdlib probe once, launch --until-file detached,
touch the stop-file after the workload, fetch the JSON). The worker
needs no shards / arborist / venv.
--remote-gpu-host HOST selects RemoteProbe; --endpoint points at the
worker's endpoint. Idle baseline now records gpu_util_mean_pct and
flags ">5% ⇒ contaminated by live traffic" so the 3090-style
contamination (live hermes traffic polluting the baseline) is visible
in the output, not silent.
Verified end-to-end: laptop-driven n=2 solo qwen, power sampled on the
4090 over SSH — idle gpu 20.2W @ 0% util (clean) + cpu 86W; load
populated; report tagged mode=remote. The on-box LocalProbe path is
unchanged (omit --remote-gpu-host).
Report schema gains mode / remote_gpu_host / gpu_available /
cpu_energy_available. Per-cell records gpu+cpu+total joules/question.
Splits the power-sampling half out for the laptop-driver / worker-
reporter architecture (fox 2026-05-20): don't ship shards to the tight-
on-disk GPU boxes; instead drive the benchmark FROM the laptop (which
holds the shards + judge + workload loop) and have the worker boxes
just serve the model and REPORT their own power.
Power sampling must run on the box (nvidia-smi + RAPL are host-local),
but nothing else does. watt_probe.py is stdlib-only — python3 +
nvidia-smi + readable RAPL, no venv, no arborist, no shards. Copy it to
the worker and run; it samples GPU power (nvidia-smi, trapezoid-
integrated to joules) + CPU package energy (RAPL energy-diff) over a
window and emits the same energy schema watt_bench's local samplers
produce. Two window modes: --duration (fixed) or --until-file (laptop
touches a stop-file — the workload-correlated mode). Records
gpu_util_mean_pct so live-traffic contamination can be detected.
Takes only the top-level RAPL package (intel-rapl:N, single colon),
not the :N:M core/uncore subdomains, so no double-count.
Verified on the 4090: scp'd to /tmp, ran with no install, real readings
(GPU 20.2W idle @ 0% util, CPU 82W, joules computed, 6 samples / 3.2s).
Next: laptop-side orchestration in watt_bench (--remote-gpu-host) that
SSH-starts this probe for each cell's window while the driver runs
retrieval + judge locally and drives the worker's LLM endpoint.
The exact llama.cpp (qwen/4090) and vLLM (hermes/3090) launch commands,
recon'd 2026-05-20, so the foxhop-states salt states can be written
accurately rather than guessed. Notes the convention (salt manages the
systemd unit; engine binaries + model artifacts stay manual on
/mnt/data as documented prereqs) and the live-hermes cutover constraint
(keep >=1 hermes online; qwen is expendable).
fox's call: capture the RAPL permission as a Makefile target run with
sudo, not an ad-hoc chmod. `sudo make rapl-access` on each GPU box
installs a udev rule (/etc/udev/rules.d/99-rapl-readable.rules) that
makes intel-rapl energy_uj world-readable on every powercap add event
(survives reboot), and applies chmod immediately so no reboot is
needed. Idempotent; reversible via `sudo make rapl-access-revoke`.
energy_uj is root-only by default (PLATYPUS side-channel mitigation,
CVE-2020-8694) — that's why bench/watt_bench.py's CpuSampler read null
CPU watts as fox during recon. After this target runs, watt_bench
reads CPU package energy directly (no --cpu-energy-cmd needed). GPU
watts via nvidia-smi never needed special perm.
Non-root guard + help entries verified; Makefile parses clean.
Toward fox's next goal: score the full serving stack on quality AND
cost — {qwen, hermes} × {llama.cpp, vLLM} × {3090, 4090} × {solo,
arborist}, measuring CG% + GPU watts + CPU watts + joules/answer per
cell.
watt_bench.py — adds CpuSampler (Intel RAPL package energy via
/sys/class/powercap/intel-rapl:*/energy_uj). RAPL exposes a cumulative
microjoule counter, so energy-over-window is an end-minus-start diff
(handles wrap) — more accurate than integrating instantaneous power.
Sums multi-package. energy_uj is root-only by default (PLATYPUS /
CVE-2020-8694), so it degrades to available=False when locked;
--cpu-energy-cmd 'sudo cat {path}' supplies a privileged reader when a
sudo rule exists. Each cell now reports gpu/cpu/total joules-per-
question + gpu joules-per-token; the report records cpu_rapl_available.
Verified: graceful degradation when locked; RAPL diff math (1->4 MJ uJ
= 3.0 J, exact).
benchmark-matrix.md — expands the cost section to the full 16-cell
(model × engine × GPU × arm) design, the per-cell metric set (quality +
GPU + CPU energy), the serving-stack inventory from 2026-05-20 recon
(4090=qwen/llama.cpp, 3090=hermes/vLLM — each box has one engine + one
model today), and the buildout gap (vLLM+qwen, llama.cpp+hermes, cross-
GPU models). Notes idle-floor asymmetry (hermes/3090 ~127W vs
qwen/4090 ~20W) as a real optimizer input.
Harness is ready; the serving-config buildout + RAPL perm grant are the
remaining (ops, fox-directed) prerequisites to run the full matrix.
Two deliverables for the cost/energy axis of the constraint optimizer.
docs/benchmark-matrix.md — shareable spec of the control experiment:
the question, fixture (386 office-holder Qs with corpus-vintage gold),
the 3-model × 3-framing × 2-arm matrix (18 cells), the verdict
vocabulary + two reads (accuracy vs grounding-fidelity), the
deterministic code judge + its Opus calibration, the results-so-far
table, and the NEW cost dimension (tokens / latency / GPU watts /
joules-per-answer measured per GPU tier). Self-contained — readable
cold by David.
bench/watt_bench.py — GPU wattage harness. Samples nvidia-smi
power.draw on the inference GPU while driving a small representative
subset, reports mean/peak watts, trapezoid-integrated joules,
joules-per-question, and joules-per-token. Tags the GPU
(--gpu-label 3090|4090) so the optimizer can compare hardware tiers.
Idle-baseline sampling separates load draw from idle. Does NOT grade
(energy is independent of correctness); saves answers + per-question
timing to JSONL for a later quality-per-joule pass via
score_with_code_judge.
Designed to run ON the GPU box (the orchestrator has no GPU; the
3090/4090 live on the inference boxes). Degrades gracefully when
nvidia-smi is absent (energy fields null) so it is testable anywhere.
Verified: PowerSampler graceful degradation + trapezoid integration
(synthetic 100->200->200W over 2s = 350 J, exact).
The headline cost finding the optimizer must weight: qwen-think
reasoning = 1300-3300 tokens/answer vs qwen-nothink ~50-100 (20-50x),
for a workload where arborist+qwen-nothink already lands 82% CG. The
energy numbers will quantify whether reasoning's premium is ever
justified — grounding-fidelity per joule, not per answer.
fox was right ("or you have defects still"): the arborist+qwen-think
empties were NOT an inherent reasoning limitation, they were
max_tokens set too low. Diagnosed with finish_reason + token usage on
a realistic large arborist-style context:
max_tokens=1024 → 4/4 empty, ALL finish_reason='length', all hit
exactly 1024 tokens
max_tokens=4096 → 0/4 empty, ALL finish_reason='stop', used
1339-3295 completion tokens
qwen-think spends 1300-3300 tokens on its (internal) reasoning trace
BEFORE emitting the tiny claim-lattice JSON. The arborist arm's budget
comes from DEFAULT_QUERY_POLICY["max_tokens"]=512 (correct for non-
reasoning single-line JSON), so reasoning refs hit finish='length'
mid-trace and return EMPTY. Fix: arb_policy["max_tokens"]=8192 for
reasoning refs (generous headroom over the 3295 observed ceiling).
This also CORRECTS a wrong claim in commit aa9d9c8's message: the
json-schema grammar does NOT suppress the reasoning trace. The
reasoning happens (it burns those 1300-3300 completion tokens); the
grammar only shapes the final emitted answer into schema-valid JSON.
So arborist+qwen-think is a genuinely distinct config, not a clone of
arborist+qwen-nothink. The earlier "grammar suppresses reasoning"
read was an artefact of toy-prompt diagnostics where reasoning fit
under 1024 tokens.
The empty-retry from aa9d9c8 stays as belt-and-suspenders for any
residual model-side empties, but with an adequate budget it should
rarely fire.
Cost note (fox's standing point, now quantified): reasoning = 1300-
3300 tokens/answer vs qwen-nothink's ~50-100 = 20-50x token cost. For
a workload where arborist+qwen-nothink already lands 82% CG, that cost
multiplier is the open question — not whether the cell is measurable
(it now is), but whether the marginal lift justifies 20-50x spend.
Two defects blocked the arborist+qwen-think cell, both diagnosed
2026-05-20:
Defect 1 — stop sequence truncates to empty. The claim_lattice path
sets stop=['\n\n'] (runaway guard tuned for single-line Hermes JSON).
A reasoning model's output trips it immediately → 100% empty answers
→ 100% ABSTAINED (measured on the first 23 items of the killed run).
Fix: MODELS gains an explicit flag; the arborist arm
clears claim_lattice_json_stop_sequences for reasoning refs. Direct
A/B confirmed: stop=['\n\n'] → ''; stop=None → valid JSON.
Defect 2 — intermittent empty completions. Even with the stop cleared,
qwen-think under json-schema grammar emits an empty completion ~1/3 of
calls (a llama.cpp reasoning+grammar artefact; qwen-nothink phase 3
had ~0 spurious empties). Fix: empty-output self-heal — reasoning refs
retry up to 3 attempts, burning the cached empty each retry. Never
fabricates: a still-empty answer after retries is recorded as empty.
6-item smoke: 0/6 (broken) → 5/6 valid JSON; residual ~1/6 are
questions that reliably break (4 consecutive empties), a documented
artefact.
Structural finding (to fold into Addendum 8): json-schema grammar
enforcement SUPPRESSES the reasoning trace — output is pure single-
line JSON, no <think> block. So arborist+qwen-think is structurally
≈ arborist+qwen-nothink; the thinking lever that moved the solo arm
is neutralized by the claim_lattice grammar. The re-run will confirm
empirically.
Non-reasoning refs (hermes, qwen-nothink) unchanged: single pass,
stop sequence intact.
Durable record of the #000057 sweep in the bench journal. Captures:
- The question: is Hermes-8B's confident present-day-officeholder
fabrication an 8B weakness, a framing artefact, or does retrieval
fix it? Crosses {hermes, qwen-nothink, qwen-think} × {plain,
source_relative, as_of_corpus} × {solo, arborist} on a 386-item
office-holder fixture with corpus-vintage gold.
- The judge methodology: Opus headless judge burned quota (79.5%
JUDGE_ERROR), replaced with the deterministic code judge
(bench/judge_code.py), calibrated against Opus's gradeable records
(CG agreement 13->47%, WRONG 56->89%, ABSTAINED 80->95%).
- Consolidated CG% scorecard, all arms on the identical final judge.
- Three findings:
1. Retrieval dominates — arb/qwen-nothink/plain 82% vs 7% solo;
no solo config approaches the retrieval arms.
2. Reasoning does NOT improve raw correctness — qwen-think/as_of
44% vs nothink 50%.
3. Reasoning's real cost is broken honest-abstention —
qwen-nothink/source_relative abstains 97% (clean); qwen-think
only 61%, reasoning itself into wrong parametric answers.
- Production recommendation: arborist + qwen-nothink, plain framing,
reasoning OFF (82% CG, ~0% abstain, 11% wrong-assert).
- Held cell noted: arborist+qwen-think running at write time, result
to be appended.
Bench %s are point-in-time measurements (not repo-derived counts),
so no AUTOCOUNT tags — consistent with addenda 1-7. test_doc_counts
3/3.
fox 2026-05-19: 'we don't need to redo anything'. The arborist+qwen-
nothink sweep would re-run qwen-nothink solo on 1158 records that
phase 1 already produced (control_sweep_2026-05-19T21-52-56Z.jsonl,
calibrated rescore on disk). Wasteful — qwen-nothink solo behaviour
doesn't depend on retrieval being on, so phase 1's numbers stand.
--skip-solo runs ONLY the arborist arm. Implementation: skip the
per-variant model loop in _process_item when the flag is set; the
arborist arm below it still runs if arborist_on. The spend banner
zeroes the solo-call count so the operator sees the actual LLM
budget for the arborist-only run.
Example: a full arborist+qwen-nothink sweep against the 386-item
fixture goes from 2316 LLM calls (1158 solo + 1158 arborist) to
1158 calls (arborist only). Halves wall-clock on single-worker
llama.cpp.
pytest 27/27, imports clean, --help shows the flag.
Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI
contradiction) so positive lexical evidence cannot be overridden by
NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen-
nothink smoke:
i=3 · who is the prime minister of Poland?
ans: 'Donald Tusk is listed as the Prime Minister of Poland.'
gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz...
NLI contradiction p=0.892 (above 0.85 threshold)
NLI entailment p=0.744 (also high on the correct clause)
Tusk WAS PM in 2010 (served 2007-2014); answer is correct against
the corpus-vintage gold. The NLI contradiction signal came from
clause-level candidate selection picking a NON-Tusk PM the source
also mentions; entailment was high on the Tusk clause. Mixed signal
that the WRONG rule then over-confidently resolved.
The fix is a rule reorder, not a threshold change — the fast path's
positive-evidence combination (specifics-in-gold AND subject-in-gold)
is a strictly stronger signal than NLI's clause-level max
contradiction, so when it fires it should win. The combination
discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from
Anthony-Albanese (Albanese ∉ gold → fast path declines → falls
through to UNGROUNDED-subject-in-gold → WRONG, unchanged).
Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27.
Poland-Tusk regression smoke: now CG via short_entity_grounded ✓.
No regression risk on the existing reconciliation cells:
- Iceland CG: short_entity_grounded was already winning (was rule
4, now rule 3 — same outcome, earlier exit)
- WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓
- Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓
- Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓
- Abstention phrases: rule 2 still fires first ✓
Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.
Fix 1 — multi-engine structured-output extras
The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper
claim_lattice_structured_output_extras() in arborist/qa/verify.py
now returns a dict carrying the schema under all three engine
conventions:
- guided_json (vLLM grammar-constrained sampling)
- json_schema (llama.cpp native shorthand)
- response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)
Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.
Fix 2 — query() accepts user-supplied extra_body, merges with defaults
query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.
bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).
Tests
+ 3 new in tests/test_verify_json.py covering helper default shape,
alternate-schema reuse, and query()'s new extra_body parameter
220 affected tests still green (verify / claim_lattice / judge /
runner suite)
pytest test_verify_json: 27/27
Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
The Arborist arm runs answer_mode='claim_lattice' (per control_sweep.py
:179, control_ab.py:155) so its answers arrive as the JSON envelope
{"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
_descaffold strips the [E1] evidence-pointer markup but the JSON
braces + key syntax remain. The verifier's strategy-2 (span) and
strategy-3 (proper-noun) extractors see brace noise instead of the
inner claim prose — every Arborist record degraded to UNGROUNDED.
The 2026-05-19T17-01-17Z sweep, re-graded with the freshly calibrated
judge (5a17f61), surfaced this: Arborist arm reported 0 CG across all
three variants in the live phase 1 output (the live run was pre-
calibration), and 29/120 CG (24%) under the calibrated rescore — clear
improvement just from theta_contra=0.85, but the JSON envelope was
still hobbling the verifier paths.
Fix: _unwrap_claim_lattice_json runs BEFORE all downstream rules.
Detection is conservative (three independent signals: starts-with-
brace AND "claims" key AND "text" key) so plain-prose answers
pass through unchanged. Multi-claim envelopes concatenate as discrete
sentences (extract_claim_spans treats each as its own span).
Malformed JSON falls back to the original answer — no silent
rewriting on broken input.
Smoke result on the Iceland Arborist case
ans: {"claims":[{"text":"The current president of Iceland is
Ólafur Ragnar Grímsson.","evidence_ids":["E1"]}]}
gold: {{Infobox Political post |post = President |body = Iceland
|incumbent = [[Ólafur Ragnar Grímsson]] ...}}
before: UNGROUNDED → FABRICATED (then WRONG after calibration)
after: short_entity_grounded → CORRECT_GROUNDED
pytest: 27/27 (added 7 unwrap-coverage tests covering single-claim
envelopes, multi-claim concatenation, plain-prose passthrough,
malformed-JSON tolerance, unrelated-JSON passthrough, and the
end-to-end Arborist-envelope CG flow). Self-test 4/4 unchanged.
Re-rescores of 17:01 sweep + phase 1 sweep run after this commit
to measure final Arborist scorecard improvement.
