Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
fox 2026-05-21: two corpora exist (2003-05-16 + 2010); the 2010 one is
loaded in ~/.arborist/shards (verified: 866K docs/shard, has Barack
Obama / YouTube articles a 2003 dump can't). CLAUDE.md Live-endpoints and
the energy-COGS report now state 2010; fabrication-bait fixtures target
post-2010 events. Historical closed-ticket prose left as point-in-time
records.
Three tickets from the 2026-05-20 Dav1dPrometheus "Protocol-Layer AGI"
working report (held outside the repo; referenced not committed):
- #000058 cache_key_9 verifier-policy mandatory-vs-legible decision +
doc reconcile. Records the five-step-#1 correction: verifier fields
already fold into governance_policy_hash, so the 9th dim is audit
legibility not a correctness gap.
- #000059 admission discipline: claim-graveyard burden-shift +
self-providence quarantine (guards the existing ingest-self-providence
self-confirmation loop).
- #000060 H-ABCDEFG same-model substrate-delta harness (jaggedness +
curvature); curvature-aware ForkScore folded into #000012 Phase 2,
not spawned as a sibling.
Reconciles CLAUDE.md cache_key invariant (8-dim -> 8 + optional 9th).
Next ID 000058 -> 000061.
2026-05-19: huge-N #000057 control-arm sweep (f63b00d → 9dc02e4) burned
our Opus quota. Pin the policy into repo CLAUDE.md so future agents
don't reach for Opus on autopilot:
- Sweep + bench-qa default endpoints: Hermes-3-8B + Qwen via uncloseai
- Opus drops out of control-arm grids until fox explicitly re-adds it
- Existing bench/results/ data stays valid; just no new Opus expansion
- Translation still uses local pinned opus-mt (sandwich MT edges), not
Hermes/Qwen 8B
- Grok rejoin path noted as an HTML comment (paid spend, fox-gated)
Adds two subsections under '## Live endpoints':
- 'Budget discipline — Hermes / Qwen first, Opus deferred'
- 'Per-call model selection (general agent work, not sweep arms)'
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
Compressed headline: when a lever's failure class is ≤3-5/75 q the
curated n=3 audit_mode bench can't resolve it (four 2026-05-18
hypotheses died there). Fix the instrument: mine ground-truth-
carrying questions (bench/mine_questions.py) graded by deterministic
retrieval recall@k via `query --dry-run` (bench/recall_at_k.py) —
no LLM/verifier/n=3-noise/5pp-floor, ±1-q resolution. Caveats codified:
measures retrieval surfacing (necessary-not-sufficient for STRICT);
complements never replaces the curated adversarial trap gate.
Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
End-to-end gap-close from the Phase 1 extractor. Five interlocking
fixes; live-verified that `what is a CPU?` → "Central processing unit"
at #1, `what is a GPU?` → "Graphics processing unit" at #1
EVIDENCE-WARRANTED 1/1; Mount Kilimanjaro / Soviet Union queries
unchanged (no regression).
(a) `synonym_expand` over-cap path is now rank-and-truncate by
source-frequency (descending) instead of hard-skip. CPU has 13
legitimate homonym expansions across the corpus; the prior
MAX_NEIGHBORS_PER_TOKEN=8 cap contributed *zero* expansion → no
canonical-article surfacing. Now: keep the 8 dominant by per-(token,
target) source-root count via _load_neighbor_source_freq.
(b) `_search_titles` orders by FTS5 bm25 ASC instead of LENGTH(title)
ASC on the FTS5-MATCH path. The length-asc tie-break was correct for
the 2026-05-02 "Back to the Future" LIKE-substring case but
counter-productive on FTS5 (tokenized; no substring junk; length-asc
preferred "Unit" / "Unite" / "B unit" over "Graphics processing
unit"). LIKE fallback keeps length-asc since the substring issue
persists there.
(c) `accept_tokens` (feeds title-search, core-keyword, title-rerank)
uses the synonym-expanded set instead of qtokens-only. The Phase 1
expansion existed but was only used in the FTS5 OR-fallback; satellite
articles saturated the budget before the canonical article entered.
(d) ARCHITECTURAL: `synonym_expand_strict()` (new — high-trust
evidence-kind subset: manual + manual_legacy + acronym_parens,
**excludes** link_reciprocity) for use in the multiplicative
`_rerank_by_title_purity` and as the source for `accept_tokens`.
Reciprocal-wikilink edges express *topical adjacency*, not synonymy
(a `Dinosaurs` page reciprocally links to `Curious George Brigade` →
edge that should not amplify retrieval); a multiplicative ranker over
them blows up. Strict view preserves the acronym-parens surfacing
(those edges ARE the phrase=expansion identity) while keeping
link-reciprocity to additive retrieval-route boosts via the broad
synonym_expand (still wired to `or_synonym_pool` for the FTS5 OR
fallback).
(e) Extractor regex tightened `[A-Z]{2,6}` → `[A-Z]{3,6}` and purged
~21K 2-letter acronym edges from shards. 2-letter acronyms (AI/ML/OS/
US/UK/IT/PC/TV) homonym-collide too often with common 2-letter QUERY
tokens like `go`/`is`/`am` — without this, "why did the dinosaurs go
extinct?" pulled Curious George Brigade via GO-acronym edges. The
high-value acronyms (CPU/GPU/RAM/DNA/FBI/WHO/…) all clear 3 chars.
Also: CLAUDE.md gains a "prefer existing ticket; only split for
Dav1d-review audience" discipline note (saved as feedback memory) —
this work is itself an example: would have been #000055 + #000056 +
#000057 under the prior pattern; instead extends #000054.
Suite: 2531 passed (no regression). bench-qa in flight separately.
Per-sentence shape check (no model) emitting kind ∈
{phrase_component_reuse, circular, vacuous, ok, empty}:
- circular: subject content-tokens ⊆ predicate's and the predicate
leads with a subject token ("Water is water").
- phrase_component_reuse: subject quotes a phrase, predicate reuses
one of that phrase's own tokens as a bare "the/a/an <token>"
referent — the 2026-05-12 field case ("the phrase 'Zionist entity'
is used as the entity"), a token collision the verifier +
deflection + title-relevance all pass and NLI returns neutral on.
Copulas inside a quoted span are skipped so 'war is peace' doesn't
break the subject/predicate split.
- vacuous: predicate is only placeholder hypernyms + filler ("X is
a thing").
Conservative — no full token-salad parsing; legit definitions pass ok.
Surfaced in inspect_cache_key + the `arborist inspect` human view
(· incoherent: <kind>). Advisory only — never writes providence_cache
/ audit_events / run_dag_root; demote-only verifier hook deliberately
not wired. 9 tests; full suite green (2500 passed).
fox's commit `bb6a89c` (2026-05-09, "30% reduction") deleted
seven docs but the indexes weren't updated to remove the dangling
references. End-of-session sweep caught + cleared:
== Phantom refs in TICKETS.md "Distinction from other docs" ==
5 dangling references to deleted docs:
- concept-relations-design.md → deleted in bb6a89c
- mesh-deploy.md → deleted in bb6a89c
- self-reference-design.md → deleted in bb6a89c
- verifier-semantic-gap-design.md → deleted in bb6a89c
- qa-modes-bench-2026-04-30.md → deleted in bb6a89c
Replaced with the high-value arch docs fox's bb6a89c commit
message explicitly named as kept:
- seven-point-program.md (north-star directive)
- cti-architecture.md (kept)
- mesh.md (mesh-deploy removed; mesh.md kept)
- benchmarks.md (operational; kept per commit message)
- bench-maxing.md (operational; kept)
- qa-modes-bench.md (rolling addenda — distinct from the
dated -2026-04-30 file that was deleted)
- soft-hash-channel-analysis.md (#000018 closure)
- soft-hash-channel-t3-bound.md (#000036 derivation)
- tool-action-dag-design.md (already present; kept)
== Phantom refs in CLAUDE.md `## Docs index` ==
5 dangling references (one of them duplicated — fox typo'd
self-reference-design.md on two consecutive lines with the
same description):
- mesh-deploy.md
- qa-modes-bench-2026-04-30.md
- bench-emergent-design.md
- verifier-semantic-gap-design.md
- self-reference-design.md (duplicate entry)
Replaced with current arch docs:
- mesh.md (mesh wire + group-key)
- soft-hash-channel-analysis.md + soft-hash-channel-t3-bound.md
- calculator-test-patterns.md (#000044 sister discipline)
- spec-methodology.md (#000019)
- v7w-frontier-catalog.md (#000013)
- v8-fork-score.md (#000012)
- pi-star-composition.md (#000015)
== Inline ref in CLAUDE.md §"Retrieval pipeline" ==
L328 cited `docs/concept-relations-design.md` for the
1.6%-storage-tax rationale. Doc deleted; rationale now lives
inline in `arborist/concepts/extract.py` + the data is
self-describing. Updated the prose to note the deletion +
where the design now lives.
== Why this matters ==
A phantom doc reference is the doc-index equivalent of a stale
phase descriptor — it sends future readers to a file that
doesn't exist. Same root cause as today's §-status drift
(in-progress prose freezing while load-bearing surface updates).
The bb6a89c commit was thorough about deletion but didn't
sweep the indexes; this commit closes that loop.
Hygiene: doc-only commit, no schema, no tests touched. Full
suite 2347 passed + 37 skipped earlier; 60 active AUTOCOUNT
claims still clean (3 passed in 2.80s).
Two parallel landings in one commit:
== §-status sweep on in-progress tickets ==
Checked #000034 / #000035 / #000036 §7 Status sections against
their file-header status. Same pattern as earlier sweep on
#000037 / #000025 / #000006 — body-section prose freezes at
earlier snapshots while file headers stay current. 2 of 3 had
drift.
**#000034 §7** lead: was "Open · awaiting go/no-go" + "Phase 1a
proposed below" + body subsection "Phase 1a (landed 2026-05-10)".
The lead line was a 2-versions-old prose snapshot contradicting
the same section's own subsection. Refresh: "In progress · Phase
1a landed 2026-05-10 (commit 1dfb8b9; test backfill a4b3056).
Phase 1b parks until v7 reference checkpoint..."
**#000035 §7**: already says "In progress · Phase 1 landed
2026-05-10". No drift; sample-confirmed clean.
**#000036 §7** lead: was "Awaits fox + cryptographer review of
constants." Refresh to current state — "pre-review polish pass
in 8916bf3; math review in flight with dav1d (forwarded
2026-05-10 Asia/Kuala_Lumpur as Tier 2 bundle: t3-bound.md +
soft-hash-analysis.md + t3_bound_calculator.py +
test_t3_bound_calculator.py + this ticket). Empirical tightening
tracked separately under #000043." Matches file header verbatim.
**#000036 closure criterion (c)** was marked "[pending —
docs-only commit]" but soft-hash-channel-analysis.md §9.3 line
475 already reads:
"§9.3 closed 2026-05-10 via the T3 per-window bound at
docs/soft-hash-channel-t3-bound.md (under #000036)"
So criterion (c) is done. Refresh: "[done — §9.3 closure landed
in line 475 of that doc]". Also clarify criterion (d) gates on
the dav1d review verdict (was ambiguous as just "explicitly
accepted by fox").
== #000037 §20 — reference #000045 follow-on ==
fox opened #000045 (Prometheus-Σ Phase 3 sleep-sweep scaffold)
in commit 0379e4c. My §20 refresh from 3b30126 said "Phase 3
gates on a renewed §12 trigger plus a Phase 3 ticket" — now
that Phase 3 ticket is named: #000045. Refresh to point at it
explicitly so future readers don't have to grep TICKETS.md to
find the follow-on.
== CLAUDE.md — AUTOCOUNT discipline rule ==
Added a paragraph to "Operational rules" pointing future
blackops shifts at the AUTOCOUNT discipline (#000044 + the
harness at tests/test_doc_counts.py). Without this, the harness
is discoverable only by accidentally hitting a test failure or
reading commit messages. With it, "tag at write time" is a
documented operational rule alongside Python-only, PYTHONUNBUFFERED,
and fail-closed.
Rule names: format, 4 metrics (tests / fixture-rows / db-rows /
db-where), code-fence-skip behavior, closed-ticket-stay-untagged
discipline, pointer to #000044 + the harness module.
== Verification ==
$ pytest tests/test_doc_counts.py
3 passed in 2.80s
No drift introduced. fox's parallel 4 commits (4b85a0a /
43380b1 / 1f882df / 0379e4c) landed independently; my
modifications stack on top cleanly. fox's commits added 155 +
89 lines to test_prometheus.py + test_prometheus_audit.py but
collected counts stayed at 42 + 17 — those landings modified
existing tests (parametrize additions / refactors), didn't add
new test functions. AUTOCOUNT claims from 3b30126 still match.
Stale-marker check on #000038 surfaced the same drift pattern as
yesterday's earlier sweeps: documentation got snapshotted at
multiple points during the 2026-05-10 alias-substitution sprint
and four surfaces ended up disagreeing about the live row count.
Live count via ``arborist --shards-dir SHARDS_DIR alias citation
list | jq length``: **74** rows.
Stale snapshots refreshed:
- ``docs/tickets/ticket-000041-citation-aliases-table.md`` line 3
(status header): "40 rows live by 2026-05-10" → "74 rows live
as of 2026-05-10" with the 40 → 54 → 74 progression noted.
- ``docs/tickets/ticket-000041-citation-aliases-table.md`` line 205
(§7 close-out): "54 rows live" → "74 rows live" + progression.
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
line 3 (status header): "(54 rows)" → "(74 rows live as of
2026-05-10; count grew 40 → 54 → 74 across the day)".
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
§7 line: "54 rows live (decision_by=...)" → "74 rows live as
of 2026-05-10".
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
references line 274: "(40 → 54 rows live)" → "(40 → 54 → 74
rows live across 2026-05-10)".
- ``docs/TICKETS.md`` line 70 (#000041 row): "54 rows live" →
"74 rows live as of 2026-05-10 (count grew 40 → 54 → 74; ...)".
- ``docs/TICKETS.md`` line 73 (#000038 row): "(54 rows in #000041
+ 13 rows in #000042)" → "(74 rows in #000041 + 13 rows in
#000042)".
- ``CLAUDE.md`` line 145: "40 fox-decided citation-aliases" →
"74 fox-decided citation-aliases" + a note about the live-count
command for future readers ("alias counts grow as fox adds
substitutions — ``arborist alias citation list | jq length``
for live count").
The refreshed numbers will themselves go stale next time fox
adds rows. Each surface now also notes the live-count
mechanism so future readers can check current state without
having to refresh the doc — same pattern as today's earlier
"track Phase 3 fire rate via the audit-line tail" closure
note. The CLAUDE.md note specifically tells future readers
where to look up the live count.
Term-aliases count (13 rows in #000042) was verified live and
matches all surfaces — no refresh needed there.
#000038 phases otherwise clean: §7 close-out + §8 residue
already accurately reflect current state (92/92 records resolve;
4 residue items genuinely-open). No phase status drift to fix.
Hygiene: docs-only commit, no code touched. No test impact.
Fox 2026-05-10: blackops kept padding decision menus with a "C — stop
here" option. Encode the discipline directly: when fox asks "next?"
list the actual moves (e.g. high-payoff Phase 3 vs low-risk cleanup
batch), don't dress up "let's call it" as a third choice. Fox keeps
going. If a task is genuinely complete, say so flat instead of as a
proposal option.
CLAUDE.md
=========
Architecture diagram refreshed to include every current arborist/
subdir (was missing capital/, concepts/, memory/, mesh/, pi_star/,
selfmodel/, substrate/, world/ and several top-level modules) and
the qa/ tree got the recently-landed members (canonical_cache.py
#000027, witness.py #000028, warrant_resolver.py #000031). New
explanatory paragraph names the topic-named convention explicitly:
> Topic-named, never version-prefixed. The substrate-paper version
> (v7 plastic-training, v8 selection/consensus, v9 falsification
> controller, …) and the live SQLite schema version (v9.8) are two
> unrelated numbering schemes that share decimals; version-prefixed
> dirs were tried 2026-05-10 and retired the same day because
> readers asked "is this schema-v7 or paper-v7?".
Future blackops shifts looking at where to put a new module now find
the answer in CLAUDE.md instead of having to reverse-engineer the
convention.
arborist/substrate/fork_score.py — fix broken import
====================================================
Previous commit 654d923 (the v7+v8 → substrate refactor) shipped
broken. Sequence:
1. ``git mv arborist/v8/fork_score.py arborist/substrate/fork_score.py``
— staged the rename.
2. Edited the file to flip ``from arborist.v8.weights ...`` →
``from arborist.substrate.weights ...`` — modified the working
tree but did NOT re-stage.
3. Selective ``git add`` of the other touched files (cli.py,
substrate/__init__.py, docs, tests) — fork_score.py's import
edit was NOT in the staged change set.
4. ``git commit`` shipped the rename WITHOUT the import fix.
HEAD's fork_score.py imports ``arborist.v8.weights`` — a path that
654d923 itself deleted. Tests passed locally because the disk had
the edit; a fresh checkout from origin/main breaks at import time.
This commit lands the import edit. Verified: ``git show
HEAD:arborist/substrate/fork_score.py`` now has
``arborist.substrate.weights``; full suite re-runs at 1643 passed.
Lesson: ``git mv`` stages the rename only. Modifications after that
need their own ``git add``. Selective-add commits should run
``git diff --cached`` (not just ``git diff``) before commit. Adding
this to my own habit list.
Internal architecture reference written 2026-05-10 after the day's
18/92 -> 92/92 push under #000031. Covers:
- per-pillar substrate map (which open textbook covers which
pillar; license + ingest path for each of the 18 substrates)
- five proven ingest patterns (HTML single-URL, HTML BFS,
textbook_tex LaTeX-source, PDF -> localhost-HTML, direct
Python API)
- discipline patterns: title-from-author backfill workaround
(until the source-side fix in #000031 Phase 1 lands), alias
audit-fail-closed (decision_by + decision_rationale per row),
multi-substitute pattern, cascade tuning
- honest tier breakdown of the 92 chains (~25 direct primary,
~50 substrate substitution, ~17 soft-fallback OR-of-3 match)
- what the substrate doesn't yet do (render layer doesn't read
derivations, process_id under-attributes alias chains as DIRECT,
no per-record tier classification in the schema)
- re-running steps for future shifts (idempotent at DB layer)
Format follows other docs/ references (cti-architecture,
concept-relations-design, tool-action-dag-design) — describes
state of the world, not proposing change.
CLAUDE.md and TICKETS.md updated to point at the cookbook from
the docs index.
No undefect/whitepaper publication — this stays internal as
requested. Future blackops shifts re-discovering the substrate
map shouldn't have to walk five bench journals.
Update #000031 ticket with the 2026-05-10 push details: 18 textbook
substrates ingested, 40 citation-alias rows + 13 term-alias rows,
cascade tuning in _build_record_query_cascade. Per-pillar end state
recorded (13/13 · 10/10 · 13/13 · 18/18 · 5/5 · 5/5 · 14/14 · 14/14).
Honest tier breakdown of the 92 chains added. Phase 1 follow-up
(source-side title-from-author backfill) and via_citation_alias
attribution fix surfaced as open follow-ups under the same ticket.
Update #000041 with full registry of 40 citation aliases now live
(distribution by substrate; per-pillar breakdown). Update #000042
with 13-row count + acknowledgement that Newton/arithmetic rows are
mostly unused (cascade picked different tokens).
Update CLAUDE.md with the 18 per-textbook make targets + the 100%
warrant-resolve milestone.
Update TICKETS.md index status for #000031, #000041, #000042.
No code change in this commit; documentation refresh only.
SelfModel binds an arborist agent's identity to bytes a verifier can
recompute: model_profile_hash, verifier_method_root, governance hash,
canonicalization/chunking versions, optional patch + memory roots,
sorted capability-claim hashes. Hard-hash committed; no soft state in
preimage. State transitions live on the row, not the body, so the
selfmodel_root stays stable across live → stale → falsified.
Surface:
- arborist.selfmodel.{canonical,snapshot,store,falsify}
- CLI: arborist selfmodel snapshot|show|falsify|list
- Schema: selfmodel_records + selfmodel_capability_claims (additive)
- Audit events: selfmodel_snapshot_landed,
selfmodel_capability_claim_added, selfmodel_falsified,
selfmodel_marked_stale (all chain via existing append_audit)
Also folds in:
- CLAUDE.md operational rule: arborist stays Python-only; non-Python
toolchains live in sibling repos. Forks/clients/servers in any
language follow our schemas + canonical encodings.
- Ticket #000016 update: ZK lives in sibling repo arborist-zk-bench;
arborist gains at most a wire-format consumer, never a Rust dep.
- Schema migrations also stub capital_ledger and memory_records
tables for tickets #000020 and #000017 respectively (additive,
empty until those modules land).
Tests: tests/test_selfmodel.py (14 cases; canonical-JSON stability,
root order-invariance, snapshot determinism, store idempotency,
audit events, falsify/mark_stale semantics, audit-chain integrity).
Full suite: 1012 passed, 36 skipped.
Single canonical entry point that ties together the four existing
bench-related docs (qa-modes-bench.md / bench-maxing.md /
bench-emergent-design.md / qa-modes-bench-2026-04-30.md) plus the
make targets, fixtures, and bench-row schema.
Sections:
1. Two harnesses, two purposes
bench/qa_sweep.py — curated regression bench
scripts/bench_emergent.py — random-word stress test
2. Four question fixtures (75 / 28 / 9 / 1 / smoke) with cell
sizes + use cases per fixture
3. Signal floor (5pp / n=3 × 9 = 27 / vLLM c=3-4 saturation)
4. Make targets cheat sheet (bench-qa, bench-emergent,
--policy KEY=VALUE A/B pattern, --resume)
5. Bench-row schema — every field a row carries
(identity / verdict / diagnostics / preflight projection /
quantifier classifier / capacity / directive compliance / time)
6. Where headlines live (qa-modes-bench.md addenda, per-ticket
§12/§13 bench sections)
7. bench-emergent log shape + #000006 rolling-amend pattern
8. How to run a focused A/B (the four-cell pattern from #000008)
9. How to interpret results (STRICT-rate, mean-ratio,
UNGROUNDED-rate, FORMAT_COLLAPSED rate, violation kind
distribution, audit-line tails)
10. Operator commands cheat sheet (--show-preflight,
--apply-quantifier-caps, --reject-broad, --soft-preflight,
Makefile shortcuts)
CLAUDE.md docs index updated to point to benchmarks.md as the
"read first" entry for bench work, and to add bench-emergent-design.md
to the index (was missing).
Three Explore agents fanned out in parallel for a docs/ + diagrams/
+ code-comment audit against the shipped state of the three
preflight tickets. This commit lands all the alignment fixes.
Core docs updates:
CLAUDE.md
- dag.py module description: stage counts now read
"7/8 quote · 9/10 CTI · 3 reject" reflecting #000009 preflight
stage + reject-broad early-return shape.
docs/cti-architecture.md §2.2 + §2.3
- §2.3 Merkle-AGI-DAG section rewritten: documents all five DAG
shapes (legacy 7/9, post-#000009 8/10, reject-broad 3),
describes the preflight stage's 5 nested CTI clauses
(classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs), pins
PREFLIGHT_NODE_VERSION = "preflight-node-v1", states the
audit-replay payoff.
- §2.2 CTI section: adds the four new modules
(quantifier, model_profiles, quantifier_reminder,
metacognition) as code anchors. Notes that pre-answer
preflight contract extends CTI upstream of retrieval.
docs/seven-point-program.md
- D3 status ½ → ¾ — pre-answer preflight contract landed via
#000008 + #000010. Code anchors + pinning tests updated.
- D4 status ½ → ¾ — preflight stage adds upstream control
commitment to the run-DAG. Code anchors include
build_reject_run_dag + preflight_node_hash.
- Status snapshot table: tickets column now references
#000008/#000009/#000010 against D1/D3/D4 directives.
- "Post-landing addendum (2026-05-03 / 2026-05-04)" subsection
summarises all three tickets + their commit shas + final
test count (993 passing, up from 734).
docs/modules.md
- Q&A pipeline table: added 4 new modules (quantifier.py,
model_profiles.py, quantifier_reminder.py, metacognition.py).
dag.py row updated to "7/8 quote · 9/10 CTI · 3 reject".
- dag.py subsection rewritten: documents all 5 DAG shapes,
describes the preflight payload's 5 clauses + question_state.
- 4 new module subsections (quantifier / model_profiles /
quantifier_reminder / metacognition) explaining each
module's purpose, signature, and how it feeds the run-DAG
preflight clause.
Diagram updates:
docs/diagrams/query-pipeline.dot + .svg
- New "PREFLIGHT (#000008 + #000010)" node inserted between
cache_check and concepts_lookup.
- New "REJECT-BROAD" node showing the 3-stage minimal DAG
escape path.
- render node label extended with the audit-line tail token
catalog.
docs/diagrams/aborist-modules.dot + .svg
- 4 new qa_* nodes in the retrieval & verifier cluster.
- 8 new edges: qa_query/qa_runner each call into all 4
preflight modules; qa_dag has dotted edges to qa_quantifier
+ qa_metacognition (preflight clause sources).
- qa_dag label updated to mention preflight_node_hash + 5 clauses.
docs/diagrams/verifier-ladder.dot + .svg
- Soft-demote violations list extended: BROAD_QUANTIFIER_RUNAWAY
/ CAP_APPLIED / SCOPE_UNBOUND, FORMAT_COLLAPSED, BARE_NAME_CLAIM.
- New "AUDIT-LINE TAILS" annotation node listing all 11 tail
tokens (#000008 broad-* + #000010 metacog + classic verifier).
- Dashed edges from each rung to tails note showing tails
compose onto labels.
Code-side stale-comment fixes (caught by 3rd Explore agent):
aborist/qa/keys.py:218
- "The four fields" → "The seven fields"; mention #000010 adds
six more for metacognition.
aborist/qa/query.py:2644
- 7-stage / 9-stage comment expanded to enumerate all four
base+preflight shapes plus the 3-stage reject path.
aborist/qa/runner.py:835
- same expansion as query.py for runner.ask() callsite.
mesh-*.dot, ingest-pipeline.dot, qa-modes-bench.md, bench-maxing.md,
bench-emergent-design.md, verifier-semantic-gap-design.md,
self-reference-design.md, concept-relations-design.md confirmed
orthogonal — no edits needed.
993 tests still passing (no behavior change). 7 files modified
across docs/ + 3 dot diagrams + 3 SVGs + 4 code-comment fixes.
#000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs):
Metric | Base | Rem | Cap | Cap+Rem
--------------------+------+------+------+--------
JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR
JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue
pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best
pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven
§10.8 strict gate "compound beats either alone by ≥5pp on every metric"
NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR
(clears floor) and beats cap-only by +17pp on pointer mean ratio
(clears floor), but is -3pp vs cap-only on JSON SR (regression,
within noise).
#000008 §12.9 — Final verdict + recommendation:
Mechanism asymmetry (clean signal):
Reminder rescues UNGROUNDED → HYBRID
Cap rescues HYBRID → STRICT
Recommendation: Option A — single-knob defaults, per-mode tailored:
claim_lattice (JSON): apply_caps=True + reminder=True
claim_lattice_pointer: apply_caps=False + reminder=True
(cap can't rescue pointer-tag discipline
upstream of cap; cap fires 20× without
verdict gain)
quote: guard mode-gated off (already default)
Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5
verification on cap+reminder before flipping defaults.
Cross-doc updates:
- CLAUDE.md: architecture diagram now lists the three new Phase 1-3
modules (quantifier.py, model_profiles.py, quantifier_reminder.py).
New "Broad-quantifier preflight guard" conventions entry covers
the 7 policy fields, six-level disable hierarchy, dry-run
defaults, CLI flags, and §12 bench summary.
- docs/qa-modes-bench.md: addendum at end pointing at #000008 §12
four-cell A/B + 4 bench artifact stamps. Original 2026-05-02
journal frozen; 2026-05-03 broad-subset findings flagged as
question-mix-dependent narrative on top of the global
per-mode recommendation.
Closes the FTS5 hyphen-tokenization asymmetry: `bi-polar is rare?`
retrieved only the Bi-Polar album/disambiguation cluster while the
medical-condition cluster (Bipolar disorder, Bipolar I/II disorder,
etc.) sat in the same shards untouched. `unicode61` splits hyphens
at index AND query time; `Bi-Polar Blues` indexes as [bi, polar, ...]
while `Bipolar disorder` indexes as [bipolar] — non-overlapping
token sets that never met.
Fix is query-layer only — no canonicalization_version bump, no
re-index, existing cache_keys stay valid:
- _hyphen_fold_variants(s): emit joined-no-hyphen variants for
every hyphenated run.
- _title_query_tokens(s): additively merges variants symmetrically
(queries AND titles when called on either).
- _filter_by_title_relevance: accept-path 5 — title stem-overlap
with hyphen-fold anchors passes the breadth gate. Rescues
`Bipolar disorder` (1-of-N qtoken match) without disrupting
non-hyphen queries (anchors empty → zero side effect).
- DEFAULT_QUERY_POLICY / DEFAULT_POLICY: hyphen_fold_v1: True
marker folds into governance_policy_hash; new records
cache-split cleanly from pre-fold records.
Live verification on /home/fox/.aborist/shards: same query now
retrieves `Bipolar disorder` (#5) and `Bipolar` disambiguation
(#7); model cites both, answer reads "Bi-polar disorder is not
rare; it affects approximately 2.8% of the U.S. population".
EVIDENCE-WARRANTED 2/2, properly grounded.
Tests: 4 new (3 unit, 1 integration with regression-pinned
Bipolar-disorder retrieval). Full suite 760 passed, 34 skipped.
Also: CLAUDE.md gains a close-when-complete hint for tickets — an
open ticket whose code already shipped is a stale map.
Two cleanup operations bundled (separate scopes, single commit
since they share the doc-tree settle):
1. Move ticket-NNNNNN-<slug>.md files into docs/tickets/. The
directory makes browsing the design log easier; the index
stays at docs/TICKETS.md (top-level pointer). Convention text
in TICKETS.md updated to spell the new path.
2. Delete three docs whose load-bearing content has either been
absorbed into the codebase or distilled into closed tickets:
- docs/naming-deferral.md (147 lines) — explained why we
don't rename claim_lattice → CTI/PROMETHEUS-Σ. Decision
stays in place; the rationale is no longer worth a
dedicated doc. Inline citations removed from
cti-architecture.md (4 refs), warrant.py, ticket-000003
(closed-ticket internal ref).
- docs/reference-frame-failure-class.md (169 lines) — Orwell
case journal that motivated the phrase-pattern retrieval
route. The route shipped; the analysis is now duplicate
with the closed Ticket #000002. Inline citation removed
from CLAUDE.md retrieval pipeline section + frame.py.
- docs/test-coverage-audit-2026-05-01.md (46 lines) —
point-in-time audit checking 16/16 of fox's §11 list. Tests
themselves live in tests/; the audit was a one-shot
checkmark exercise.
References updated:
CLAUDE.md, aborist/qa/frame.py, aborist/qa/retrieval_plan.py,
aborist/qa/warrant.py, docs/cti-architecture.md, docs/TICKETS.md,
docs/tickets/ticket-000003 + ticket-000004 (internal links).
Net: -362 lines + tickets/ subdir. 751/34 tests still pass.
The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
The c=4 sample-shuffled bench at 15:07Z lands the post-Sprint-1b
+ post-Sprint-2 + post-DRY + post-keep-alive + post-shuffle
state.
Headlines:
quote 0.50 → 0.54 (+4pp)
claim_lattice_pointer 0.23 → 0.20 (-3pp)
claim_lattice (JSON) 0.44 → 0.42 (-2pp)
Quote's +4pp is the cleanest lift of the sprint set: the per-mode
24KB cap (Sprint 1b) surfaces tighter retrievals that quote can
ground verbatim, and the bucket data confirms quote peaks at
8-16KB (0.58 strict-rate). JSON's peak migrated to its targeted
32-64KB bucket (0.48 strict-rate, vs 0.38 at 16-32KB) — Sprint
1b's intent confirmed at the per-bucket level even though the
aggregate slipped 2pp.
Pointer's slight drop is consistent with Sprint 2's smoke result
— the chunk-specificity Rule 9 didn't lift Hermes-3-8B's
lazy-anchoring at n=3. The structural fix will need a stronger
intervention than a prompt nudge.
Wall-clock & throughput:
11:31Z: cell-grouped, c=4, n=2, 426 tasks, 51 min, 8.4/min
15:07Z: sample-shuffled, c=4, n=3, 639 tasks, 48 min, 13.3/min
Sample-shuffled scheduling delivers +58% throughput at same
concurrency. n=3 (50% more work) ran in 6% LESS wall-clock.
Per-call mean latency dropped 35-42% across all modes — vLLM's
continuous batcher fills better when fed a diverse request
stream instead of cache_key-correlated cells.
Concurrency sweep: c=3 peak, c=4 within 4% (chosen), c=5 12%
slower, c=6 brutal (45% slower). vLLM saturates at c=3-4 on
this endpoint.
Errors: 6, all on 'tell me about the roman empire' question.
Root cause traced & fixed in 41d1d9b (lone UTF-16 surrogates
in Wikipedia chunk content broke httpx's outbound JSON encode
— different path from the 3b91223 SHA-256 hashers fix which
hardened the OUTPUT side). Next bench: 0 errors.
Captures the post-Rule-8 / post-warrant-generalization /
post-frame-detector bench taken with --concurrency 4 (~51 min
wall-clock on 426 runs).
Headlines (vs 2026-04-30 post-retry rerun):
quote 0.47 → 0.50 (+3pp)
claim_lattice_pointer 0.24 → 0.23 (−1pp)
claim_lattice (JSON) 0.50 → 0.44 (−6pp)
Pointer mode is essentially flat. The earlier 'pointer regressed
13pp' alarm was wrong — that compared to the pre-retry
2026-04-30 numbers (0.36) instead of the post-retry rerun (0.24)
which is the correct reference.
Real news: JSON's 6pp drop is the honesty cost of Rule 8
title-relevance promotion + warrant-class generalization. Lattice
modes now hit 99% directive coverage (D2/D3/D4/D6/D7); the 1% gap
is 2 UTF-16 surrogate errors per mode on one question.
Pointer-mode failure pattern shifted to lazy-anchor: 49/142 rows
have lazy_anchor_ratio >= 0.75. JSON's structured-per-claim
linkage absorbs Rule 8's pressure better than pointer's
prose-with-tags. 19/71 questions show JSON > pointer by 50pp+.
Recommended context budget surfaced for the first time:
- claim_lattice (JSON) peaks at 32-64KB (0.51)
- pointer + quote peak at 16-32KB
CLAUDE.md headline numbers refreshed; docs index points to the
new journal alongside the prior 2026-04-30 reference.
Three small fixes flagged by the whitepaper rewrite sweep:
1. Note the four-rung ladder render in the providence-cache layer
description (audit_mode is schema-level; lattice modes display
POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED, with
ENTAILMENT-VERIFIED reserved).
2. Add a Conventions entry for the ladder right after the three
answer modes — names the rung discrimination rules, what each
violation demotes to, and that quote/span/entity/paraphrase
modes keep their original tokens.
3. Replace the 'Open tickets' section with a one-liner pointing to
docs/TICKETS.md as authoritative; all five shipped tickets are
closed as of 2026-05-02. The previous listing claimed #000001-3
were open, which has been false since the prior session.
New `docs/concept-relations-design.md`: architecture reference for
the per-shard concept_relations layer that replaced the legacy
frozenset module (commit 5fd458a). Covers:
- Why phase 1 (hand-curated frozensets) didn't scale.
- Append-only schema + the three by-construction properties (idempotent
re-derivation via UNIQUE, per-shard storage, Merkle-orthogonal).
- Built-in `link_reciprocity_synonym` extractor reading the existing
`edges` table — no new crawler, works for Wikipedia AND HTML sites.
- Measured storage: 95.58 MB across 4 wiki shards (3.47M docs,
10.75M resolved edges, 55,148 reciprocal pairs, 289,848 synonyms),
4m16s wall-clock backfill. 1.6% tax on the 6 GB corpus.
- Three storage compactions considered & rejected, each with the
specific trade-off it loses on (drop idx_concept_evid → painful
purge debugging; BLOB source_root → schema inconsistency; FK
normalization → JOIN in retrieval hot path).
- How-to: backfill, manual add, purge.
- Adding new extractors.
- Deferred follow-ons (CLI commands, Wikipedia See-also extractor,
category extractor, hatnote extractor).
CLAUDE.md item 5 in the retrieval-pipeline list updated to point at
the new module path (aborist/concepts/) and the design doc.
TICKETS.md reference list updated to mention the new design doc.
Promotes the title-relevance sidecar (diagnose_title_relevance,
qa/inspect.py) to a hard check inside both verify_claim_lattice
and verify_claim_lattice_json. New violation kind TITLE_MISMATCH;
demote-to-HYBRID semantics matching the existing WARRANT_MISSING
pattern.
Catches the 2026-05-02 fox-surfaced retrieval-driven hallucination
class:
Q: "explain spin glass modeling & tensors?"
Pre-Rule-8: EVIDENCE-WARRANTED 1/1, claim cited to Quantum
chromodynamics chunk (single-line "See Also: spin
glass" reference). Token-coverage check passed
accidentally on shared physics vocabulary.
Post-Rule-8: POINTER-LINKED-PARTIAL · title mismatch 1/1.
Cited source title 'Quantum chromodynamics' shares
zero stemmed tokens with claim's {spin, glass,
modeling, tensor, ...} → demote.
Implementation:
_claim_title_overlap(claim_text, source_title) returns True iff
the source title shares ≥1 stemmed content token with the claim.
Uses qa.evidence._content_tokens (≥4-char, post-stopword) and
inline minimal stem (s-strip on tokens >4 chars, skip ss-enders)
to avoid an import cycle.
Per-claim loop in both verifiers checks every cited source's
title; ANY-match suffices (only TITLE_MISMATCH when ALL cited
titles miss). Vacuous-pass when claim or title has no
extractable tokens.
Renderer (_ladder_rung_for_lattice) treats TITLE_MISMATCH
alongside WARRANT_MISSING as the POINTER-LINKED-triggering
signal — both indicate citation/claim structural misalignment.
_render_warrant_tail surfaces "· title mismatch" alongside
"· warrant missing" so an operator sees the specific failure
mode at the audit-line.
Test fixtures updated where the synthetic claims were too minimal
(e.g. "Velociraptors are shown attacking workers" cited to
"Jurassic Park (film)" — claim had no topic anchor). Real model
output naturally references the topic (the model sees the title
in the evidence map and uses it); the fixture revisions reflect
that. 4 new tests in test_verify_json.py covering the helper +
end-to-end TITLE_MISMATCH demote.
Live verification:
Spin-glass query: POINTER-LINKED-PARTIAL · title mismatch ✓
Homer/Mr. Burns: EVIDENCE-WARRANTED ✓ (no regression)
CLAUDE.md updated with the Rule 8 convention; existing
diagnose_title_relevance sidecar marked legacy / dict-form for
per-cache-key inspect use.
Full suite: 738 passed (was 734, +4).
The 2026-05-01 distilled architectural directive becomes the
audit lens applied before every new ticket / feature / prompt
edit. Each of the seven directives gets:
- status (✓ / ½ / ✗)
- code anchors (verifier paths, renderer hooks)
- pinning tests (per-rule coverage in tests/)
- tickets covering remaining work
- bench signal (which row column reports it)
Status snapshot at landing:
D1 (no LLM in verifier) ✓
D2 (pointer clauses) ✓
D3 (CTI internally) ½ → ticket #000002
D4 (retrieval map AND evidence map) ½ → ticket #000001
D5 (deterministic pointer verify) ✓
D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair)
D7 (honest labels) ✓
D8 (test-pinning before automation) discipline
CLAUDE.md docs index gains a "North-star" section pointing at the
program; ticket entries gain a "Directive" column threading them
back to the program.
Walked the five-step algorithm on CLAUDE.md itself.
Step 1 — requirements check: every section earned its place via a
specific operator-failure context. Sections that had grown into
prose essays got compressed back to load-bearing rules + pointers.
Step 2 — delete:
- Bench-maxing prose section (~100 lines) → moved verbatim to
docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list
pointing at the doc.
- Conventions section: each rule tightened to one paragraph max,
with a `See aborist/qa/verify.py` (or similar) trailing pointer
so the operator can jump straight to the rationale in code.
Verbose duplications between conventions and retrieval-pipeline
sections (e.g. wikitext base prose explained twice) collapsed.
- Retrieval pipeline: each of the 9 stages now references the
relevant module (`qa/concepts.py`, `query.py`) instead of
re-narrating the failure case in prose.
- Architecture tree: minor trim, removed redundant comments
where the filename already names the role.
Step 3 — simplify: source papers section dropped one item that was
duplicate (PDF + RST point to same content); kept the canonical
source.
Steps 4 + 5 — n/a (this is a doc, not a process).
New section: Docs index. Lists every architectural / design doc
in docs/ plus a ticket sub-list (open tickets with their files).
Closes the discoverability gap where TICKETS.md existed but agents
didn't know to look at it.
Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information
density is up — bench prose doesn't bloat the entry-point doc, and
each convention now points to its source. The `docs/bench-maxing.md`
extraction is the real win: separation of "rules I must not break"
(CLAUDE.md) from "discipline I should internalize" (bench-maxing.md).
Engineering discipline added between operational rules and the
orientation protocol. Walk the steps in sequence; skipping makes
the next ones expensive.
1. Make the requirements less dumb (every requirement gets a name,
not a department).
2. Delete the part or the process (10%+ put-back rate or you
aren't deleting hard enough).
3. Simplify and optimize (only after 1 + 2).
4. Accelerate cycle time (only after simplifying — don't dig the
grave faster).
5. Automate (last, not first).
Connected to existing aborist conventions:
- Step 1 traces every policy knob / schema invariant / verifier
check to a specific defect or owner. Aligns with the
"feedback memory" pattern (record the WHY).
- Step 2 reinforces verifier-stays-binary and "no soft signals
in the hard chain" — both are deletion-first rules.
- Step 5 is why bench harnesses, distill runners, and ingest CLIs
earned automation only after their underlying ops were
simplified to single-purpose stages.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Catch-up after a productive afternoon. Three new conventions section
items + retrieval-pipeline addendum:
- Three answer modes documented: quote, claim_lattice_pointer (G0/CTI),
claim_lattice (JSON). Bench evidence (2026-04-30) on Hermes-3-8B:
quote 0.47 strict-rate, pointer 0.36, JSON 0.49. Both lattice modes
share verifier_method='claim_lattice' (CHECK constraint
compatibility); answer_mode + json_fixups disambiguate downstream.
- Claim-count ceiling (TOO_MANY_CLAIMS): default 12/answer, demote
STRICT→HYBRID without truncating. Bench finding context
(york-england runaway) preserved so future-me knows why the cap
exists.
- Deflection sidecar (qa.inspect.diagnose_deflection): subject-anchor
heuristic catches Mars-BDFL pattern. Sidecar-only — never feeds
back into providence. Wired into bench/qa_sweep.py as a per-row
signal + summary-table column.
- Retrieval pipeline stage 9: template-phrase stopwords. The
'all there know' additions closing york-england, with the AND-mode
FTS5 mechanism documented so a future change to MATCH semantics
doesn't accidentally re-open the miss.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Adds "Bench-maxing — measure deltas, not opinions" section to
CLAUDE.md capturing the patterns established during the
2026-04-30 JSON-mode hardening journey (retry → trim-and-verify
→ pointer-IDs → stop-sequence → self-heal):
- Bench before AND after every change. n=3 single-sample
noise is ±20pp; signal under 5pp is unreliable.
- Slight prompt adjustments are fine — local-minima are not.
One-thing-per-bench-cycle so deltas are attributable.
- Bench is the scoreboard; live fixtures
(tests/test_qa_quality_live.py) are the gates.
- Self-heal beats retry — repair malformed output rather
than re-running the LLM. See _repair_truncated_json.
- Name the failure → fix → re-bench → confirm. Each commit
references the specific failure mode it targets.
- Honest HYBRID beats false-positive STRICT every time.
Verifier improvements drop STRICT count when the count was
inflated by token-coincidence false-positives — this is
a WIN, not a regression.
- Per-question fixtures use entity-presence not byte-match;
Hermes is non-deterministic and intermittent failures are
quality-drift signals to investigate upstream, not test-
flake to suppress.
- Doc captures the journey not just end state — future-you
reads docs/qa-modes-bench-*.md and sees WHY each commit
landed.
Architecture diagram gains aborist/qa/evidence.py and
aborist/qa/parse_claims.py; aborist/qa/dag.py callout makes the
quote-vs-pointer DAG-shape split explicit. New conventions entry
covers the full G0 contract — two-layer ids (pointer-id E1/E2
shown to the model, evidence-id E######## used by cache & DAG),
pointer-line protocol, deterministic verifier (no entailment in the
proof path), strict no-quotes rule, spotlight rendering with
literal-span interpolation, per-chunk evidence granularity with
query-relevance ordering, 9-stage CTI run-DAG, no iterative repair,
and the Makefile testing default vs library defaults split.
The retrieval contract was undocumented — `_filter_by_title_relevance`
has three accept paths, each with its own scaling rules, and the recent
breadth + stem refinements (2026-04-29 catches) lived only in commit
messages. New "Retrieval pipeline" section between Live endpoints and
Hot path / gotchas covers the eight stages in order:
1. Parallel FTS5 (body-BM25 + title-LIKE), unioned per shard
2. Body-coverage sqrt rerank — counter BM25 short-doc bias
3. Title-token boost
4. Three accept paths in _filter_by_title_relevance:
title overlap (breadth-scaled: ≤2 require ALL, 3+ require N-1),
TF-IDF core keyword match, body density (same breadth scaling
plus min_mentions=3 depth)
5. Rivalry exclusion (Intel/AMD groups)
6. Stem-aware token matching: trailing-`s` strip on >4-char tokens
(skip `ss`-enders) — 'supermans girlfriend' now collapses onto
'superman'/'girlfriend' instead of admitting `Girlfriends` TV show
7. Per-source context cap = max_context_chars / top_k — prevents
80 KB+ bibliographies from monopolizing the budget
8. Wikitext base-prose normalization before LLM (cross-ref to
the Wikitext base prose convention)
Each item includes the commit ref or the catch date so an operator
can trace why the rule exists.
Different agents have different value functions on the same providence
records. Today aborist bakes one canonicalization policy into
governance_policy_hash and calls it universal. This adds two knobs that
let agents express their preference without breaking provenance.
- aborist/qa/keys.py: `canonical_question(q, mode=...)` and
`question_hash(q, mode=...)` accept "strict" (NFC + ws-collapse only;
every variant gets its own hash) or "equivalence_class" (default —
additionally lowercase + trailing-punct strip + article strip).
New constants: QUESTION_DEDUP_MODES, FIDELITY_MODES, defaults.
- aborist/qa/query.py + aborist/qa/runner.py: each call computes the
primary cache_key under policy["question_dedup"]. New `fidelity`
parameter:
"strict" only primary cache_key checked
"equivalence_class" primary first; if miss AND alternate mode
produces a different cache_key, try alternate
Cross-silo fallback works because _ckey_for_mode rewrites
policy["question_dedup"] to the alternate mode before computing
governance_policy_hash — so the fallback ckey matches what an agent
under that mode would have written. Lookups can find each other's
records when fidelity permits. Result dict gains `lookup_path` ∈
{"strict", "equivalence_class", "strict_fallback",
"equivalence_class_fallback", "miss"}.
- aborist/cli.py: `--question-dedup` and `--fidelity` flags on the
`query` subcommand. Human render annotates cache_hit lines that
came from a fallback ckey ("cached via equivalence_class_fallback").
- tests/test_query.py: three new regressions
* strict-policy + strict-fidelity: variants get distinct cache_keys
* cross-silo fallback: strict-policy reads eq-class-policy records
via fidelity=equivalence_class
* strict-fidelity refuses fallback (audit-grade): cache miss even
when the alternate silo has a hit
- CLAUDE.md: dedup-mode and fidelity convention bullets updated.
This is V1 + V2 of the substrate move sketched in conversation: write
policy determines which silo a record lives in; read fidelity determines
how loosely an agent walks across silos. Records stay exactly-keyed
(provenance hard); routing is per-agent (preference soft). Same
verifier_no_diagnostics discipline holds — no soft signal enters the
hard chain.
442 tests pass.
- Top-level providence summary now mentions 4 strategies (quote, span,
entity, paraphrase) and that verifier_method records which fired.
- audit_mode bullet expanded with strategy 4 (paraphrase): token-coverage
≥0.85 with ≥4 content tokens after stopword filter, fires only on
prose-shaped spans (entity strategy still owns proper-noun lists).
Note that quote strategy gets NO paraphrase fallback — verbatim
citation is the contract.
- Trailing-citation strip documented: peels (Source: ...)/URL tails
before substring testing, refuses genuine prose parentheticals.
- Sidecar invariant now points at aborist inspect --cache-key X:
read-only, diagnoses each unverified span (verbatim_in_base /
trailing_artifact / paraphrase / partial_paraphrase / no_overlap),
never writes to providence_cache or audit_events.
- File tree shows aborist/qa/inspect.py; CLI line lists `inspect`.
Notes that `query` human-renders by default (--json or JSON=1).
- New question_hash convention: trailing-punctuation strip + standalone
English article strip so 'who is X', 'who is X?', 'who is the X' all
hash to the same equivalence class.
Whitepaper section 13.8 already current — fox shipped 13.8.5 (citation
strip) and 13.8.6 (sidecar) plus the 4-strategy 13.8.1 rewrite.
- Wikitext base prose section now reflects reality: to_base() runs in
two places (before LLM call in query.py/runner.py AND inside
verify_quotes) gated by policy["base_version"]. Both sides see prose;
Hermes can quote source paragraphs verbatim instead of escaping
[[wikilinks]], and Wikipedia chunks ship with ~43% fewer tokens.
- Build/test block now points at the real targets: verify-shards,
analyze-shards, chain-check-shards. The bare 'make analyze' target
doesn't exist — only the cross-shard variants do.
- New "Hygiene after any state-changing op" callout: chain-check-shards
for one-second per-shard break count, analyze-shards for full audit.
- Audit-chain invariant block points at the same targets.
- Orientation protocol updated: --shards-dir replaces --db (the latter
refers to a single-DB setup that's no longer the canonical path), and
chain-check-shards is added to the per-session ritual.
- aborist/qa/verify.py: three strategies tried in sequence. quote uses
sequential pairing (1st & 2nd `"`, 3rd & 4th, ...) which eliminates
the phantom inter-pair captures regex pairing produced on adjacent
quote pairs like `"title" prose "quote"`. span checks bullet/sentence
lines verbatim. entity matches multi-word proper nouns; gated by
policy ∈ {strict, hybrid, drop, proximity}. Default proximity
promotes to STRICT only when N=3 verified entities cluster within
W=300 chars in source — separates structural grounding (cast list,
infobox) from incidental mention (scattered plot summary).
- aborist/store.py: schema CHECK now `('STRICT','HYBRID','UNGROUNDED')`.
Migration helper rebuilds legacy `('STRICT','HYBRID','VISUAL')`
tables via temp-table copy, translating VISUAL → UNGROUNDED in the
SELECT. Idempotent — DDL inspection skips the rebuild on already-
migrated DBs.
- aborist/cli.py: new `aborist reclassify` re-runs the verifier
against existing live records under the current entity policy; no
LLM calls. --compare runs all four policies side-by-side, --dry-run
reports transitions without writing. Cold-source records skipped.
One providence_reclassify audit event per changed row.
- AuditMode.UNGROUNDED replaces VISUAL across search backend, FTS5,
test fixtures, CLAUDE.md. The substrate (Merkle-AGI) name was
about FOR-style visualization; in the RAG layer the semantic is
"no recoverable grounding," so the label now says that.
DEFAULT_QUERY_POLICY gains entity_policy + entity_proximity_n +
entity_proximity_window so any tuning folds into governance_policy_hash
and invalidates cache cleanly.
Adds aborist/qa/verify.py — three-strategy verifier (explicit quotes,
bullet/sentence spans, multi-word proper nouns) that lexically checks
every claim against retrieved context under norm-v1 + lowercase. The
strategy that fires is recorded as verifier_method for diagnostics.
Entity strategy gates classification via an entity_policy
(strict/hybrid/proximity/drop) so a single proper-noun match no longer
overclaims STRICT — proximity (default) requires a cluster of 3+
verified entities within 300 chars.
Wires into ask() and query(). System prompts now require verbatim
quoted spans for every factual claim, restated via a user-turn
grounding_reminder one message before sources arrive (recent
user-turn instructions outweigh decayed system-turn rules under long
context in Hermes).
providence_cache gains 5 columns (audit_mode, n_quotes, n_verified,
unverified_quotes, verifier_method) with CHECK constraints. A
connect-time _migrate_audit_mode() ALTERs legacy DBs idempotently.
Cache hits return the persisted audit_mode rather than asserting
STRICT unconditionally.
CLI:
emergent list VISUAL/HYBRID records; --aggregate ranks
unverified quotes by frequency (corpus-growth signal).
reclassify re-run the verifier against live providence records;
cold-source records skipped; --dry-run reports
transitions without writing; each change writes one
'providence_reclassify' audit event.
The stub from `git init` predated everything. Replace with a concise
working map: what aborist IS now, where the source papers live, what
the schema invariants are, what conventions never to silently change
(merkle ordering, chunker default = chunking_version bump = mass cache
falsification), where the live endpoints are, and the parser hot-path
gotchas.
Adds: source-paper inventory, architecture tree, schema invariants
(8-dim cache key, audit chain integrity, surface/core kinds, hot/warm/
cold tiers), Merkle conventions, live endpoint URLs, hot-path
performance notes, orientation protocol that runs `make test`,
`stats`, and `analyze`.
Preserves: blackops identity rules, no Co-Authored-By, fail-closed,
PYTHONUNBUFFERED for long-running processes.
