Brain-stormed off opencompletion's activity24-math-plot.yaml
(SymPy + numpy + matplotlib pipeline). Three things in there
map to π* shapes; one doesn't.
Maps:
- algebra-symbolic@v1 — symbolic expression → expand + canonical
ordering. Closes the (x+1)**2 ≡ x**2+2x+1
equivalence class.
- calculus-derivative@v1 — d/dx(f) via sp.diff, re-canonicalized
through algebra-symbolic@v1.
- calculus-integral@v1 — ∫f dx via sp.integrate; sentinel for
unevaluated cases.
- calculus-limit@v1, calculus-series@v1, linear-algebra@v1 —
additional SymPy-friendly phases.
- function-sampled@v1 — bridges symbolic expressions to the
existing time-series-quantized@v1
format. Two functions that render
identically (within sample tolerance)
collapse to same canonical bytes. This
is what plotting CAN become in π* terms.
Doesn't map:
- PNG plot rendering — different DPIs / fonts / palettes all
valid; image bytes aren't canonical. Stays as an output adapter
that COMPOSES with function-sampled@v1.
Hard constraint: SymPy is an OPTIONAL dependency ([math] extra).
Each new π* registers only when sympy is importable, mirroring
the existing html / wikitext / crawler pattern. Fresh checkout
without sympy keeps passing the full test suite (graceful skip).
Multi-phase rollout. Recommendation: Phase 1 (algebra-symbolic@v1)
+ Phase 2 (calculus-derivative@v1) in one commit (~400 LOC total
+ tests + 5S fixtures). Subsequent phases (integral, limit,
series, linalg, function-sampled) each their own commit.
Forward links in the ticket: #000027 (canonical persistence —
algebra/calculus answers inherit audit chain for free), #000028
(witness — symbolic answers become witnessable), 5T/5F (new
fixture surface for symbolic LLM calibration), composition algebra
(deriv+arith, expr+sample compose naturally).
Index updated. Next ID 000031.
ClaimPackSource ingests Grok-4 companion bundles (axiomsg4-v2.json +
theoremsg4-v2.json) at the right grain — one Document per axiom or
theorem record. Each record carries Δ (LaTeX symbolic) + ∇verbose
prose, explicit source citation (Mendelson, Enderton, Hilbert,
Newton, Kolmogorov, Łukasiewicz), foundational-group taxonomy, and a
runicLabel that rides as soft metadata only (runtime mints its own
pointer IDs per CTI architecture). Pillar-level
provenance.references arrays become outbound pillar_reference edges.
Lenient JSON parser strips ```json fences and double-escapes lone
LaTeX backslashes (\Theta, \heart, \vec) without corrupting
already-correct \\to pairs — walks left-to-right and pass-throughs
legal escape sequences. Malformed bundles raise rather than return
empty; silent zero-doc would be a footgun.
CLI surface: --source claim_pack with a repeatable --bundle FILE
flag mirroring html source's --url action=append. Single --path
also accepted for one-bundle ingest.
Drive-by: removed a function-local `from arborist.store import
connect` inside _cmd_ingest's providence branch that was shadowing
the module-level binding via Python's "any local assignment makes
the name local for the entire function" rule, breaking every
non-providence ingest with UnboundLocalError. Comment left in
place explaining why not to re-add it.
Smoke-tested on /home/fox/Downloads/{axiomsg4,theoremsg4}-v2.json
end-to-end: 78 docs (55 axioms + 23 theorems across 7 pillars),
14 deduped pillar-reference edges, 78 audit events, 10/10 sampled
Merkle proofs verify, FTS5 search returns Modus Tollens for
"modus tollens".
Honest ceiling: kind=surface for every record. The pack is
pre-distilled but its provenance is asserted not proven — until
Mendelson/Enderton/Hilbert texts are themselves ingested as
surfaces, the verifier has no derivations.proof_blob to compute
and claim-pack records max out at ANCHOR-WARRANTED on the
four-rung ladder. That's a follow-up ticket, not this one.
Hard constraints honored: no new audit ledger (audit_events
remains the only chained-sha256 ledger; bundle's self-validation
fields ride as metadata only); no kind=core without surface
ancestor; cache_key invariants untouched.
15 unit tests cover lenient parser, slug stability, ref
resolution, doc grain, URI stability, content layout, extra
metadata, edge emission, error paths. All 1280 tests in
make test pass.
Closes#000027. Closes#000028 (cache-leg wired).
#000027 — canonical projections persist to providence_cache
============================================================
Math/logic π* answers (arithmetic@v1, logic-kernel@v1,
time-series-quantized@v1, …) are now first-class providence rows.
Pre-fix: question → kernel → answer → return. No cache, no audit
event, no run_dag, no inspect/burn/replay surface.
Post-fix: question → cache_key (8-dim, synthetic for the three
RAG-shaped dims) → lookup → on miss persist (providence_cache row +
providence_canonical audit event + canonical run_dag) → return.
Synthetic cache_key dimensions for canonical rows (per ticket §2.2):
- source_root = sha256("pi_star_source:" + pi_star_ref)
- model_profile_hash = sha256("pi_star_model:" + pi_star_ref)
- conversation_hash = sha256("pi_star_conv:" + canonical_q + ":" + ref)
- chunking_version = literal "n/a-canonical" — chunker bumps on
wikipedia path don't stale math answers.
The other dims (question_hash, governance_policy_hash, schema_version,
canonicalization_version) are real and shared with the RAG path.
Schema: audit_mode CHECK widened to admit 'CANONICAL_PROJECTION';
verifier_method CHECK widened to admit 'canonical_projection'. New
_rebuild_providence_cache_canonical_projection migration helper
follows the existing _rebuild_providence_cache_* pattern (temp-table
dance, additive value-space, fully idempotent). Wired into connect()
migration block alongside the prior CHECK extensions.
Cache-hit policy: trust the row. Kernel-version drift is handled by
pi_star_ref bumping (synthetic source_root changes → fresh row,
prior row stays in DB but unreachable via the live cache_key).
Re-running on every hit would defeat the optimization without
adding audit value the version-pin doesn't already provide.
Policy gate: canonical_projection_preflight_persist (default True).
Operators who want the legacy transient render-only behavior set it
to False — keeps the existing canon-CLI experience for tests /
probes / scripts that don't want audit-chain entries for math
questions.
CLI render: `CANONICAL · via canonical_projection` for persisted
rows. Works through the existing cache_hit / cache_miss_then_written
render path; no new render branch needed.
`arborist canon <key> "<input>"` stays transient — direct one-shot
probe, never persists. Boundary preserved per ticket §2.6.
#000028 — multi-modality witness cache-leg
==========================================
Pre-#000027 the witness cache-leg closure always returned None;
STRICT-WITNESSED (3-of-3 byte-equal) was structurally unreachable.
Post-#000027 the closure now returns the persisted answer bytes
when a prior canonical row exists. Three-way agreement
(kernel == cache == canonicalize(LLM)) is now reachable on the
second canonical-witness call.
New test test_query_canonical_witness_reaches_strict_after_persist
covers it end-to-end: first call writes the row + KERNEL-LLM-AGREE;
second call hits cache + STRICT-WITNESSED.
Tests
=====
- tests/test_canonical_cache.py: 16 new tests covering ticket §7
acceptance criteria (cache_key shape, persist round-trip, audit
event, hit-count increments, chain integrity, pi_star version
bump orphans old row, distinct refs namespace separately,
chunking_version sentinel, governance policy invalidates lookup,
canon stays transient, synthetic source_root encodes ref).
- tests/test_canonical_projection.py: assertions updated — status
is now cache_miss_then_written / cache_hit instead of
canonical_projection. Added a transient-mode test pinning the
policy gate.
- tests/test_witness.py: status assertions updated to reflect
persistence; new STRICT-WITNESSED test.
- tests/test_directives.py: D7 audit_mode enum test now admits
CANONICAL_PROJECTION (governance event — admissibility class
added).
Full suite: 1367 passed, 36 skipped (was 1306; +61 new).
Real-shard smoke
================
$ make query Q="0.1 + 0.2" BURN=1
→ cache_miss_then_written, ~300ms wall, row written
$ make query Q="0.1 + 0.2"
→ cache_hit, ~40ms wall, hit_count++
$ make chain-check-shards
→ 0 breaks per shard
Acted on response review (2026-05-09,
response_ticket-000027-canonical-projections-in-providence-cache.txt).
#000027 — Hard-constraint phrasing corrected. Original draft said
"no schema changes / no new admissibility mode," which would
mislead an implementer into skipping the CHECK-constraint
extension. The current SCHEMA_SQL CHECK rejects both
'CANONICAL_PROJECTION' (audit_mode) and 'canonical_projection'
(verifier_method) — five locations in store.py listed in §9.1.
Updated to: "no new columns / no new tables / 8-dim cache_key
preserved / CHECK widened via existing rebuild migration." Same
spirit as the original draft, but unambiguous. §2.6 amended to
note the CHECK widening is value-space, not column/table change.
Forward-link added pointing at #000028 as the witness layer that
runs ON TOP of this persistence primitive (not a Phase 2).
#000028 — Post-MVP follow-up appendix. MVP shipped in 656b573
(witness.py 394 LOC + tests 354 LOC, 28/28 passing). Five items
captured for follow-up:
- terminology: ticket uses "modality" but kernel/cache/LLM are
epistemic witness channels, not carrier modalities; suggest a
one-line clarification near §1 to prevent cross-carrier
misreads.
- capital ledger integration (#000020): MVP records LLM latency
on the result dict but doesn't thread cost into the ledger;
~15 LOC follow-up to wire it; ForkScore (#000012) needs this
to compare witness-on vs witness-off forks honestly.
- sample-rate policy field (canonical_witness_sample_rate):
explicit out-of-scope per §2.4, but flagged so it isn't
re-discovered when calibration-data hunger appears.
- cache-leg dependency on #000027: MVP works with stub closure
returning None; STRICT-WITNESSED is unreachable until #000027
lands; swap is one-line when persistence ships.
- 5F / 5T bench-data integration: divergence events are
supervised calibration data; suggest a future
`make bench-witness-divergence` target.
- threat-model caveats from review §7: correct claim is "no
clean single-channel adversarial path," not "no adversarial
path"; future dual-kernel witness option captured.
No code changes in this commit — ticket text only.
Design doc only; no code in this commit. Spec the persistence pattern
for canonical-projection answers — same providence_cache table, same
audit chain, same admissibility ledger, distinguished by
verifier_method="canonical_projection" and audit_mode=
"CANONICAL_PROJECTION". Synthetic-but-deterministic values for the
LLM-only cache_key dimensions (source_root, model_profile_hash,
conversation_hash) bound to pi_star_ref so kernel-version migrations
work via the existing falsification pipeline.
chunking_version = "n/a-canonical" sentinel keeps RAG chunker bumps
from mass-staling unrelated math/logic answers.
Builds on existing _canonical_projection_preflight() short-circuit
(currently transient compute, no row written). Ships with #000027 and
fox confirmation on §8 open questions before code lands.
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.
## Lazy concept_relations loading
Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.
Refactored arborist/concepts/query.py:
- New _load_neighbors_for(shards_dir, tokens) — targeted
WHERE token IN (...) OR target IN (...) query. Returns just the
direct synonym neighborhood for the given tokens (~300 rows for
a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
expansion exceeds max_total (the cap). Most queries never reach
the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
loaders. Eager _load_indices / _get_indices kept for any
back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.
All 14 concept tests pass unchanged — the contract is preserved.
Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):
metric pre-fix post-Phase-1 post-lazy-concepts
wall_ms 14,500 13,400 9,300 (-36%)
search_ms 9,900 10,600 5,000 (-49%)
SQLite executes 10,623 3,687 3,708 ~same
synonym_expand 2,966 2,840 ~0 (lazy hit)
Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).
## Real-shard baseline artifact (Phase 2)
bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:
- bench/results/real-shard-baseline.json (durable; commit_sha,
shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)
Question set in bench/fixtures/real-shard-baseline-v1.jsonl:
virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
(rivalry path), and two canonical-projection cases (math + logic
preflight short-circuit).
First baseline run (commit e78814c plus this fan-out, BURN=1):
audit_mode n notes
STRICT 4 (virt-back, France, Mac OS X, Linux)
HYBRID 2 (Microsoft founder, AMD/Intel)
CANONICAL 2 (0.1+0.2, A IMPL B; <1 ms each)
wall median 4.1 s (range 0.6 ms – 7.2 s)
primary used 4 / 8
`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).
Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.
`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.
## Test status
Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
`connect()` used to run executescript(SCHEMA_SQL) + 7 forward-
migration probes on every open. Profile of `who wrote virt-back?`
on 38 GB of real shards (warm cache) showed 588 connect() calls
per query, each running the full probe sequence — 10,623 total
SQLite executes. Migrations are forward-only and idempotent within
a code version, so once we've run them on a path in this process
there's no work to do on subsequent opens.
Cache shape: `set[str]` keyed by `str(Path(p).resolve())`.
Migration block runs once per (path, process); subsequent calls on
the same shard skip it entirely. Per-connection PRAGMAs
(foreign_keys=ON, synchronous=NORMAL, cache_size, temp_store,
mmap_size) still run every time — SQLite scopes foreign_keys
per-connection and our schema's FK CASCADE behavior depends on it.
That's why `PRAGMA foreign_keys = ON` moved out of the cached
SCHEMA_SQL block into the always-run pragma section.
Cache invalidation: explicit only.
`store.invalidate_migration_cache(path)` for callers who replace a
shard at the same path (snapshot-restore flows). `_clear_migration_
cache()` for tests. We don't auto-detect file replacement —
(dev, inode) is unreliable under tmpfs inode reuse, and (mtime,
size) drifts naturally as SQLite operates on the file (WAL
checkpoints, page growth). Path-only with explicit invalidation
is the honest contract.
Re-profile (same query, same shards, warm cache):
metric before after
_migrate_* (each function) 586 7 ← per shard
executescript 586 7
SQLite executes 10,623 3,687 (-65%)
wall (warm) 14.5 s 13.4 s
The warm-cache wall delta is small because the probes were many-
but-cheap; residual cost lives in FTS5 search (6.7 s) and
synonym_expand (2.8 s, both separate concerns). The 65% execute
drop is the cold-cache win — each redundant executescript() had
been triggering disk reads at the 75 s scale the reviewer reported.
Tests (6, all green): first connect runs all 7 probes, second
connect runs zero, schema integrity preserved across re-opens,
explicit invalidation re-probes, distinct paths each get one probe,
clear-cache helper works.
Full suite: 1306 passed, 36 skipped. Found and fixed an FK CASCADE
regression mid-implementation: PRAGMA foreign_keys = ON was inside
SCHEMA_SQL, so memoization was silently turning it off on subsequent
opens. test_burn_doc.py caught it. Moved to the per-connection
pragma block.
Ticket #000026 status: Phase 1 landed; Phase 2 (baseline artifact)
and Phase 3 (warrant-quality finding) queued.
Profiled the `virt-back` query against ~38 GB of real shards
(2026-05-08, warm cache). Total wall 14.5 s; LLM 2.7 s; search
9.9 s — that's the latency budget breakdown.
Headline finding: 588 SQLite `connect()` calls per query, each
running 7 forward-migration probes on already-fully-migrated
shards. 10,623 SQLite executes total. Reviewer's 75 s cold-cache
report tracks the same shape; warm cache only masks part of it.
Ticket bundles:
- Phase 1 (latency fix): per-process migration memoization. Smallest
patch; no schema impact; ~30 LOC + regression test. Target <5 s
search on real shards.
- Phase 2 (baseline artifact): bench/scripts/real_shard_baseline.py
+ bench/results/real-shard-baseline.json + make bench-real-shard.
Captures wall/search/LLM timings, primary-source-at-rank-1, used
flags, capital ledger + memory deltas, ForkScore preview.
- Phase 3 (findings): fold the warrant-quality observation
(copyright-footer vs author-metadata) into the baseline report
as a finding. Implementation deferred to a follow-up ticket.
Out of scope captured explicitly: authorship warrant ladder
(separate ticket post-latency), CI re-enable (small commit, not
a ticket), public docs refresh (adjacent, separable), ForkScore
live wiring (#000012 Phase 1b).
Index updated. Next ID 000027.
Reviewer note (2026-05-08): the review claimed the index showed
"5 open, 24 closed" but no count line ever existed in TICKETS.md;
the 6 active rows (5 open + 1 rolling) are correctly listed. No
fix needed.
`make query Q="0.1 + 0.2"` used to return `no_sources` because
arithmetic-shaped input has no FTS5 hits in any text shard. Two
surfaces close that gap.
**`arborist canon <key> "<input>"`** — direct π* call, no shards,
no LLM, no audit chain. Pure projection:
$ arborist canon arithmetic@v1 "0.1 + 0.2" → 3/10
$ arborist canon logic-kernel@v1 "A IMPL B" → (NOT A OR B)
$ arborist canon --list → registry contents
$ arborist canon --json arithmetic@v1 "0.1+0.2" → SHA-256 envelope
**Math/logic preflight in `arborist query`** — pure-arithmetic and
pure-propositional questions short-circuit RAG and answer through
arithmetic@v1 / logic-kernel@v1 directly. Synthetic
`audit_mode=CANONICAL_PROJECTION`, renders as
`CANONICAL · via <pi_star_ref>`:
$ arborist query "0.1 + 0.2"
0.1 + 0.2
CANONICAL · via arithmetic@v1 0.0s (projected)
3/10
$ arborist query "(NOT B) IMPL (NOT A)"
(NOT B) IMPL (NOT A)
CANONICAL · via logic-kernel@v1 0.0s (projected)
(NOT A OR B)
Sniff is conservative: pure-arithmetic shape (digits + ops, no
letters) or pure-propositional shape (uppercase atoms + reserved
keywords only). Natural-language wrapping ("what is 0.1+0.2?")
falls through to RAG. PiStarError on a shape match also falls
through — preflight is best-effort, never blocking.
Disable per-call: `--no-canonical-preflight` flag,
`policy["canonical_projection_preflight"]=False`.
No schema changes: CANONICAL_PROJECTION is a render-layer audit_mode
token. No providence_cache writes, no audit_events, no
governance_policy_hash bump. The canonical bytes ARE the answer;
SHA-256 of the bytes is the equivalence-class identity (already
committed via the π* registry).
Side housekeeping: arborist/pi_star/__init__.py docstring caught up
with reality — six concrete π*'s ship today, only tabular-pinned@v1
remains as a stub.
31 new tests (preflight sniff + dispatch, query short-circuit,
contrapositive equivalence-class collapse, CLI subcommand exit codes
and JSON envelope, --no-canonical-preflight policy gate). Full
suite: 1300 passed, 36 skipped.
Completes the bridge: every 5F sub-battery now has an embedded
(Phase 1a) AND a live (Phase 1b.2) path. Per-task detail.source
distinguishes synthetic from live signal.
| Sub-battery | Live surface | Live fixtures |
|---|---|---|
| Formulate | qa.parse_claims.parse_pointer_claims | 15 (prior) |
| Feedback Loop | store.append_audit + memory.snapshot (temp shard) | 12 (prior) |
| Function | parse_pointer_claims + shape evaluators | 12 |
| Finetuning | selfmodel.store_snapshot + claims_for round-trip | 10 |
| Falsification | qa.verify.verify_quotes (real verifier) | 12 |
Function live: input_text routes through real parse_pointer_claims;
the same Phase-1a shape evaluators (shape_match / pointer_set_match
/ threshold_on_metric) run on the live-parsed output. Tests
parser→shape pipeline against actual organism behavior.
Finetuning live: gated via fixture's "live": true flag. Runner
writes parent + child SelfModel + capability_claim to a fresh temp
shard via real arborist.selfmodel.store_snapshot, reads back via
claims_for, runs the same improvement check. Tests the SelfModel
persistence + claim-attach surface, not just static fixture data.
Falsification live: answer_text + context routes through real
arborist.qa.verify.verify_quotes. Live signals translate the
verifier's audit_mode + verifier_method + unverified_quotes into
flat tags (UNGROUNDED, STRICT_<method>, HYBRID_<method>,
UNVERIFIED_QUOTE) the runner matches against expected_reason. One
fixture (5f-fal-live-003) deliberately documents a known soft-
signal gap — the entity strategy treats Insulin/Penicillin claims
as equivalent because both share "Alexander Fleming". Production
catches it via title-relevance + claim-lattice verifier; the
fixture pins the soft-path limit so future verifier changes
re-trigger review.
Surface:
- bench/batteries/b_5f.py — _live_function_produced,
_live_finetuning_measure, _live_falsification_violations
helpers + two-mode dispatch in run_function / run_finetuning /
run_falsification.
- bench/fixtures/5f/{function,finetuning,falsification}-live-v1.jsonl
(12 + 10 + 12 fixtures).
- Makefile: bench-5f-{function,finetuning,falsification}-live
targets; bench-5f-live aggregate now covers all five sub-batteries.
- Tests: 9 new in tests/test_bench_batteries.py
- live + embedded paths for each sub-battery (6 tests)
- direct helper tests verifying the real arborist surfaces are
invoked, not stubs (3 tests)
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at synthetic Phase-1a fixtures so `runner --all` behavior is
identical and deterministic; live mode invoked via explicit
--fixtures path or make targets.
Full suite: 1210 passed, 36 skipped.
5F battery is now the first to bridge synthetic→live across every
sub-battery. The pattern + live fixture format are reusable for
5R Phase 1b.2 (when SelfModel-backed workspace ops want live
verification).
First sub-battery to bridge from synthetic gold output to actual
organism behavior. run_formulate now supports two fixture modes
selected per-task:
- Embedded (Phase 1a): produced_lattice in the fixture. 10 seed
fixtures continue to pass via this path.
- Live (Phase 1b.2): only input_text in the fixture; runner calls
arborist.qa.parse_claims.parse_pointer_claims(input_text) and
matches the live output against expected_lattice.
Embedded takes precedence if both fields are present. Per-task
detail.source ("embedded" | "live") surfaces in bench output so
synthetic vs live signal is distinguishable.
Surface:
- bench/batteries/b_5f.py — _live_produced_lattice helper +
two-mode dispatch in run_formulate
- bench/fixtures/5f/formulate-live-v1.jsonl — 15 live-mode fixtures
with input_text + expected_lattice (no produced_lattice)
- Makefile: bench-5f-formulate-live target
- Tests: 4 new in tests/test_bench_batteries.py
- live path routes through real parser, all 15 pass
- embedded path still works (10 Phase-1a fixtures)
- _live_produced_lattice helper directly verifies parse output
- fixture missing both fields fails cleanly with explanatory reason
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at formulate-v1.jsonl so `runner --all` behavior is identical;
live-mode fixtures invoked via explicit --fixtures path.
Full suite: 1196 passed, 36 skipped.
Pattern set. Function/Finetuning/Falsification/Feedback Loop
follow in subsequent commits.
Phase 2 of #000021. React/Rearrange/Restore/Replicate/Resonate over
the workspace surface — selfmodel_records (#000014) + memory_records
(#000017), both landed earlier today. Closes the gap that gated 5R
since the substrate work shipped.
Sub-battery semantics (per SQD whitepaper §9.3 + ticket #000021 §4.2):
- React: incorporate new fact/constraint. Workspace = (snapshot_t0,
snapshot_t1, expected_delta). Pass = added_facts present + removed_facts
absent in t+1.
- Rearrange: restructure without semantic shift. Re-canonicalize
different surface forms through a named π*; pass = bytes match
expected_equivalent flag. Tests the order-invariance contracts in
SelfModel (capability_claim_hashes sorted) and Memory (branches
sorted by branch_id).
- Restore: retrieve prior fact. Workspace = (history[], current_facts[]).
Pass = fact in current OR any historical snapshot.
- Replicate: independent canonical encodings via π*. Same input run
N times must yield byte-equal output. Tests determinism contract.
- Resonate: variance across N runs. Deterministic π*'s yield
distinct=1; expected_max_distinct=1 enforces zero-variance contract.
Surface:
- bench/batteries/b_5r.py (5 deterministic runners; no LLM-as-judge)
- bench/fixtures/5r/{react,rearrange,restore,replicate,resonate}-v1.jsonl
(30 each = 150 new fixtures)
- runner.py registers 5r in _BATTERIES + _DEFAULT_FIXTURES
- Makefile: bench-5r + bench-suite (5S+5T+5F+5R aggregate)
Final tally:
5S syntax/semantics/syllogism/synthesis/semiotics 108
5T transfer/transfer-learning/triangulation/... 154
5F function/finetuning/falsification/... 50
5R react/rearrange/restore/replicate/resonate 150
TOTAL: 462 fixtures across 21 sub-batteries — 100% pass.
Tests: 6 new in tests/test_bench_batteries.py + adjustment to
test_session_integration.py for the 312→462 count + 5R sub-battery
presence assertion. Full suite: 1192 passed, 36 skipped.
Closes#000021. Phase 3 (external-corpus expansion) remains open
under the ticket but does not gate closure — the complete
Dav1DPrometheus surface is now executable infrastructure.
The top-level module diagram still used the old aborist- (one R) prefix
from before the package rename. arborist.unturf.com served from the
symlinked docs/_source/diagrams/, so the live URL
arborist.unturf.com/en/diagrams/arborist-modules.svg 404'd because the
file on disk was aborist-modules.svg.
Other diagram base names (ingest-pipeline, mesh-*, query-pipeline,
verifier-ladder) were never aborist-prefixed; only this one needed it.
Per fox's 2026-05-08 review of fbd99a8: implement adaptation_efficiency
and feedback_efficiency in run_finetuning + run_feedback_loop with
explicit sentinels — not Python floating-point accidents.
_efficiency(gain, cost) helper:
- cost > 0: standard ratio
- cost == 0, gain > 0: EFFICIENCY_INFINITE (free improvement)
- cost == 0, gain == 0: EFFICIENCY_UNDEFINED (= 0.0; no signal)
- cost == 0, gain < 0: -EFFICIENCY_INFINITE (free regression)
Battery-level metrics report mean_finite (computed over finite
values only) + infinite_count + neg_infinite_count so the mean
stays dimensionally truthful and consumers can pivot on the special
buckets separately.
Phase 1a cost proxies:
- run_finetuning: _capital_cost_delta sums resource_budget
(max_compute_ms_delta * 1e-3 + max_storage_delta_bytes / 1e6).
Phase 1b.2 will replace with real capital_ledger reads.
- run_feedback_loop: chain length = cost. Phase 1b.2 capital_ledger
integration replaces it.
Tests added (7):
- _efficiency over four boundary cases
- 5F finetuning + feedback_loop emit the new metrics keys
- Synthetic zero-cost finetuning fixture verifies +inf path
Full suite: 1110 passed, 36 skipped.
Closes the one actionable from the fbd99a8 review.
Captures design for an action-provenance DAG layer downstream of
final_label - five new stages (action_plan -> tool_call ->
tool_output -> postcondition_check -> action_label) - and why this
stays a research doc rather than an open ticket today.
Three options analyzed: Option A in-tree action DAG (identity
drift), Option B sidecar package (recommended; preserves the
verified-answer-cache identity by chaining a separate action_root
that cross-links into run_dag_root), Option C out-of-scope.
Promotion criteria spelled out so the doc graduates to a ticket
when the first agent use case shows up. Cross-links #000001
(upstream provenance gap), #000022 (LossReport, same axiom one
stage upstream), #000012 (v8 selection could later score action
histories), and the 2026-05-07 arborist-vs-donto comparison.
TICKETS.md gains a pointer in "Distinction from other docs" so
future shifts find the doc.
Per the three review responses (~/Downloads/RESPONSE_*) folded in
2026-05-08, the 5S / 5T / 5F tickets are corrected from "text-only
with future hooks" to "carrier-aware design from day one." Phase 1
implementation stays text / claim-lattice / memory-root only, but the
fixture schema MUST accommodate future visual / world / code / audio /
sensor / hidden-channel-detection carriers without re-authoring.
Common corrections across all three tickets:
- Mandatory fixture metadata: carrier, domain, pi_star_ref,
loss_report_refs, modality_notes.
- Unsupported carriers MUST fail or skip explicitly with
reason="unsupported_carrier" — never silently accepted.
- No LLM-as-judge in any runner.
- Hidden-channel work is defensive only (detection / flagging),
never generation or concealment.
Per-ticket headlines:
#000023 — 5S
Syntax / Semantics / Semiotics defined as carrier-general operations
over sign-bearing representations. Semiotics gets the biggest
correction: visual symbols, layout, metadata, encoded sign systems
are valid carriers (Phase 1 still text-only). Synonym source
policy: concept_relations.relation_kind='synonym' only for v1
positives.
#000024 — 5T
Vocabulary alignment with Dav1DPrometheus authoritative wording
(Transfer→Transfer Learning, Truth→Truthtables, Timing→Time).
Transitivity gets a typed-relation whitelist (implies, subset_of,
ancestor_of, before, less_than) — not all edges transitive.
Truthtables capped at N=2..4 to avoid combinatorial blowup. Time
is the first sub-battery where v8 substrate (memory_root #000017
+ selfmodel #000014) becomes a measurable bench target.
#000025 — 5F
New axis. Function/Finetuning/Falsification/Formulate/Feedback
Loop. Folds in the state-space synthesis: SQD + v7 + 5S/5T/5F +
arborist together instantiate a discrete state-space/time
Ω_t = (W, I, C, L, MRoot, SMRoot, PRoot, BRoot, ARoot) with
Ω_{t+1} = T(Ω_t, Δ_t). Counting / mathematics / logic / time
emerge as auditable operations over committed state, not text from
a latent model. adaptation_efficiency and feedback_efficiency
metrics hook into the capital ledger (#000020) so v8 fork choice
has cost-aware fitness signals. Falsification fixtures tagged
with verifier_method_root so verifier shape changes warn rather
than false-fail.
All three tickets remain "open · awaiting go/no-go" — design-only.
Implementation tickets land in follow-up commits when fox approves
the corrected scope.
Source: Legally Unprecedented Dav1DPrometheus (BasementAGI host,
Where The mAGIc Happens). Honoring his framework.
Three new design-only tickets surfacing the gaps between arborist's
current bench harness (ticket #000021 Phase 1a, landed) and
Dav1DPrometheus's authoritative 5S/5F/5T evaluation framework.
- #000023 — 5S Phase 1b: real implementations + fixtures for
Syllogism, Synthesis, Semiotics (currently stubbed).
- #000024 — 5T Phase 1b: rename Transfer→Transfer Learning,
Truth→Truthtables, Timing→Time to honor Dav1DPrometheus's
vocabulary; ship real Triangulation, Truthtables, Transitivity,
Time runners (currently stubbed). Time integrates with
memory_root (#000017) for the first measurable use of v8
substrate as fitness target.
- #000025 — 5F battery: entirely new — Function, Finetuning,
Falsification, Formulate, Feedback Loop. arborist had no 5F
coverage before this ticket; the SQD whitepaper omitted the
axis. Each sub-battery integrates with surfaces already shipped
(selfmodel_records, providence_cache.falsification_state,
memory_branch_summaries).
Source attribution: Legally Unprecedented Dav1DPrometheus
(BasementAGI host). Honoring his framework as the authoritative
taxonomy for non-embodied AGI evaluation.
Next ID bumped 000023 → 000026.
Per fox's "partial punt on larger ones" — ships the bench/ skeleton +
small seed fixture sets so future v8/v7-W/SelfModel work can cite a
real fitness target. Full Phase 1 (50-200 fixtures per sub-battery)
and Phases 2-3 stay open in the ticket.
Phase 1a delivers:
- bench/batteries/{base,b_5s,b_5t,runner}.py — Battery protocol,
BatteryResult, fixture-digest helpers, CLI runner.
- Seed fixtures:
- bench/fixtures/5s/syntax-v1.jsonl — 10 tasks against
wikitext-base@v1 and claim-lattice@v1
- bench/fixtures/5s/semantics-v1.jsonl — 8 equivalence tasks
- bench/fixtures/5t/transfer-v1.jsonl — 4 paraphrase-invariance
tasks
- Runners for 5S Syntax, 5S Semantics, 5T Transfer. Other 5S/5T
sub-batteries are stubs returning zero-task results.
- Makefile targets: bench-5s, bench-5t, bench-5s5t.
- runtime_digest field captures the active π* registry fingerprint
so a registry change surfaces in bench results.
Tests: tests/test_bench_batteries.py (17 cases). Full suite:
1076 passed, 36 skipped. `make bench-5s5t` runs end-to-end and
emits JSON results.
Ticket #000021 status: in progress · Phase 1a landed; Phase 1b/2/3
remain open.
Doc-only landing. docs/spec-methodology.md codifies the discipline
arborist already practices — versioning rule, round-trip discipline,
soundness/completeness honesty, default-value greenfield rule,
sidecar separation — so new π*, V, and policy-field authors don't
re-derive it from audit-chain failures.
Three author-class sections each ship with:
- Five questions the author must answer before landing.
- Worked example drawn from arborist's existing surface.
- One-page checklist.
Worked examples cited:
- π* — wikitext-base@v1
- V — paraphrase strategy
- policy field — quantifier_guard_apply_caps
Cross-references to bench-maxing, seven-point-program, pi-star-
composition, concept-relations-design, and CLAUDE.md.
New arborist.pi_star/ namespace centralizes canonical projections
under a name@version registry. Two existing canonicalizers re-homed
as registered π*'s:
- wikitext-base@v1 wraps arborist.wikitext.to_base
- claim-lattice@v1 wraps arborist.qa.parse_claims.parse_pointer_claims
Four stubs registered for follow-up modality tickets:
code-py-ast@v1, logic-kernel@v1, time-series-quantized@v1,
tabular-pinned@v1 — each raises NotImplementedError with a pointer
to ticket #000015.
Composition algebra in compose.py: PiStarComposition exposes
outer ∘ inner as a first-class π* with its own registry key
(default "<inner-name>-then-<outer-name>@v1"). canonical_composition_id
returns a SHA-256 fingerprint suitable for governance hash inclusion.
Order-sensitive: a∘b ≠ b∘a → different fingerprints.
Documentation: docs/pi-star-composition.md covers the rule (type-
compatible, deterministic, equivalence-class preserving), lossy vs
invertible compositions, worked text→claim-lattice example,
cross-domain anchor projections (future), authoring checklist.
Re-home is non-breaking: arborist.wikitext.to_base remains importable.
Tests: tests/test_pi_star.py (19 cases). Full suite: 1059 passed,
36 skipped.
CapitalProfile (8 forms: living, material, financial, intellectual,
experiential, social, cultural, spiritual) attached per state-changing
op as a sibling-table row in capital_ledger. Sibling semantics: ledger
rows reference an audit_event_hash but do NOT enter the audit-event
preimage, so retroactive cost re-estimation cannot break the chain.
Surface:
- arborist.capital.{profile,store}
- profile_for_op dispatch with per-op estimators (ingest/qa/distill)
- record/summary/op_cost/top_by_form
- CLI: arborist capital summary|op-cost|top
Wire-up at three op sites:
- ingest.py — one row per batch (doc_count + total_bytes)
- qa/runner.py — one row per cache-miss (answer_chars + llm_seconds)
- distill/runner.py — one row per derivation (positive intellectual)
Estimator constants are heuristic v1 (ESTIMATOR_VERSION pin in the
schema). Re-estimation is supported by re-running estimators against
the recorded inputs_blob and writing a new row with a bumped version
pin; old rows stay queryable.
Tests: tests/test_capital.py (13 cases). Sibling-table invariant
verified: audit chain stays intact across capital writes.
Full suite: 1025 passed, 36 skipped.
SelfModel binds an arborist agent's identity to bytes a verifier can
recompute: model_profile_hash, verifier_method_root, governance hash,
canonicalization/chunking versions, optional patch + memory roots,
sorted capability-claim hashes. Hard-hash committed; no soft state in
preimage. State transitions live on the row, not the body, so the
selfmodel_root stays stable across live → stale → falsified.
Surface:
- arborist.selfmodel.{canonical,snapshot,store,falsify}
- CLI: arborist selfmodel snapshot|show|falsify|list
- Schema: selfmodel_records + selfmodel_capability_claims (additive)
- Audit events: selfmodel_snapshot_landed,
selfmodel_capability_claim_added, selfmodel_falsified,
selfmodel_marked_stale (all chain via existing append_audit)
Also folds in:
- CLAUDE.md operational rule: arborist stays Python-only; non-Python
toolchains live in sibling repos. Forks/clients/servers in any
language follow our schemas + canonical encodings.
- Ticket #000016 update: ZK lives in sibling repo arborist-zk-bench;
arborist gains at most a wire-format consumer, never a Rust dep.
- Schema migrations also stub capital_ledger and memory_records
tables for tickets #000020 and #000017 respectively (additive,
empty until those modules land).
Tests: tests/test_selfmodel.py (14 cases; canonical-JSON stability,
root order-invariance, snapshot determinism, store idempotency,
audit events, falsify/mark_stale semantics, audit-chain integrity).
Full suite: 1012 passed, 36 skipped.
Both items surfaced by the 2026-05-07 arborist-vs-donto comparison
report (/home/fox/Downloads/arborist_vs_donto.pdf).
Ticket #000022 — adapter LossReport (PRD I9 analogue). Today wikitext
to_base() and html_page _normalize_text drop <ref> tags, image/category
wikilinks, HTML chrome, whitespace runs without recording any of it;
only the canonicalization-version pin survives. Ticket proposes a
sidecar adapter_loss_reports table (Option A) over Merkle-bound
loss_root (B) or audit-chain entries (C), preserving arborist's
"soft signals are sidecars" discipline. ~1.6% storage tax expected,
matching concept_relations. Doc-only; no code in this commit.
Federation doc-discoverability: PDF author concluded "federation
exists in code but the public docs page returned 404" because the
mesh page lives at /api/mesh.html and the concepts orientation
never surfaces the topic. Adds a "Federation across peers" section
to concepts.rst pointing at api/mesh, a :ref:\`federation\` anchor
+ context lead on api/mesh.rst, and a footer link in concepts.rst's
"Where to go next." Sphinx build passes; api/mesh.html#federation
deep link resolves.
Per shared-things convention: prefer 'our' for community-owned things;
'the' implies fixed singular ownership. The permacomputer is collective
infrastructure, so 'our' fits better than 'the'.
Updated three places (single canonical preamble text):
- LICENSE (Permacomputer Preamble section, our text — not the AGPL)
- README.md License section
- docs/_source/conf.py rst_epilog (per-page footer on RTD)
Previous attempt to move the full project TOC to the right via
html_sidebars={'**': []} stripped the theme's left sidebar — the right
sidebar in sphinx-book-theme renders only page-local TOC, not the
project tree, so the project structure disappeared and the page
looked unthemed.
Restore the default layout:
- Left: full project toctree (themed sphinx-book-theme sidebar)
- Right: 'On this page' (current page sections), expanded to depth 3
If we want a unified right-side project TOC later, that needs
pydata-sphinx-theme + secondary_sidebar_items override, not just
hiding the left.
sphinx_rtd_theme only renders left-side navigation. Switch to
sphinx-book-theme which puts the full project TOC on the right and
leaves the reading area centered.
conf.py:
- html_theme = 'sphinx_book_theme'
- html_sidebars = {'**': []} hides the left sidebar so the right TOC
is the single navigation surface
- show_toc_level=3 expands subpages; show_navbar_depth=2 controls top nav
requirements.txt: sphinx-rtd-theme → sphinx-book-theme>=1.1
Makefile cuts (64 → 58 documented targets):
- ingest-cur-parallel, ingest-old-parallel: parallel-shared mode
superseded by attached (no WAL contention)
- distill-shards: sequential never preferred over parallel variant
- bench-qa-quick: bench-qa-smoke covers same use case (~30s vs ~10s)
- ingest-grok, ingest-grok-media: single-DB grok rare; -attached is
canonical path
All cuts land in code that the underlying CLI still exposes — operators
who need the dropped variant call '.venv/bin/aborist ingest --shard ...'
directly. No behavior loss, just shortcut removal.
Docs improvements:
- New Concepts page (docs/_source/concepts.rst): orientation on what
aborist is, three layers (surface/core/providence), Merkle commitment,
8-dim cache key, audit chain, trichotomy + four-rung ladder, layered
verifier, falsification state, sidecars. Embeds module-graph and
verifier-ladder SVG diagrams.
- New Cookbook page (docs/_source/cookbook.rst): 8 recipes — recrawl,
falsify, ingest-self-providence, mixed-corpus query, LLM endpoint
override, integrity after bulk ops, bench, retrieval tuning.
- Quickstart embeds query-pipeline SVG diagram.
- docs/_source/diagrams symlinks to docs/diagrams so Sphinx can include
the SVGs (was orphaned, only README referenced them).
Better Makefile RTD page (docs/_source/_ext/makefile_targets.py):
- Group by workflow phase (Setup → Fetch → Ingest → Distill → Query →
Verify → Operations → Tests → Docs → Clean) instead of alphabetical
prefix. Tells a new operator the order they'd actually run things.
- Phase descriptions added; targets prefixed with 'make ' for copy-paste.
- Uncategorized leftover surfaces missing entries in PHASES list.
New docs/_source/quickstart.rst — install, two end-to-end paths
(Wikipedia 2003 + crawler), after-the-answer commands, query pipeline
overview, and links to deeper reference pages. Mirrors the README's
quickstart but adapts cross-references to the Sphinx structure.
Added to index.rst as a 'Getting started' toctree section above the
API modules — RTD users land on it first.
Use the canonical Read the Docs theme. Build dropped from 33 warnings
to 3 (most furo warnings were sidebar template lookups in the dark/light
mode switcher).
requirements.txt updated to pull sphinx-rtd-theme instead of furo.
- docs/_source/license.rst — new RTD page that literalincludes the
repo's LICENSE file (single source of truth, no duplication)
- conf.py rst_epilog — appended to every RST source so every doc page
carries the Permacomputer Preamble + AGPL-3.0-only notice + link to
the full license
- index.rst — adds 'Project / License' section to the toctree
The LICENSE file already had the full Permacomputer Preamble + complete
GNU Affero GPL v3 text (matches the whitepaper version).
Adds a Sphinx extension at docs/_source/_ext/makefile_targets.py that
parses the project Makefile's '## description' annotations and writes
docs/_source/api/makefile.rst at build time. Same convention 'make help'
uses, so the reference stays in sync with the source.
Generated page is grouped by target prefix (fetch-, ingest-, distill-,
docs-, etc.) and rendered as a list-table. Shows on RTD alongside the
autodoc API modules.
Generated file is gitignored — RTD regenerates on every build.
Initial docstring pass focused on syntax/style; this pass verified each
docstring against actual function behavior. Found and corrected:
WRONG (claimed behavior didn't match):
- _cmd_verify: claimed Q&A/audit verification — actually round-trips
Merkle proofs on N random documents
- _cmd_snapshot_verify: claimed Merkle proof round-trip — actually
re-derives snapshot root and checks for drift
- _cmd_evict: claimed 'archive unused content' — actually NULLs content
and removes FTS row; cores never evict
- _cmd_stats: listed 'index size' which is not in stats() output
OVERSTATEMENT (claim stronger than contract):
- _cmd_ask: 'grounded answer' — verifier may return UNGROUNDED
- _cmd_snapshot_list: 'named' — snapshots have hash roots, not names
MISSING IMPORTANT BEHAVIOR:
- _cmd_rehydrate: didn't mention drift-detection exit code
- _cmd_mesh_status: didn't mention 'enabled' flag (most important field)
- _cmd_distill: didn't mention recursive core→core distillation
- MerkleTree.proof(): didn't mention IndexError on out-of-range
VAGUE:
- _cmd_search: 'Search the corpus with FTS5' → mention output formats
Sphinx rebuild successful (29 warnings, down from 31).
Implements Read the Docs infrastructure to generate API documentation
directly from code docstrings. Replaces static modules.md (1200+ lines).
New structure:
- docs/_source/conf.py — Sphinx configuration (furo theme)
- docs/_source/index.rst — Main TOC
- docs/_source/api/*.rst — Module groups (substrate, storage, retrieval,
qa, distill, mesh, cli)
- docs/_source/Makefile — Local build targets
- docs/_source/README.md — Documentation on building and extending
Makefile integration:
- make docs-api — generate HTML (output: docs/_source/_build/html/)
- make docs-api-clean — remove build artifacts
Build output (40 HTML files):
- API module reference with docstrings
- Source code links (:viewcode: extension)
- Full-text search index
- Module index (genindex, py-modindex)
Sphinx installed in venv as dev dependency. HTML is browseable at
docs/_source/_build/html/index.html (open in browser after build).
This justifies the deletion of modules.md: code docstrings + Sphinx
autodoc = automatically-generated, always-current API reference.
Delete redundant, superseded, or design-log docs:
- qa-modes-bench-2026-04-30: prior snapshot (rolling journal is current)
- verifier-semantic-gap-design: unimplemented future proposal
- bench-emergent-design: stress-test design (script self-documents)
- modules.md: API reference (code + docstrings are source of truth)
- self-reference-design: v1 shipped, v2 scoped to future; history in git
- concept-relations-design: live system documented in code
- mesh-deploy: runbook for off-by-default system; mesh wire future work
Reduces docs/ from 23 files to 16, keeping north-star (seven-point-program),
architecture (cti, mesh.md), and operational docs (benchmarks, qa-modes-bench,
bench-maxing). All deleted docs recoverable from git history.
Single canonical entry point that ties together the four existing
bench-related docs (qa-modes-bench.md / bench-maxing.md /
bench-emergent-design.md / qa-modes-bench-2026-04-30.md) plus the
make targets, fixtures, and bench-row schema.
Sections:
1. Two harnesses, two purposes
bench/qa_sweep.py — curated regression bench
scripts/bench_emergent.py — random-word stress test
2. Four question fixtures (75 / 28 / 9 / 1 / smoke) with cell
sizes + use cases per fixture
3. Signal floor (5pp / n=3 × 9 = 27 / vLLM c=3-4 saturation)
4. Make targets cheat sheet (bench-qa, bench-emergent,
--policy KEY=VALUE A/B pattern, --resume)
5. Bench-row schema — every field a row carries
(identity / verdict / diagnostics / preflight projection /
quantifier classifier / capacity / directive compliance / time)
6. Where headlines live (qa-modes-bench.md addenda, per-ticket
§12/§13 bench sections)
7. bench-emergent log shape + #000006 rolling-amend pattern
8. How to run a focused A/B (the four-cell pattern from #000008)
9. How to interpret results (STRICT-rate, mean-ratio,
UNGROUNDED-rate, FORMAT_COLLAPSED rate, violation kind
distribution, audit-line tails)
10. Operator commands cheat sheet (--show-preflight,
--apply-quantifier-caps, --reject-broad, --soft-preflight,
Makefile shortcuts)
CLAUDE.md docs index updated to point to benchmarks.md as the
"read first" entry for bench work, and to add bench-emergent-design.md
to the index (was missing).
Bench-emergent stress test ran another 100 cycles under the
post-#000008/9/10/11 substrate. Total accumulated: 300 cycles.
Verdict distribution shift on last 100 vs 134-cycle baseline:
STRICT 5% (7/134) → 0% (0/100) -5pp
HYBRID 22% (29/134) → 16% (16/100) -6pp
UNGROUNDED 73% (98/134) → 84% (84/100) +11pp
Zero false-positive STRICTs across 100 random-word triplets.
The 5pp drop in STRICT-rate isn't a regression — it's the
verifier ladder + new preflight contracts doing their job.
Random-word triplets are genuinely ungrounded for the most
part; the prior 5% STRICT rate included false-positives that
the post-hardening verifier now catches.
Violation profile (last 100 cycles, claim_lattice JSON):
CITATION_MISMATCH: 86 dominant gate
TOO_MANY_EVIDENCE_IDS: 24
SUBJECT_TOKENS_ABSENT: 12 Rule 9 firing on parroting
DEFLECTION_DETECTED: 12
TITLE_MISMATCH: 10
...
metaphor_deflection fires 6/100 — still rare. Item 3
(calibration) is now closer to sample-size threshold (~30
signals across 300 cycles; needs ~50-100 to calibrate).
No new tuning candidates surface. Original three remain at
their resolution states.
Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.
#000025 — Metacog test fixture expansion:
bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
covering edge cases per detector kind: temporal (4 cases),
contradiction (4), false-premise (5), out-of-corpus (3), multi-
trigger (2), well-formed controls (5). Documents two known
detector ceilings: Q11 over-fires on past-tense factoid
("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
NASA-fake) miss false premises that lack a presupposition
pattern match. Fixture now serves as long-term regression suite.
#000026 — --show-preflight full clause render:
build_run_dag() and build_reject_run_dag() gain optional
preflight_payload kwarg. When supplied, the canonical 5-clause
CTI payload (classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs + question_state + node_version)
persists alongside the leaf hash in run_dag_blob.
aborist providence --show-preflight CACHE_KEY now renders the
full payload + verifies the persisted hash matches the
recomputed canonical hash (audit-replay tamper detection).
Legacy rows fall through cleanly: payload_hash_check reports
"unavailable: legacy row predates preflight_payload persistence".
#000027 — Latency profile:
Microbenched preflight: 0.46ms/question (negligible). Single
fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
33-35s in Addendum 3 was vLLM concurrency contention at c=4
(per qa-modes-bench.md saturation note), not substrate
overhead. Added preflight_ms + soft_preflight_ms to timings
dict for explicit confirmation in future cycles.
#000028 — Auto-quality-check sweep revival:
scripts/bench_emergent.py running with EMERGENT_N=100 in
background (PID 125680). Will accumulate cycles into
bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
Async — not blocking on completion.
#000029 — #000011 SOFT_PREFLIGHT_HINT implementation:
aborist/qa/soft_preflight.py — new module. SoftPreflightHint
dataclass + soft_preflight_question() pure function. 9
canonical labels mapping to soft analogues of #000010 hard
detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
Constrained-generation prompt (max_tokens=128, temp=0.0) asks
the model to pick ONE label + one-line rationale.
Fail-closed across every parse path:
- chat_client raises → SOFT_PARSE_FAIL
- response unparseable → SOFT_PARSE_FAIL
- label outside enum → SOFT_PARSE_FAIL
Sidecar enforces SOFT_ prefix at the normalize step so a
model that drops the prefix still gets caught.
Wired into query() between preflight & retrieval. Default
OFF (`soft_preflight_enabled: False`). NOT folded into
_VERIFIER_POLICY_FIELDS — soft hints don't gate cache
identity (#000011 §4). Audit-line tail renders as
"· soft: <label>" (e.g. "· soft: time sensitive") so the
signal is visually distinct from hard tails.
--soft-preflight CLI flag opts in per-call. End-to-end
live-verified on "When did Mr. Burns become Homer's biological
father?" — produces:
EVIDENCE-WARRANTED · via claim_lattice
· false premise · soft: time sensitive
1/1 16.4s
Hard `· false premise` (from #000010 deterministic detector)
composed with soft `· soft: time sensitive` (from #000011
sidecar). The model classified a different shape than the hard
detector — by design; soft hints are independent advisory
signals, not redundant with the hard layer.
25 new tests pin: default-OFF behavior, parse-failure modes,
label normalization (SOFT_ prefix enforced), all 8 actionable
labels round-trip, fail-closed on client exceptions, dataclass
JSON round-trip, rationale-length cap.
Other:
- #000010 §13.3 documents 2/5 metacog-trigger questions return
STRICT despite hard-detector warning — direct empirical
motivation for #000011 design.
- tests/test_dag.py extends with 3 _extract_preflight_hash_*
helper tests (cleaning #000009 §7.2 unfinished state).
- bench/emergent_log.jsonl adds new cycles from background run.
#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
