Three new design-only tickets surfacing the gaps between arborist's
current bench harness (ticket #000021 Phase 1a, landed) and
Dav1DPrometheus's authoritative 5S/5F/5T evaluation framework.
- #000023 — 5S Phase 1b: real implementations + fixtures for
Syllogism, Synthesis, Semiotics (currently stubbed).
- #000024 — 5T Phase 1b: rename Transfer→Transfer Learning,
Truth→Truthtables, Timing→Time to honor Dav1DPrometheus's
vocabulary; ship real Triangulation, Truthtables, Transitivity,
Time runners (currently stubbed). Time integrates with
memory_root (#000017) for the first measurable use of v8
substrate as fitness target.
- #000025 — 5F battery: entirely new — Function, Finetuning,
Falsification, Formulate, Feedback Loop. arborist had no 5F
coverage before this ticket; the SQD whitepaper omitted the
axis. Each sub-battery integrates with surfaces already shipped
(selfmodel_records, providence_cache.falsification_state,
memory_branch_summaries).
Source attribution: Legally Unprecedented Dav1DPrometheus
(BasementAGI host). Honoring his framework as the authoritative
taxonomy for non-embodied AGI evaluation.
Next ID bumped 000023 → 000026.
Per fox's "partial punt on larger ones" — ships the bench/ skeleton +
small seed fixture sets so future v8/v7-W/SelfModel work can cite a
real fitness target. Full Phase 1 (50-200 fixtures per sub-battery)
and Phases 2-3 stay open in the ticket.
Phase 1a delivers:
- bench/batteries/{base,b_5s,b_5t,runner}.py — Battery protocol,
BatteryResult, fixture-digest helpers, CLI runner.
- Seed fixtures:
- bench/fixtures/5s/syntax-v1.jsonl — 10 tasks against
wikitext-base@v1 and claim-lattice@v1
- bench/fixtures/5s/semantics-v1.jsonl — 8 equivalence tasks
- bench/fixtures/5t/transfer-v1.jsonl — 4 paraphrase-invariance
tasks
- Runners for 5S Syntax, 5S Semantics, 5T Transfer. Other 5S/5T
sub-batteries are stubs returning zero-task results.
- Makefile targets: bench-5s, bench-5t, bench-5s5t.
- runtime_digest field captures the active π* registry fingerprint
so a registry change surfaces in bench results.
Tests: tests/test_bench_batteries.py (17 cases). Full suite:
1076 passed, 36 skipped. `make bench-5s5t` runs end-to-end and
emits JSON results.
Ticket #000021 status: in progress · Phase 1a landed; Phase 1b/2/3
remain open.
Doc-only landing. docs/spec-methodology.md codifies the discipline
arborist already practices — versioning rule, round-trip discipline,
soundness/completeness honesty, default-value greenfield rule,
sidecar separation — so new π*, V, and policy-field authors don't
re-derive it from audit-chain failures.
Three author-class sections each ship with:
- Five questions the author must answer before landing.
- Worked example drawn from arborist's existing surface.
- One-page checklist.
Worked examples cited:
- π* — wikitext-base@v1
- V — paraphrase strategy
- policy field — quantifier_guard_apply_caps
Cross-references to bench-maxing, seven-point-program, pi-star-
composition, concept-relations-design, and CLAUDE.md.
New arborist.pi_star/ namespace centralizes canonical projections
under a name@version registry. Two existing canonicalizers re-homed
as registered π*'s:
- wikitext-base@v1 wraps arborist.wikitext.to_base
- claim-lattice@v1 wraps arborist.qa.parse_claims.parse_pointer_claims
Four stubs registered for follow-up modality tickets:
code-py-ast@v1, logic-kernel@v1, time-series-quantized@v1,
tabular-pinned@v1 — each raises NotImplementedError with a pointer
to ticket #000015.
Composition algebra in compose.py: PiStarComposition exposes
outer ∘ inner as a first-class π* with its own registry key
(default "<inner-name>-then-<outer-name>@v1"). canonical_composition_id
returns a SHA-256 fingerprint suitable for governance hash inclusion.
Order-sensitive: a∘b ≠ b∘a → different fingerprints.
Documentation: docs/pi-star-composition.md covers the rule (type-
compatible, deterministic, equivalence-class preserving), lossy vs
invertible compositions, worked text→claim-lattice example,
cross-domain anchor projections (future), authoring checklist.
Re-home is non-breaking: arborist.wikitext.to_base remains importable.
Tests: tests/test_pi_star.py (19 cases). Full suite: 1059 passed,
36 skipped.
CapitalProfile (8 forms: living, material, financial, intellectual,
experiential, social, cultural, spiritual) attached per state-changing
op as a sibling-table row in capital_ledger. Sibling semantics: ledger
rows reference an audit_event_hash but do NOT enter the audit-event
preimage, so retroactive cost re-estimation cannot break the chain.
Surface:
- arborist.capital.{profile,store}
- profile_for_op dispatch with per-op estimators (ingest/qa/distill)
- record/summary/op_cost/top_by_form
- CLI: arborist capital summary|op-cost|top
Wire-up at three op sites:
- ingest.py — one row per batch (doc_count + total_bytes)
- qa/runner.py — one row per cache-miss (answer_chars + llm_seconds)
- distill/runner.py — one row per derivation (positive intellectual)
Estimator constants are heuristic v1 (ESTIMATOR_VERSION pin in the
schema). Re-estimation is supported by re-running estimators against
the recorded inputs_blob and writing a new row with a bumped version
pin; old rows stay queryable.
Tests: tests/test_capital.py (13 cases). Sibling-table invariant
verified: audit chain stays intact across capital writes.
Full suite: 1025 passed, 36 skipped.
SelfModel binds an arborist agent's identity to bytes a verifier can
recompute: model_profile_hash, verifier_method_root, governance hash,
canonicalization/chunking versions, optional patch + memory roots,
sorted capability-claim hashes. Hard-hash committed; no soft state in
preimage. State transitions live on the row, not the body, so the
selfmodel_root stays stable across live → stale → falsified.
Surface:
- arborist.selfmodel.{canonical,snapshot,store,falsify}
- CLI: arborist selfmodel snapshot|show|falsify|list
- Schema: selfmodel_records + selfmodel_capability_claims (additive)
- Audit events: selfmodel_snapshot_landed,
selfmodel_capability_claim_added, selfmodel_falsified,
selfmodel_marked_stale (all chain via existing append_audit)
Also folds in:
- CLAUDE.md operational rule: arborist stays Python-only; non-Python
toolchains live in sibling repos. Forks/clients/servers in any
language follow our schemas + canonical encodings.
- Ticket #000016 update: ZK lives in sibling repo arborist-zk-bench;
arborist gains at most a wire-format consumer, never a Rust dep.
- Schema migrations also stub capital_ledger and memory_records
tables for tickets #000020 and #000017 respectively (additive,
empty until those modules land).
Tests: tests/test_selfmodel.py (14 cases; canonical-JSON stability,
root order-invariance, snapshot determinism, store idempotency,
audit events, falsify/mark_stale semantics, audit-chain integrity).
Full suite: 1012 passed, 36 skipped.
Both items surfaced by the 2026-05-07 arborist-vs-donto comparison
report (/home/fox/Downloads/arborist_vs_donto.pdf).
Ticket #000022 — adapter LossReport (PRD I9 analogue). Today wikitext
to_base() and html_page _normalize_text drop <ref> tags, image/category
wikilinks, HTML chrome, whitespace runs without recording any of it;
only the canonicalization-version pin survives. Ticket proposes a
sidecar adapter_loss_reports table (Option A) over Merkle-bound
loss_root (B) or audit-chain entries (C), preserving arborist's
"soft signals are sidecars" discipline. ~1.6% storage tax expected,
matching concept_relations. Doc-only; no code in this commit.
Federation doc-discoverability: PDF author concluded "federation
exists in code but the public docs page returned 404" because the
mesh page lives at /api/mesh.html and the concepts orientation
never surfaces the topic. Adds a "Federation across peers" section
to concepts.rst pointing at api/mesh, a :ref:\`federation\` anchor
+ context lead on api/mesh.rst, and a footer link in concepts.rst's
"Where to go next." Sphinx build passes; api/mesh.html#federation
deep link resolves.
Per shared-things convention: prefer 'our' for community-owned things;
'the' implies fixed singular ownership. The permacomputer is collective
infrastructure, so 'our' fits better than 'the'.
Updated three places (single canonical preamble text):
- LICENSE (Permacomputer Preamble section, our text — not the AGPL)
- README.md License section
- docs/_source/conf.py rst_epilog (per-page footer on RTD)
Previous attempt to move the full project TOC to the right via
html_sidebars={'**': []} stripped the theme's left sidebar — the right
sidebar in sphinx-book-theme renders only page-local TOC, not the
project tree, so the project structure disappeared and the page
looked unthemed.
Restore the default layout:
- Left: full project toctree (themed sphinx-book-theme sidebar)
- Right: 'On this page' (current page sections), expanded to depth 3
If we want a unified right-side project TOC later, that needs
pydata-sphinx-theme + secondary_sidebar_items override, not just
hiding the left.
sphinx_rtd_theme only renders left-side navigation. Switch to
sphinx-book-theme which puts the full project TOC on the right and
leaves the reading area centered.
conf.py:
- html_theme = 'sphinx_book_theme'
- html_sidebars = {'**': []} hides the left sidebar so the right TOC
is the single navigation surface
- show_toc_level=3 expands subpages; show_navbar_depth=2 controls top nav
requirements.txt: sphinx-rtd-theme → sphinx-book-theme>=1.1
Makefile cuts (64 → 58 documented targets):
- ingest-cur-parallel, ingest-old-parallel: parallel-shared mode
superseded by attached (no WAL contention)
- distill-shards: sequential never preferred over parallel variant
- bench-qa-quick: bench-qa-smoke covers same use case (~30s vs ~10s)
- ingest-grok, ingest-grok-media: single-DB grok rare; -attached is
canonical path
All cuts land in code that the underlying CLI still exposes — operators
who need the dropped variant call '.venv/bin/aborist ingest --shard ...'
directly. No behavior loss, just shortcut removal.
Docs improvements:
- New Concepts page (docs/_source/concepts.rst): orientation on what
aborist is, three layers (surface/core/providence), Merkle commitment,
8-dim cache key, audit chain, trichotomy + four-rung ladder, layered
verifier, falsification state, sidecars. Embeds module-graph and
verifier-ladder SVG diagrams.
- New Cookbook page (docs/_source/cookbook.rst): 8 recipes — recrawl,
falsify, ingest-self-providence, mixed-corpus query, LLM endpoint
override, integrity after bulk ops, bench, retrieval tuning.
- Quickstart embeds query-pipeline SVG diagram.
- docs/_source/diagrams symlinks to docs/diagrams so Sphinx can include
the SVGs (was orphaned, only README referenced them).
Better Makefile RTD page (docs/_source/_ext/makefile_targets.py):
- Group by workflow phase (Setup → Fetch → Ingest → Distill → Query →
Verify → Operations → Tests → Docs → Clean) instead of alphabetical
prefix. Tells a new operator the order they'd actually run things.
- Phase descriptions added; targets prefixed with 'make ' for copy-paste.
- Uncategorized leftover surfaces missing entries in PHASES list.
New docs/_source/quickstart.rst — install, two end-to-end paths
(Wikipedia 2003 + crawler), after-the-answer commands, query pipeline
overview, and links to deeper reference pages. Mirrors the README's
quickstart but adapts cross-references to the Sphinx structure.
Added to index.rst as a 'Getting started' toctree section above the
API modules — RTD users land on it first.
Use the canonical Read the Docs theme. Build dropped from 33 warnings
to 3 (most furo warnings were sidebar template lookups in the dark/light
mode switcher).
requirements.txt updated to pull sphinx-rtd-theme instead of furo.
- docs/_source/license.rst — new RTD page that literalincludes the
repo's LICENSE file (single source of truth, no duplication)
- conf.py rst_epilog — appended to every RST source so every doc page
carries the Permacomputer Preamble + AGPL-3.0-only notice + link to
the full license
- index.rst — adds 'Project / License' section to the toctree
The LICENSE file already had the full Permacomputer Preamble + complete
GNU Affero GPL v3 text (matches the whitepaper version).
Adds a Sphinx extension at docs/_source/_ext/makefile_targets.py that
parses the project Makefile's '## description' annotations and writes
docs/_source/api/makefile.rst at build time. Same convention 'make help'
uses, so the reference stays in sync with the source.
Generated page is grouped by target prefix (fetch-, ingest-, distill-,
docs-, etc.) and rendered as a list-table. Shows on RTD alongside the
autodoc API modules.
Generated file is gitignored — RTD regenerates on every build.
Initial docstring pass focused on syntax/style; this pass verified each
docstring against actual function behavior. Found and corrected:
WRONG (claimed behavior didn't match):
- _cmd_verify: claimed Q&A/audit verification — actually round-trips
Merkle proofs on N random documents
- _cmd_snapshot_verify: claimed Merkle proof round-trip — actually
re-derives snapshot root and checks for drift
- _cmd_evict: claimed 'archive unused content' — actually NULLs content
and removes FTS row; cores never evict
- _cmd_stats: listed 'index size' which is not in stats() output
OVERSTATEMENT (claim stronger than contract):
- _cmd_ask: 'grounded answer' — verifier may return UNGROUNDED
- _cmd_snapshot_list: 'named' — snapshots have hash roots, not names
MISSING IMPORTANT BEHAVIOR:
- _cmd_rehydrate: didn't mention drift-detection exit code
- _cmd_mesh_status: didn't mention 'enabled' flag (most important field)
- _cmd_distill: didn't mention recursive core→core distillation
- MerkleTree.proof(): didn't mention IndexError on out-of-range
VAGUE:
- _cmd_search: 'Search the corpus with FTS5' → mention output formats
Sphinx rebuild successful (29 warnings, down from 31).
Implements Read the Docs infrastructure to generate API documentation
directly from code docstrings. Replaces static modules.md (1200+ lines).
New structure:
- docs/_source/conf.py — Sphinx configuration (furo theme)
- docs/_source/index.rst — Main TOC
- docs/_source/api/*.rst — Module groups (substrate, storage, retrieval,
qa, distill, mesh, cli)
- docs/_source/Makefile — Local build targets
- docs/_source/README.md — Documentation on building and extending
Makefile integration:
- make docs-api — generate HTML (output: docs/_source/_build/html/)
- make docs-api-clean — remove build artifacts
Build output (40 HTML files):
- API module reference with docstrings
- Source code links (:viewcode: extension)
- Full-text search index
- Module index (genindex, py-modindex)
Sphinx installed in venv as dev dependency. HTML is browseable at
docs/_source/_build/html/index.html (open in browser after build).
This justifies the deletion of modules.md: code docstrings + Sphinx
autodoc = automatically-generated, always-current API reference.
Delete redundant, superseded, or design-log docs:
- qa-modes-bench-2026-04-30: prior snapshot (rolling journal is current)
- verifier-semantic-gap-design: unimplemented future proposal
- bench-emergent-design: stress-test design (script self-documents)
- modules.md: API reference (code + docstrings are source of truth)
- self-reference-design: v1 shipped, v2 scoped to future; history in git
- concept-relations-design: live system documented in code
- mesh-deploy: runbook for off-by-default system; mesh wire future work
Reduces docs/ from 23 files to 16, keeping north-star (seven-point-program),
architecture (cti, mesh.md), and operational docs (benchmarks, qa-modes-bench,
bench-maxing). All deleted docs recoverable from git history.
Single canonical entry point that ties together the four existing
bench-related docs (qa-modes-bench.md / bench-maxing.md /
bench-emergent-design.md / qa-modes-bench-2026-04-30.md) plus the
make targets, fixtures, and bench-row schema.
Sections:
1. Two harnesses, two purposes
bench/qa_sweep.py — curated regression bench
scripts/bench_emergent.py — random-word stress test
2. Four question fixtures (75 / 28 / 9 / 1 / smoke) with cell
sizes + use cases per fixture
3. Signal floor (5pp / n=3 × 9 = 27 / vLLM c=3-4 saturation)
4. Make targets cheat sheet (bench-qa, bench-emergent,
--policy KEY=VALUE A/B pattern, --resume)
5. Bench-row schema — every field a row carries
(identity / verdict / diagnostics / preflight projection /
quantifier classifier / capacity / directive compliance / time)
6. Where headlines live (qa-modes-bench.md addenda, per-ticket
§12/§13 bench sections)
7. bench-emergent log shape + #000006 rolling-amend pattern
8. How to run a focused A/B (the four-cell pattern from #000008)
9. How to interpret results (STRICT-rate, mean-ratio,
UNGROUNDED-rate, FORMAT_COLLAPSED rate, violation kind
distribution, audit-line tails)
10. Operator commands cheat sheet (--show-preflight,
--apply-quantifier-caps, --reject-broad, --soft-preflight,
Makefile shortcuts)
CLAUDE.md docs index updated to point to benchmarks.md as the
"read first" entry for bench work, and to add bench-emergent-design.md
to the index (was missing).
Bench-emergent stress test ran another 100 cycles under the
post-#000008/9/10/11 substrate. Total accumulated: 300 cycles.
Verdict distribution shift on last 100 vs 134-cycle baseline:
STRICT 5% (7/134) → 0% (0/100) -5pp
HYBRID 22% (29/134) → 16% (16/100) -6pp
UNGROUNDED 73% (98/134) → 84% (84/100) +11pp
Zero false-positive STRICTs across 100 random-word triplets.
The 5pp drop in STRICT-rate isn't a regression — it's the
verifier ladder + new preflight contracts doing their job.
Random-word triplets are genuinely ungrounded for the most
part; the prior 5% STRICT rate included false-positives that
the post-hardening verifier now catches.
Violation profile (last 100 cycles, claim_lattice JSON):
CITATION_MISMATCH: 86 dominant gate
TOO_MANY_EVIDENCE_IDS: 24
SUBJECT_TOKENS_ABSENT: 12 Rule 9 firing on parroting
DEFLECTION_DETECTED: 12
TITLE_MISMATCH: 10
...
metaphor_deflection fires 6/100 — still rare. Item 3
(calibration) is now closer to sample-size threshold (~30
signals across 300 cycles; needs ~50-100 to calibrate).
No new tuning candidates surface. Original three remain at
their resolution states.
Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.
#000025 — Metacog test fixture expansion:
bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
covering edge cases per detector kind: temporal (4 cases),
contradiction (4), false-premise (5), out-of-corpus (3), multi-
trigger (2), well-formed controls (5). Documents two known
detector ceilings: Q11 over-fires on past-tense factoid
("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
NASA-fake) miss false premises that lack a presupposition
pattern match. Fixture now serves as long-term regression suite.
#000026 — --show-preflight full clause render:
build_run_dag() and build_reject_run_dag() gain optional
preflight_payload kwarg. When supplied, the canonical 5-clause
CTI payload (classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs + question_state + node_version)
persists alongside the leaf hash in run_dag_blob.
aborist providence --show-preflight CACHE_KEY now renders the
full payload + verifies the persisted hash matches the
recomputed canonical hash (audit-replay tamper detection).
Legacy rows fall through cleanly: payload_hash_check reports
"unavailable: legacy row predates preflight_payload persistence".
#000027 — Latency profile:
Microbenched preflight: 0.46ms/question (negligible). Single
fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
33-35s in Addendum 3 was vLLM concurrency contention at c=4
(per qa-modes-bench.md saturation note), not substrate
overhead. Added preflight_ms + soft_preflight_ms to timings
dict for explicit confirmation in future cycles.
#000028 — Auto-quality-check sweep revival:
scripts/bench_emergent.py running with EMERGENT_N=100 in
background (PID 125680). Will accumulate cycles into
bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
Async — not blocking on completion.
#000029 — #000011 SOFT_PREFLIGHT_HINT implementation:
aborist/qa/soft_preflight.py — new module. SoftPreflightHint
dataclass + soft_preflight_question() pure function. 9
canonical labels mapping to soft analogues of #000010 hard
detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
Constrained-generation prompt (max_tokens=128, temp=0.0) asks
the model to pick ONE label + one-line rationale.
Fail-closed across every parse path:
- chat_client raises → SOFT_PARSE_FAIL
- response unparseable → SOFT_PARSE_FAIL
- label outside enum → SOFT_PARSE_FAIL
Sidecar enforces SOFT_ prefix at the normalize step so a
model that drops the prefix still gets caught.
Wired into query() between preflight & retrieval. Default
OFF (`soft_preflight_enabled: False`). NOT folded into
_VERIFIER_POLICY_FIELDS — soft hints don't gate cache
identity (#000011 §4). Audit-line tail renders as
"· soft: <label>" (e.g. "· soft: time sensitive") so the
signal is visually distinct from hard tails.
--soft-preflight CLI flag opts in per-call. End-to-end
live-verified on "When did Mr. Burns become Homer's biological
father?" — produces:
EVIDENCE-WARRANTED · via claim_lattice
· false premise · soft: time sensitive
1/1 16.4s
Hard `· false premise` (from #000010 deterministic detector)
composed with soft `· soft: time sensitive` (from #000011
sidecar). The model classified a different shape than the hard
detector — by design; soft hints are independent advisory
signals, not redundant with the hard layer.
25 new tests pin: default-OFF behavior, parse-failure modes,
label normalization (SOFT_ prefix enforced), all 8 actionable
labels round-trip, fail-closed on client exceptions, dataclass
JSON round-trip, rationale-length cap.
Other:
- #000010 §13.3 documents 2/5 metacog-trigger questions return
STRICT despite hard-detector warning — direct empirical
motivation for #000011 design.
- tests/test_dag.py extends with 3 _extract_preflight_hash_*
helper tests (cleaning #000009 §7.2 unfinished state).
- bench/emergent_log.jsonl adds new cycles from background run.
#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
Fan-out execution of the deferred-but-not-blocking pile from
prior status reports.
#000009 §7.2 — bench harness preflight_hash field:
- aborist/qa/query.py surfaces `preflight_hash` on result dict
(miss path, reject path, and cache-hit path via new helper
`_extract_preflight_hash_from_blob` that pulls the stage hash
out of persisted run_dag_blob).
- bench/qa_sweep.py adds 12-char preflight_hash prefix to bench
rows. Mirrors cache_key truncation pattern. Operators can
grep / SQL-filter bench JSONL by preflight policy state.
- 3 new tests in tests/test_dag.py for the extract helper.
#000009 §7.2 — `aborist providence --show-preflight CACHE_KEY_PREFIX`:
- New CLI flag pulls the preflight stage payload from a row's
run_dag_blob. Match by 12-char prefix. Renders preflight stage
hash + run-DAG stage list. Operator tool for inspecting which
policy state governed a cached row.
- Live verified on a real cache row (8a212fecb2a9 — current CEO
of OpenAI question, 10-stage CTI shape with preflight at idx 1).
- Legacy rows (predating #000009) report a clean fall-through
message: "run_dag has no preflight stage (predates #000009)".
#000010 §12.6 — reference-frame plumbing into QuestionState:
- Pre-retrieval preflight runs with reference_frames=()
(frame_detection needs source titles, not available yet).
Post-retrieval, query.py re-runs preflight_question() with
the detected frames so the result-dict + run-DAG QuestionState
carry frame-aware logical_statuses (specifically
`reference_frame_ambiguous` when 2+ frames match).
- Live verified on Orwell-style question; logical_statuses now
correctly includes `reference_frame_ambiguous` in the result.
Metacog-trigger bench fixture (#000010 §13.3):
- bench/qa_questions_metacog_subset.txt — 6 questions, one per
detector kind plus a well-formed control.
- Bench artifact 2026-05-04T02-18-42Z. Detector accuracy 6/6
on fixture; 2 of 5 trigger questions return STRICT on lattice
mode despite metacog warning (JSON STRICT on
George-Washington-stop-being-president-of-France false-premise
+ uploaded-contract out-of-corpus questions). Audit-line tails
correctly surface the warnings.
- qa-modes-bench.md Addendum 4 captures the per-question matrix
+ interpretation. #000010 §13.3 cross-references with bench
artifact stamp.
#000011 SOFT_PREFLIGHT_HINT design ticket opened:
- docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md
captures the design proposal per #000010 §18 / source doc.
Implementation deferred — design only.
- Sidecar would add model-assisted preflight as a soft signal
(`SOFT_FALSE_PREMISE_SUSPECTED` etc.) that NEVER enters the
verifier hard path. Strict guardrail: cannot create
PREFLIGHT_OK or PREFLIGHT_BLOCKED without deterministic
support.
- Validated by §13.3 finding: deterministic detectors flag
correctly; corpus-accidental grounding produces 2/5 STRICT
on trigger questions; soft sidecar would add independent
semantic skepticism.
- TICKETS.md index row added; Next ID bumped to 000012.
996 tests passing (3 new for the extract helper).
Cross-doc consistency:
- qa-modes-bench.md Addenda 1+2+3+4 chronological
- #000010 §13.1 (broad subset) + §13.2 (full bench) + §13.3
(metacog trigger subset)
- #000011 design captured but not implemented
Cross-references the 2026-05-03T23-30-12Z full 75-question
regression bench (Addendum 3 in qa-modes-bench.md) into:
- #000010 §13.2 — full-bench validation alongside the §13.1
broad-subset validation. Same verdict: defaults stay on.
Substrate-level wins beyond noise-bounded SR deltas:
pointer FORMAT_COLLAPSED: 0/225 (eliminated globally)
pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only)
JSON mean ratio: +3pp
- seven-point-program.md addendum — bench-validation paragraph
after the test-count line. Names the SR deltas + the
substrate-level wins; concrete evidence the post-landing
directive coverage claim survives full-corpus measurement.
No code changes; 993 tests still passing.
Addendum 3 to qa-modes-bench.md. Validates that flipping
quantifier_reminder_enabled=True for lattice modes (per #000010
§12.10 / §13.1) doesn't regress narrow-question performance.
Prior validation (Addendum 2) covered the 9-question broad subset
only. This run sweeps the full 75-question bench/qa_questions.txt
(~10% broad, ~89% narrow), 225 runs per mode, comparing against
the frozen 2026-05-02T15-07Z baseline.
Findings:
Mode | Pre-flip SR | Post-flip SR | Δ
----------+-------------+--------------+--------
quote | 0.54 | 0.52 | -2pp (within 5pp floor)
pointer | 0.20 | 0.21 | +1pp
JSON | 0.42 | 0.44 | +2pp
Mean ratio: -1/+2/+3pp — all within noise band.
Substrate-level wins beyond the headline metrics:
- pointer FORMAT_COLLAPSED: 0/225 across the full sweep.
Reminder eliminates collapse mode globally, not just on broad.
- pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when
reminder was off). Citation discipline propagates beyond the
rows where the reminder text actually fires — the model's
session attention reinforces.
- JSON mean ratio +3pp consistent with broad-subset finding.
Quote-mode is essentially unchanged because it's mode-gated off
the guard by default.
Latency 33-35s/call this evening (vs 17-19s in prior runs) is
endpoint contention, not a substrate regression — preflight adds
zero LLM calls.
Verdict: no regression. The default flip ships clean across the
full corpus. Defaults stay on. The substrate is strictly more
honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without
sacrificing throughput on non-broad questions.
Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}.
Post-landing validation cell. Same 9-question broad subset as
§12.6 reminder-only baseline; this run flips both
metacognition_enabled and quantifier_reminder_enabled to False so
we can isolate the preflight contribution.
Metric | OFF | ON | Δ
--------------------------+-----------+----------+----------
pointer mean ratio | 0.483 | 0.643 | +16pp
JSON mean ratio | 0.570 | 0.735 | +17pp
JSON UNGROUNDED rate | 7/27 | 1/27 | -22pp
pointer FORMAT_COLLAPSED | 2/27 | 0/27 | -100%
any-mode STRICT-rate | within ±11pp noise (27-sample n=3 floor)
Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all clear the
5pp signal floor on lattice modes. STRICT-rate moves are within
Hermes nondeterminism. The #000010 default flip is doing what
was claimed.
On this subset none of the metacognition detectors fire (no
temporal / contradiction / false-premise / out-of-corpus shapes),
so the delta effectively isolates the reminder contribution from
#000008. A metacog-trigger subset bench is deferred.
Documentation:
- docs/qa-modes-bench.md Addendum 2 captures the comparison
table + interpretation + verdict.
- docs/tickets/ticket-000010-... §13.1 cross-references with
bench artifact stamp.
Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}.
Three Explore agents fanned out in parallel for a docs/ + diagrams/
+ code-comment audit against the shipped state of the three
preflight tickets. This commit lands all the alignment fixes.
Core docs updates:
CLAUDE.md
- dag.py module description: stage counts now read
"7/8 quote · 9/10 CTI · 3 reject" reflecting #000009 preflight
stage + reject-broad early-return shape.
docs/cti-architecture.md §2.2 + §2.3
- §2.3 Merkle-AGI-DAG section rewritten: documents all five DAG
shapes (legacy 7/9, post-#000009 8/10, reject-broad 3),
describes the preflight stage's 5 nested CTI clauses
(classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs), pins
PREFLIGHT_NODE_VERSION = "preflight-node-v1", states the
audit-replay payoff.
- §2.2 CTI section: adds the four new modules
(quantifier, model_profiles, quantifier_reminder,
metacognition) as code anchors. Notes that pre-answer
preflight contract extends CTI upstream of retrieval.
docs/seven-point-program.md
- D3 status ½ → ¾ — pre-answer preflight contract landed via
#000008 + #000010. Code anchors + pinning tests updated.
- D4 status ½ → ¾ — preflight stage adds upstream control
commitment to the run-DAG. Code anchors include
build_reject_run_dag + preflight_node_hash.
- Status snapshot table: tickets column now references
#000008/#000009/#000010 against D1/D3/D4 directives.
- "Post-landing addendum (2026-05-03 / 2026-05-04)" subsection
summarises all three tickets + their commit shas + final
test count (993 passing, up from 734).
docs/modules.md
- Q&A pipeline table: added 4 new modules (quantifier.py,
model_profiles.py, quantifier_reminder.py, metacognition.py).
dag.py row updated to "7/8 quote · 9/10 CTI · 3 reject".
- dag.py subsection rewritten: documents all 5 DAG shapes,
describes the preflight payload's 5 clauses + question_state.
- 4 new module subsections (quantifier / model_profiles /
quantifier_reminder / metacognition) explaining each
module's purpose, signature, and how it feeds the run-DAG
preflight clause.
Diagram updates:
docs/diagrams/query-pipeline.dot + .svg
- New "PREFLIGHT (#000008 + #000010)" node inserted between
cache_check and concepts_lookup.
- New "REJECT-BROAD" node showing the 3-stage minimal DAG
escape path.
- render node label extended with the audit-line tail token
catalog.
docs/diagrams/aborist-modules.dot + .svg
- 4 new qa_* nodes in the retrieval & verifier cluster.
- 8 new edges: qa_query/qa_runner each call into all 4
preflight modules; qa_dag has dotted edges to qa_quantifier
+ qa_metacognition (preflight clause sources).
- qa_dag label updated to mention preflight_node_hash + 5 clauses.
docs/diagrams/verifier-ladder.dot + .svg
- Soft-demote violations list extended: BROAD_QUANTIFIER_RUNAWAY
/ CAP_APPLIED / SCOPE_UNBOUND, FORMAT_COLLAPSED, BARE_NAME_CLAIM.
- New "AUDIT-LINE TAILS" annotation node listing all 11 tail
tokens (#000008 broad-* + #000010 metacog + classic verifier).
- Dashed edges from each rung to tails note showing tails
compose onto labels.
Code-side stale-comment fixes (caught by 3rd Explore agent):
aborist/qa/keys.py:218
- "The four fields" → "The seven fields"; mention #000010 adds
six more for metacognition.
aborist/qa/query.py:2644
- 7-stage / 9-stage comment expanded to enumerate all four
base+preflight shapes plus the 3-stage reject path.
aborist/qa/runner.py:835
- same expansion as query.py for runner.ask() callsite.
mesh-*.dot, ingest-pipeline.dot, qa-modes-bench.md, bench-maxing.md,
bench-emergent-design.md, verifier-semantic-gap-design.md,
self-reference-design.md, concept-relations-design.md confirmed
orthogonal — no edits needed.
993 tests still passing (no behavior change). 7 files modified
across docs/ + 3 dot diagrams + 3 SVGs + 4 code-comment fixes.
Architectural feedback at ~/Downloads/RESPONSE-ticket-000009-... .txt
(2026-05-04) flagged five gaps in the c36e85c landing. Most
critical: reject-broad early-return path emitted no run_dag_blob,
so audit replay couldn't see that a rejection happened (let alone
under what policy state).
A — reject-path DAG (the critical gap):
aborist/qa/dag.py: build_reject_run_dag() — 3-stage minimal DAG
question → preflight → final_label. final_label payload carries
rejection_reason + answer_text_hash so two rejections under
different policy state produce different roots.
query.py reject path now wires it: returns run_dag_root +
run_dag_blob on the rejection result dict. Live-verified end-
to-end on `make query Q="winners of all major sports?"
REJECT_BROAD=1 BURN=1`.
Audit replay rule: 3 stages always means reject path. Operators
can read the stage list and tell instantly without parsing the
payload.
B — nested CTI clauses:
preflight_node_hash() payload restructured from flat 3-key to
nested 5-clause:
classifier — quantifier classifier output (#000008)
answer_contract — guard / cap / reject / metacog state (per-run)
prompt_contract — reminder enabled / injected / template_id
evidence_contract — exposure budget, line discipline
policy_refs — governance_policy_hash, model_profile_hash,
answer_mode (reference, not raw policy)
Plus question_state (metacog) as its own clause and top-level
stage + node_version. Single DAG stage; nested clauses inside
for diff legibility (feedback §3).
C — node_version field:
PREFLIGHT_NODE_VERSION = "preflight-node-v1" pinned in the
payload so legacy runs without the node can be unambiguously
labeled `unavailable_legacy_run` by audit tools (feedback §9).
D — reference hashes only:
policy_refs uses governance_policy_hash + model_profile_hash
rather than bundling raw policy booleans. Avoids
double-committing already-hashed state (feedback §4).
E — reminder_template_id:
prompt_contract.reminder_template_id = "broad-quantifier-bounded-v1"
or "broad-quantifier-unbounded-v1" depending on scope_bound_hint,
populated only when reminder actually fires.
F — stage name kept as `preflight` (not `quantifier_preflight`):
Node carries both #000008 quantifier AND #000010 metacognition
payloads. node_version disambiguates schema for audit tools.
G — docs/cti-architecture.md update deferred to a small follow-up.
Bug fixes:
- free-variable shadowing on verifier_policy_hash /
model_profile_hash / question_hash — local re-imports inside
the reject branch shadowed module-top imports used elsewhere
in query() / runner(); now use the module-top names.
- reject path question_hash signature: takes `mode=` not
`dedup_mode=` — fixed in the reject DAG builder caller.
Hash compatibility:
Rows written between c36e85c and this commit have hash payloads
matching the OLD flat 3-key shape. The persisted run_dag_blob
captures the actual payload that was hashed, so those rows
still verify via verify_run_dag(). New rows use the nested
5-clause shape.
7 new tests in tests/test_dag.py:
- hash sensitivity to answer_contract / prompt_contract /
policy_refs flips (audit-replay payoff demonstrations)
- PREFLIGHT_NODE_VERSION pinning
- reject DAG: 3-stage shape, root changes with preflight hash,
round-trips through verify_run_dag
993 tests passing (6 net new); 36 skipped.
Live verification:
make query Q="winners of all major sports?" REJECT_BROAD=1 BURN=1
→ status=broad_quantifier_rejected, run_dag_root populated,
blob carries 3-stage shape.
make query Q="winners of all major sports?" BURN=1
→ 10-stage shape preserved (question → preflight → retrieval
→ ... → final_label).
Ticket #000009 status: closed · re-landed 2026-05-04 with §8
corrections.
Closes ticket #000009 zero-shot. Scope expanded to cover BOTH
ticket #000008 (broad-quantifier) AND ticket #000010
(meta-cognition) preflight contracts in a single combined node —
both share the same audit-replay gap and inserting two separate
nodes between question and retrieval was operationally awkward.
aborist/qa/dag.py:
+ preflight_node_hash() — combines QuestionState +
quantifier classifier output + behavioral policy_state into
one canonical SHA-256 hex.
+ build_run_dag() gains optional preflight_hash parameter.
When supplied, inserts {"stage": "preflight", "hash": ...}
at position 1 (between question and retrieval).
Backward-compat: None → original 7/9-stage shapes preserved
for legacy run_dag_root re-validation.
Quote-mode: 7 → 8 stages with preflight.
Pointer-mode CTI: 9 → 10 stages with preflight.
aborist/qa/query.py + runner.py:
Both build the preflight payload from question_state +
quantifier dict + 10-field policy_state (guard_enabled,
guard_apply_caps, guard_apply_caps_mode_gated,
claim_cap_resolved, claim_cap_actually_applied,
reminder_enabled, reminder_eligible, reject_broad_active,
metacognition_enabled, block_on_contradiction).
This means two cache rows that share the same question + same
model output + same verifier verdict but DIFFERENT preflight
policy state now produce different run_dag_root values. Audit
replay can pin the policy decision per row.
9 new tests in tests/test_dag.py:
- preflight_node_hash determinism
- hash bumps on question_state change
- hash bumps on policy_state change (the audit-replay payoff)
- all-None defensive shape
- 7→8 stage transition (quote mode)
- 9→10 stage transition (pointer mode), preflight at index 1
- run_dag_root bumps when preflight_hash bumps
- verify_run_dag round-trips through preflight stage
Live verification: latest providence_cache row carries
['question', 'preflight', 'retrieval', 'evidence_map', 'prompt',
'raw_answer', 'parsed_claim_lattice', 'verify', 'render',
'final_label'] — preflight stage living in the persisted DAG.
987 tests passing (9 new); 36 skipped.
Tickets:
#000009 status: closed · landed 2026-05-03 (zero-shot)
#000010 cross-ref updated: "DAG binding shipped via #000009"
What's NOT in this ticket (logged in §7.2):
- CLI flag for inspecting preflight node from cache_key
- Bench harness preflight_hash field for cross-row comparison
- SOFT_PREFLIGHT_HINT (model-assisted preflight sidecar)
Closes ticket #000010 (Meta-Cognition Preflight Guard). Mechanism
complete; defaults preserve the dry-run discipline pattern from
#000008.
Phase 2 — wire preflight into query() and runner.ask():
- preflight_question() runs after policy resolution + quantifier
classification, before retrieval.
- QuestionState surfaces on miss path, cache-hit path, AND
reject-broad early-return path of query() — schema column-
aligned across all four returns.
- runner.ask() carries the same fields for `aborist ask` parity.
Phase 3 — policy fields + governance hash + CLI flags:
- 6 new policy fields, all default-on except
metacognition_block_on_contradiction (default False — label-
only by default; opt-in via --block-on-contradiction).
- All 6 folded into _VERIFIER_POLICY_FIELDS so flipping any
invalidates prior cache records on lookup.
- 2 new CLI flags on `aborist query`:
--no-preflight Level 2 master kill
--block-on-contradiction strict mode (hard-block on
lexical contradictions)
Phase 4 — audit-line labels + bench fields + tests:
- _render_warrant_tail extended with 5 metacog tail tokens:
· false premise
· contradictory
· stale risk
· out of corpus
· frame ambiguous
- Bench rows in qa_sweep.py gain 7 new bounded-size projection
fields (logical_statuses, question_shape, preflight_result,
temporal_sensitivity, has_false_premise, has_contradiction,
corpus_requirement). Full QuestionState stays on result dict
for CLI render only.
- tests/test_metacognition.py grew from 42 → 68 tests
(16 new: 6 governance + 6 audit-line tail + 4 default-policy
pinning).
Live verified end-to-end:
$ make query-dry Q="Who is the current CEO of OpenAI?" BURN=1
UNGROUNDED · via claim_lattice · stale risk
$ make query-dry Q="When did Mr. Burns become Homer's biological
father?" BURN=1
UNGROUNDED · via claim_lattice · false premise
978 tests passing; 36 skipped.
What's NOT shipped (deferred):
- Run-DAG node binding for metacognition_preflight stage —
joins ticket #000009 Phase 5 (same audit-replay gap; both
nodes can land together).
- Reference-frame plumbing — frame_detection runs post-retrieval,
preflight here is pre-retrieval; deferred until two-pass
or post-classification update lands.
- SOFT_PREFLIGHT_HINT (model-assisted sidecar) — source doc §18
reserves this label; hard rule preserved (no LLM in preflight
hard path).
- Bench A/B measuring preflight on vs off — quick to run once
stack settles.
Ticket #000010 status: closed · landed 2026-05-03.
Implements the Meta-Cognition Preflight Guard (M0 / MCTL) per
fox's directive at ~/Downloads/meta-cognition_for_hermes(1).txt
(2026-05-03).
aborist/qa/metacognition.py:
- QuestionState dataclass (frozen, JSON-serializable via to_dict)
- preflight_question() pure function: classifies a question
deterministically into a QuestionState before generation
- 4 new detectors:
detect_temporal_sensitivity() — current/latest/today/CEO/etc.
detect_contradiction() — lexical pairs (unmarried+spouse,
always+never, alive+dead, etc.)
detect_false_premise() — presupposition patterns:
when did X stop/become Y,
why did X cause Y,
how did X become Y
detect_out_of_corpus() — my-uploaded-X / file-I-sent shapes
- Reuses #000008 quantifier classifier (no duplication)
- Composes 8 LogicalStatus values:
well_formed, under_specified, false_premise_suspected,
contradictory_question, out_of_corpus_risk, stale_risk,
reference_frame_ambiguous, broad_quantifier_unbounded
- Three preflight results: PREFLIGHT_OK / _PARTIAL / _BLOCKED
- Per-detector enable switches in policy:
metacognition_enabled (master kill)
metacognition_temporal_check
metacognition_contradiction_check
metacognition_false_premise_check
metacognition_out_of_corpus_check
metacognition_block_on_contradiction (default False — label
only by default; opt-in
to hard-block)
- preflight_policy_hash for governance binding (Phase 3)
- PREFLIGHT_VERSION = "metacognition-v0.1"
Hard rule (D1): no LLM in this hard path. Pure regex + lexical
matching. Model-assisted preflight, if added later, labels itself
SOFT_PREFLIGHT_HINT (not implemented in this phase).
42 new tests cover the seven test cases from source doc §14
(false-premise, contradictory, broad-quantifier, reference-frame,
time-sensitive, out-of-corpus, model-cutoff) plus per-detector
unit tests, gating (master kill, per-detector disable,
block-on-contradiction opt-in), determinism (question_hash
stable, policy_hash bumps on flip), and serialization.
Ticket #000010 opened with status `open · in progress
(zero-shot 2026-05-03)`. TICKETS.md index updated; Next ID bumped
to 000011.
Phases 2-4 still queued (wire into query/runner, policy fields +
governance, audit-line labels + bench fields).
962 tests passing (42 new); 36 skipped.
Closes#000008 with status `closed · landed in 4f2b5a6` per the
docs/TICKETS.md convention. The preflight guard mechanism + bench
cycle + default flip all shipped 2026-05-03; the design log stays
in place.
Opens #000009 — Quantifier preflight run-DAG node binding. Splits
the Phase 5 follow-up out of #000008 §11.11 into its own ticket.
Scope: bind the classifier output + policy decision into
`run_dag_root` so audit replay can distinguish guard-on vs guard-off,
cap-applied vs not, reminder-injected vs skipped. Currently those
appear on the result dict but are NOT in the run-DAG hash. Required
to close the audit-replay gap that blocks the §9.5 Merkle-AGI-DAG
framing from fully holding. Estimated 3-4h. Awaiting go/no-go.
Makefile shortcuts for the #000008 CLI flags (operator ergonomics):
BROAD=1 → --apply-quantifier-caps (flip cap apply-gate)
REJECT_BROAD=1 → --reject-broad (preflight rejection)
ALLOW_BROAD=1 → --allow-broad (emergent search)
Available on both `make query` and `make query-dry`. Default
behavior unchanged: ANSWER_MODE=claim_lattice (JSON), reminder ON
for lattice modes, cap operator-opt-in.
Smoke-tested:
$ make query-dry Q="winners of all major sports?" BROAD=1
→ cap applies on JSON; classifier reports ALL/unbounded
$ make query-dry Q="winners of all major sports?" REJECT_BROAD=1
→ preflight rejection, exit-1 (consistent with UNGROUNDED)
TICKETS.md index:
#000008 closed · landed in `4f2b5a6`
#000009 open · awaiting go/no-go (D3, D4)
Next ID bumped 000009 → 000010
n=5 verification of cap+reminder cell (135 runs):
Metric | n=3 | n=5
--------------------+-----------+------------
JSON SR | 0.30 | 0.33 ← matches cap-only
JSON UNGROUNDED rate| 1/27 (4%) | 2/45 (4%) ← matches reminder-only
pointer SR | 0/27 | 0/45 ← unchanged across all cells
The §12.8 0.30 was Hermes nondeterminism. n=5 confirms cap+reminder
delivers cap-only's STRICT-rate AND reminder-only's UNGROUNDED-rescue.
§10.8 strict gate met at n=5:
vs reminder-only on JSON SR: +11pp (clears floor)
vs cap-only on JSON UNGROUNDED: -18pp (clears floor)
vs cap-only on ptr mean ratio: +12pp (clears floor)
Defaults flipped — Option A landing (per-mode tailored):
quantifier_reminder_enabled False → True
(load-bearing on both lattice modes)
NEW field: quantifier_apply_caps_modes = ["claim_lattice"]
(allowlist for which modes apply caps
when apply_caps=True; JSON-only since
cap-on-pointer is wasted noise per
§12.10 0/45 STRICT data)
quantifier_guard_apply_caps False → False (UNCHANGED)
(operator opts in via
--apply-quantifier-caps; preserves
§10.11.3 dry-run discipline)
Cap-application gate now reads:
if apply_caps AND mode in apply_caps_modes AND cap is not None:
effective_max_claims = looked_up_cap
quantifier_apply_caps_modes folded into _VERIFIER_POLICY_FIELDS so
flipping the allowlist invalidates prior cache records.
5 new tests pin: reminder default ON for both runner.DEFAULT_POLICY
and query.DEFAULT_QUERY_POLICY; apply_caps_modes default
["claim_lattice"]; governance-hash invalidation on allowlist flip;
apply_caps default still False (dry-run preserved).
920 tests passing (5 new); 36 skipped.
Operator behavior:
$ aborist query "winners of all major sports?"
→ reminder ON, cap OFF (default after this commit)
$ aborist query --apply-quantifier-caps "..."
→ cap applies on claim_lattice (JSON) only
$ aborist query --apply-quantifier-caps \
--policy quantifier_apply_caps_modes='["claim_lattice","claim_lattice_pointer"]' "..."
→ Option D for one call
Phase 5 (run-DAG node binding for quantifier_preflight) and
cross-model Qwen/GPT-4 verification remain as follow-ups per §11.11.
#000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs):
Metric | Base | Rem | Cap | Cap+Rem
--------------------+------+------+------+--------
JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR
JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue
pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best
pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven
§10.8 strict gate "compound beats either alone by ≥5pp on every metric"
NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR
(clears floor) and beats cap-only by +17pp on pointer mean ratio
(clears floor), but is -3pp vs cap-only on JSON SR (regression,
within noise).
#000008 §12.9 — Final verdict + recommendation:
Mechanism asymmetry (clean signal):
Reminder rescues UNGROUNDED → HYBRID
Cap rescues HYBRID → STRICT
Recommendation: Option A — single-knob defaults, per-mode tailored:
claim_lattice (JSON): apply_caps=True + reminder=True
claim_lattice_pointer: apply_caps=False + reminder=True
(cap can't rescue pointer-tag discipline
upstream of cap; cap fires 20× without
verdict gain)
quote: guard mode-gated off (already default)
Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5
verification on cap+reminder before flipping defaults.
Cross-doc updates:
- CLAUDE.md: architecture diagram now lists the three new Phase 1-3
modules (quantifier.py, model_profiles.py, quantifier_reminder.py).
New "Broad-quantifier preflight guard" conventions entry covers
the 7 policy fields, six-level disable hierarchy, dry-run
defaults, CLI flags, and §12 bench summary.
- docs/qa-modes-bench.md: addendum at end pointing at #000008 §12
four-cell A/B + 4 bench artifact stamps. Original 2026-05-02
journal frozen; 2026-05-03 broad-subset findings flagged as
question-mix-dependent narrative on top of the global
per-mode recommendation.
§12.7 captures the 2026-05-03T12-47-23Z cap-only A/B (apply_caps=
True, reminder=False) on the 9-question broad subset.
Mode | Baseline | Reminder | Cap-only
----------+----------+----------+----------
JSON SR | 0.19 | 0.22 | 0.33 ← +14pp
JSON U | 7 | 1 | 6 ← reminder dominates
JSON S | 5 | 6 | 9 ← cap dominates
ptr ratio | 0.473 | 0.643 | 0.516
JSON ratio| 0.524 | 0.735 | 0.643
§10.8 gate MET on JSON mode (+14pp STRICT-rate).
Headline insight: cap and reminder help in DIFFERENT ways.
- Reminder rescues UNGROUNDED → HYBRID (restates citation rule).
- Cap rescues HYBRID → STRICT (forces fewer-but-better claims).
The two are complementary, not redundant. If §12.8 (cap+reminder)
confirms the compound effect, that's the §10.8 trigger to land
Option D as default. Predicted JSON SR ~0.40 if effects compound.
Pointer mode still 0/27 STRICT under cap-only — TOO_MANY_CLAIMS
fires 20× (vs 7× baseline) but pointer-tag failures upstream of
the cap still gate the verdict.
Single-source-of-truth section for what was actually built in the
2026-05-03 implementation pass. Complements §8 (commit table) and
§9.6 (per-phase notes) with a full inventory:
- §11.1 New modules: aborist/qa/{quantifier,model_profiles,quantifier_reminder}.py
- §11.2 Modified modules: query.py, runner.py, keys.py, cli.py, qa_sweep.py
- §11.3 New test files: 4 new + 1 extended; 120 new tests total
- §11.4 Seven new policy fields in _VERIFIER_POLICY_FIELDS
- §11.5 Four new CLI flags on `aborist query`
- §11.6 Four new violation kinds (3 soft + 1 hard) with audit tails
- §11.7 New result-dict fields on miss + cache-hit paths
- §11.8 Eight new bench-row fields
- §11.9 Implementation-time decisions not in §9/§10:
RUNG_PRIORITY ordering, bounded-domain anchors, reminder templates,
reject answer_text format, render branch, EXPLICIT_COUNT fallback,
quote-mode opt-out
- §11.10 Live verification artifacts (Winners-of-all-major-sports
rejected; Beatles-bounded NOT rejected)
- §11.11 What was NOT shipped: quantifier_preflight run-DAG node,
three-clause CTI contract DAG binding, A/B/D bench measurements,
cross-model verification — all queued as Phase 5 / bench follow-up
Implementation-time decisions section is the most operationally
useful — captures judgment calls made during coding that aren't in
the design docs but are now binding via tests.
Updates §8 Status, §9.6 Phase details, §9.8 Test surface, §9.11
commit sequence to reflect actual implementation state:
§8 Status — Phase 0 through Phase 4 all landed 2026-05-03 across
six commits (2ffed00 → 5a60e85). 906 tests passing (120 new).
Defaults preserve §10.11.3 dry-run discipline:
quantifier_guard_apply_caps=False, reminder=False, reject=False.
Six-level disable hierarchy fully wired. Live verification recorded
for both reject-broad-fires (Winners of all major sports?) and
reject-broad-skips (name all members of the Beatles → bounded
universal, NOT rejected).
§9.6 Phase details — each phase tagged LANDED <sha>; description
trimmed to what actually shipped vs the original proposal.
§9.8 Test surface — every check box flipped to [x] with the
matching test file path and test count. The two reject-broad
integration tests remain skipped (exercised by live bench).
§9.11 Implementation commit sequence — actual SHAs replace planned
commit numbers. Notes that DAG/audit binding for quantifier_preflight
node ("commit 7" in plan) is NOT shipped — tracked as Phase 5
follow-up. Optional now; required for §9.5 Merkle-AGI-DAG framing
to fully hold.
Next steps are bench measurement, not code: full bench under dry-run,
classifier review, then flip apply_caps and measure §10.8 deltas.
Closes the bench-side gap surfaced in §5.2: JSONL was carrying summary
numbers only, blinding the harness to FORMAT_COLLAPSED rate and per-
violation-kind distributions. Without these, A/B/D bench measurements
on the broad-quantifier preflight guard would be guesses.
- query() result dict surfaces format_collapsed + raw_answer (lattice
modes only) so the bench can read them directly instead of re-deriving
from cache rows that --burn overwrites.
- Each bench row gains format_collapsed, violation_kinds (sorted unique
list — full payloads stay off the row to keep size bounded), and
answer_brackets (count of [E\d+] in raw_answer for lattice modes).
- _summarize aggregates per-mode FC count (only explicit True; None
means check didn't apply), per-kind tallies (each kind once per row),
and lattice-only bracket sum/n.
- Markdown renderer adds a `## format-collapse + violation kinds`
section with per-mode FC rate, mean raw brackets, and one column per
observed violation kind. Degrades gracefully when the sweep produces
no violations.
- 5 new bench-harness tests pin the aggregation rules.
Re-baseline (2026-05-02T20-58-57Z) sharpens §5.1 analysis dramatically:
NO_EVIDENCE_POINTER fires 3/3 in pointer mode and is the dominant gate,
not TITLE_MISMATCH (1/3) as §5.1 inferred from JSONL alone. FORMAT_
COLLAPSED actually fires 1/3 — not the rare corner the first baseline
called it. Implies Option B (prompt reminder) is the load-bearing fix
for the verdict; Option A (cap reduction) only moves secondary kinds.
§5.3 sub-investigation closed on first read — SCHEMA_INVALID:1 in
pointer mode is a legitimate kind emitted by verify_claim_lattice for
empty-claim-text (verify.py:1242) and bare-name-claim (verify.py:1270),
not a JSON-mode leak.
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.
Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.
verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict
Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
into verifier_policy_hash
- threaded through ask() and query() call sites
CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail
Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.
Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).
Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.
Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.
New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.
Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
gain `subject_tokens_absent_threshold` kwarg + per-claim check
block (mirrors TITLE_MISMATCH plumbing, sits right after it
in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
`verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value
Live verification (cache-split cleanly via policy-hash bump):
pre-fix cache_key 08dbd2c1… : STRICT (false positive)
post-fix cache_key 6a519636… : UNGROUNDED
Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT
Full suite: 776 passed, 34 skipped.
Re-aggregated bench/emergent_log.jsonl at 200 cycles. Distribution
holds (79.5% UNGROUNDED, 17.5% HYBRID, 3.0% STRICT). Six STRICTs
total; five defensible (known-truth or non-relationship grounding).
The sixth — `steer/reply/correcter` — is the FIRST CONFIRMED
EVIDENCE-WARRANTED false-positive in 200 cycles. Decompressed all
10 chunks of the cited document (Glossary of language teaching
terms and ideas, 33.5K chars) and confirmed the words `correcter`,
`steer`, and `reply` appear NOWHERE in the source. Yet STRICT 1/1
verified, no violations.
Mechanism: per-claim citation-coverage Rule 5 (≥30% claim tokens
in cited span) passed on GENERIC linguistic vocabulary (language,
communication, grammar, exchange, relationships) which appears in
any linguistics article. The question-distinctive tokens
(correcter, steer, reply) rode along unverified. PREMISE_PARROTING
shape predicted in original ticket §D, exhibit A.
Headline correction: from "zero false-positives" to "one confirmed
false-positive (0.5%)."
Action item: opens design space for #000008 — PREMISE_PARROTING
detector. Lexical signal: question-distinctive tokens absent from
cited chunk while Rule 5 still passes on generic vocabulary →
SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays
binary, stays lexical, folds into verifier_policy_hash.
The ticket file was at docs/ticket-000006-... at top-level; the
tickets/ subdir convention landed before #6 was opened. Move into
docs/tickets/ to match every other ticket.
Amend with the 62-entry post-ticket delta. bench/emergent_log.jsonl
now holds 134 cycles total. Distribution stable vs original 72:
106 UNGROUNDED (79.1%), 22 HYBRID (16.4%), 6 STRICT (4.5%). Ten
new grounded cases all carry appropriate violation labels — zero
EVIDENCE-WARRANTED false-positives across all 134 cycles. Original
tuning candidates: Tomas-deflection resolved by a1dd330,
xxviii-STRICT kept as designed, metaphor sidecar calibration
deferred (still rare).
Reframe ticket as a rolling research log. Emergent stress-testing
is an ongoing thread, not a one-shot defect-fix; future
bench-emergent re-runs append new ## Amend sections here, and any
code-level tunings open their own tickets linking back.
TICKETS.md row reflects the rolling-research framing. Stale
"## Next ID" footer scrubbed from the ticket body (TICKETS.md is
canonical for the next-id counter).
