The doc-drift pattern recurred four times today on 2026-05-10
(commits 6cbbf95, 14bcb99, 5c21e83, 30a9488). Each fix was the
same shape: walk a doc, find a count that drifted from live truth
during the hours after the doc was written, refresh it. Cost: ~5
min per drift × 4 = 20 min of manual catching, with no guarantee
the next drift gets caught before someone external reads it.
Per fox's selection: regression test that makes drift loud at
test time instead of relying on visual catching.
== Mechanism ==
`tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT
tags of the form:
<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->
Two metrics supported:
- `tests` — pytest collected count for path. Batches every
tagged path into one `pytest --collect-only` subprocess
(~0.5s total).
- `fixture-rows` — non-blank-non-comment line count in a JSONL
fixture.
GitHub and most markdown renderers strip HTML comments, so
readers see only `N`. The tags are invisible in rendered output
but make the claim machine-checkable. Three tests in the file:
1. `test_doc_autocount_claims_match_live` — the core invariant
2. `test_autocount_tags_are_well_formed` — open/close balance
3. `test_autocount_metric_names_are_documented` — fail-closed on
undocumented metrics (catches typos)
Failure message names the doc file, line number, and the
claimed-vs-live diff. Example:
`docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27`
== 29 tags installed across 5 docs ==
While installing tags I had to read the surrounding prose, which
surfaced six stale counts that had drifted same-day:
`docs/soft-hash-channel-analysis.md`:
- L392 14 → 23 tests for phi_alignment_probe
- L417 20 → 27 tests for anchor_prg
- L463 14 → 23 tests for phi_alignment_probe (status section)
`docs/seven-point-program.md`:
- L77 68 → 58 tests for metacognition (drift -10; the file
shed tests during a refactor and the doc didn't catch up)
- L78 9 tests for `test_dag.py::test_preflight_*` — removed
count entirely; pytest selector subsets aren't currently
supported by the AUTOCOUNT metric set (would need a
`tests-matching` metric; not worth the surface for one claim).
- L110 24 → 33 tests for test_dag.py
`docs/calculator-test-patterns.md`:
- L35 33 → 23 tests for warrant_resolver
- L35 10 → 9 tests for warrant_chain
- L16, L265 51 → 53 tests for t3_bound_calculator (kept
initial-shipment provenance in prose)
== Coverage installed ==
calculator-test-patterns.md 3 tagged claims
soft-hash-channel-analysis.md 5 tagged claims
warrant-substrate-cookbook.md 14 tagged claims
seven-point-program.md 3 tagged claims
tickets/ticket-000006-bench-... 4 tagged claims
---
29 tagged claims
Every count that drifted today is now tagged. Future drift
fires the regression test at the next pytest run instead of
waiting for human catching.
== Discipline pattern ==
Walk this pattern for any new doc that names a count:
1. Surround the number with the tag pair:
`<!--AUTOCOUNT:tests:tests/test_foo.py-->N<!--/AUTOCOUNT-->`
2. Run `pytest tests/test_doc_counts.py` (~3.5s)
3. If it passes, the claim is now machine-verified
Aim to tag counts on first authorship. Retrofitting is cheap
but only catches drift after the fact.
== Out of scope ==
Test counts inside source code (docstrings, CLI --help) are not
scanned — would expand the test surface significantly and the
drift pattern hasn't manifested there. Add `**/*.py` scope when
that pattern surfaces.
Alias-row counts and claim-pack-record counts could be tagged
with new `db-rows:<table>` and `db-where:<sql>` metrics; deferred
until the next drift on those numbers (none caught today after
30a9488's cookbook refresh).
== Verification ==
$ .venv/bin/pytest tests/test_doc_counts.py -v
3 passed in 3.89s
$ .venv/bin/pytest -q
2276 passed, 54 skipped in 153.21s
No new dependencies. No schema changes. No source-code changes.
Cross-references the 2026-05-03T23-30-12Z full 75-question
regression bench (Addendum 3 in qa-modes-bench.md) into:
- #000010 §13.2 — full-bench validation alongside the §13.1
broad-subset validation. Same verdict: defaults stay on.
Substrate-level wins beyond noise-bounded SR deltas:
pointer FORMAT_COLLAPSED: 0/225 (eliminated globally)
pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only)
JSON mean ratio: +3pp
- seven-point-program.md addendum — bench-validation paragraph
after the test-count line. Names the SR deltas + the
substrate-level wins; concrete evidence the post-landing
directive coverage claim survives full-corpus measurement.
No code changes; 993 tests still passing.
Three Explore agents fanned out in parallel for a docs/ + diagrams/
+ code-comment audit against the shipped state of the three
preflight tickets. This commit lands all the alignment fixes.
Core docs updates:
CLAUDE.md
- dag.py module description: stage counts now read
"7/8 quote · 9/10 CTI · 3 reject" reflecting #000009 preflight
stage + reject-broad early-return shape.
docs/cti-architecture.md §2.2 + §2.3
- §2.3 Merkle-AGI-DAG section rewritten: documents all five DAG
shapes (legacy 7/9, post-#000009 8/10, reject-broad 3),
describes the preflight stage's 5 nested CTI clauses
(classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs), pins
PREFLIGHT_NODE_VERSION = "preflight-node-v1", states the
audit-replay payoff.
- §2.2 CTI section: adds the four new modules
(quantifier, model_profiles, quantifier_reminder,
metacognition) as code anchors. Notes that pre-answer
preflight contract extends CTI upstream of retrieval.
docs/seven-point-program.md
- D3 status ½ → ¾ — pre-answer preflight contract landed via
#000008 + #000010. Code anchors + pinning tests updated.
- D4 status ½ → ¾ — preflight stage adds upstream control
commitment to the run-DAG. Code anchors include
build_reject_run_dag + preflight_node_hash.
- Status snapshot table: tickets column now references
#000008/#000009/#000010 against D1/D3/D4 directives.
- "Post-landing addendum (2026-05-03 / 2026-05-04)" subsection
summarises all three tickets + their commit shas + final
test count (993 passing, up from 734).
docs/modules.md
- Q&A pipeline table: added 4 new modules (quantifier.py,
model_profiles.py, quantifier_reminder.py, metacognition.py).
dag.py row updated to "7/8 quote · 9/10 CTI · 3 reject".
- dag.py subsection rewritten: documents all 5 DAG shapes,
describes the preflight payload's 5 clauses + question_state.
- 4 new module subsections (quantifier / model_profiles /
quantifier_reminder / metacognition) explaining each
module's purpose, signature, and how it feeds the run-DAG
preflight clause.
Diagram updates:
docs/diagrams/query-pipeline.dot + .svg
- New "PREFLIGHT (#000008 + #000010)" node inserted between
cache_check and concepts_lookup.
- New "REJECT-BROAD" node showing the 3-stage minimal DAG
escape path.
- render node label extended with the audit-line tail token
catalog.
docs/diagrams/aborist-modules.dot + .svg
- 4 new qa_* nodes in the retrieval & verifier cluster.
- 8 new edges: qa_query/qa_runner each call into all 4
preflight modules; qa_dag has dotted edges to qa_quantifier
+ qa_metacognition (preflight clause sources).
- qa_dag label updated to mention preflight_node_hash + 5 clauses.
docs/diagrams/verifier-ladder.dot + .svg
- Soft-demote violations list extended: BROAD_QUANTIFIER_RUNAWAY
/ CAP_APPLIED / SCOPE_UNBOUND, FORMAT_COLLAPSED, BARE_NAME_CLAIM.
- New "AUDIT-LINE TAILS" annotation node listing all 11 tail
tokens (#000008 broad-* + #000010 metacog + classic verifier).
- Dashed edges from each rung to tails note showing tails
compose onto labels.
Code-side stale-comment fixes (caught by 3rd Explore agent):
aborist/qa/keys.py:218
- "The four fields" → "The seven fields"; mention #000010 adds
six more for metacognition.
aborist/qa/query.py:2644
- 7-stage / 9-stage comment expanded to enumerate all four
base+preflight shapes plus the 3-stage reject path.
aborist/qa/runner.py:835
- same expansion as query.py for runner.ask() callsite.
mesh-*.dot, ingest-pipeline.dot, qa-modes-bench.md, bench-maxing.md,
bench-emergent-design.md, verifier-semantic-gap-design.md,
self-reference-design.md, concept-relations-design.md confirmed
orthogonal — no edits needed.
993 tests still passing (no behavior change). 7 files modified
across docs/ + 3 dot diagrams + 3 SVGs + 4 code-comment fixes.
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
#000004 — Directive coverage in bench summary. Filed as closed
(landed in commit acd1f9c) for design-log traceability. Captures
the v1 substrate: per-row directive_compliance helper, _summarize
aggregation, markdown directive-coverage section, 5 unit tests.
#000005 — Label ladder migration (POINTER-LINKED / ANCHOR-WARRANTED
/ EVIDENCE-WARRANTED / ENTAILMENT-VERIFIED). Status open. Migration
from today's two-rung EVIDENCE-LINKED to a four-rung ladder where
each rung names a strictly stronger property. Schema column stays
{STRICT, HYBRID, UNGROUNDED} so v9.8 cache_key invariants hold;
renderer-level mapping only. Reserves ENTAILMENT-VERIFIED for a
future committed entailment engine without forcing premature
implementation.
TICKETS.md index gains both rows. Next ID bumps to 000006.
seven-point-program.md status snapshot updated:
- D7 row links #000005 (ladder migration proposal).
- D8 row links #000004 (closed substrate).
- New "Anti-regression test layer" section maps each directive
to its structural pin in tests/test_directives.py.
Note on fox's roadmap item #000006 (Internal CTI Expansion / frame
lattice / multi-frame answers): scope overlaps with #000002
(Reference-Frame Polarity Contract / Module L). NOT opening as
a separate ticket — Module L already covers it.
The 2026-05-01 distilled architectural directive becomes the
audit lens applied before every new ticket / feature / prompt
edit. Each of the seven directives gets:
- status (✓ / ½ / ✗)
- code anchors (verifier paths, renderer hooks)
- pinning tests (per-rule coverage in tests/)
- tickets covering remaining work
- bench signal (which row column reports it)
Status snapshot at landing:
D1 (no LLM in verifier) ✓
D2 (pointer clauses) ✓
D3 (CTI internally) ½ → ticket #000002
D4 (retrieval map AND evidence map) ½ → ticket #000001
D5 (deterministic pointer verify) ✓
D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair)
D7 (honest labels) ✓
D8 (test-pinning before automation) discipline
CLAUDE.md docs index gains a "North-star" section pointing at the
program; ticket entries gain a "Directive" column threading them
back to the program.
