Bench-emergent stress test ran another 100 cycles under the
post-#000008/9/10/11 substrate. Total accumulated: 300 cycles.
Verdict distribution shift on last 100 vs 134-cycle baseline:
STRICT 5% (7/134) → 0% (0/100) -5pp
HYBRID 22% (29/134) → 16% (16/100) -6pp
UNGROUNDED 73% (98/134) → 84% (84/100) +11pp
Zero false-positive STRICTs across 100 random-word triplets.
The 5pp drop in STRICT-rate isn't a regression — it's the
verifier ladder + new preflight contracts doing their job.
Random-word triplets are genuinely ungrounded for the most
part; the prior 5% STRICT rate included false-positives that
the post-hardening verifier now catches.
Violation profile (last 100 cycles, claim_lattice JSON):
CITATION_MISMATCH: 86 dominant gate
TOO_MANY_EVIDENCE_IDS: 24
SUBJECT_TOKENS_ABSENT: 12 Rule 9 firing on parroting
DEFLECTION_DETECTED: 12
TITLE_MISMATCH: 10
...
metaphor_deflection fires 6/100 — still rare. Item 3
(calibration) is now closer to sample-size threshold (~30
signals across 300 cycles; needs ~50-100 to calibrate).
No new tuning candidates surface. Original three remain at
their resolution states.
Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.
#000025 — Metacog test fixture expansion:
bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
covering edge cases per detector kind: temporal (4 cases),
contradiction (4), false-premise (5), out-of-corpus (3), multi-
trigger (2), well-formed controls (5). Documents two known
detector ceilings: Q11 over-fires on past-tense factoid
("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
NASA-fake) miss false premises that lack a presupposition
pattern match. Fixture now serves as long-term regression suite.
#000026 — --show-preflight full clause render:
build_run_dag() and build_reject_run_dag() gain optional
preflight_payload kwarg. When supplied, the canonical 5-clause
CTI payload (classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs + question_state + node_version)
persists alongside the leaf hash in run_dag_blob.
aborist providence --show-preflight CACHE_KEY now renders the
full payload + verifies the persisted hash matches the
recomputed canonical hash (audit-replay tamper detection).
Legacy rows fall through cleanly: payload_hash_check reports
"unavailable: legacy row predates preflight_payload persistence".
#000027 — Latency profile:
Microbenched preflight: 0.46ms/question (negligible). Single
fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
33-35s in Addendum 3 was vLLM concurrency contention at c=4
(per qa-modes-bench.md saturation note), not substrate
overhead. Added preflight_ms + soft_preflight_ms to timings
dict for explicit confirmation in future cycles.
#000028 — Auto-quality-check sweep revival:
scripts/bench_emergent.py running with EMERGENT_N=100 in
background (PID 125680). Will accumulate cycles into
bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
Async — not blocking on completion.
#000029 — #000011 SOFT_PREFLIGHT_HINT implementation:
aborist/qa/soft_preflight.py — new module. SoftPreflightHint
dataclass + soft_preflight_question() pure function. 9
canonical labels mapping to soft analogues of #000010 hard
detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
Constrained-generation prompt (max_tokens=128, temp=0.0) asks
the model to pick ONE label + one-line rationale.
Fail-closed across every parse path:
- chat_client raises → SOFT_PARSE_FAIL
- response unparseable → SOFT_PARSE_FAIL
- label outside enum → SOFT_PARSE_FAIL
Sidecar enforces SOFT_ prefix at the normalize step so a
model that drops the prefix still gets caught.
Wired into query() between preflight & retrieval. Default
OFF (`soft_preflight_enabled: False`). NOT folded into
_VERIFIER_POLICY_FIELDS — soft hints don't gate cache
identity (#000011 §4). Audit-line tail renders as
"· soft: <label>" (e.g. "· soft: time sensitive") so the
signal is visually distinct from hard tails.
--soft-preflight CLI flag opts in per-call. End-to-end
live-verified on "When did Mr. Burns become Homer's biological
father?" — produces:
EVIDENCE-WARRANTED · via claim_lattice
· false premise · soft: time sensitive
1/1 16.4s
Hard `· false premise` (from #000010 deterministic detector)
composed with soft `· soft: time sensitive` (from #000011
sidecar). The model classified a different shape than the hard
detector — by design; soft hints are independent advisory
signals, not redundant with the hard layer.
25 new tests pin: default-OFF behavior, parse-failure modes,
label normalization (SOFT_ prefix enforced), all 8 actionable
labels round-trip, fail-closed on client exceptions, dataclass
JSON round-trip, rationale-length cap.
Other:
- #000010 §13.3 documents 2/5 metacog-trigger questions return
STRICT despite hard-detector warning — direct empirical
motivation for #000011 design.
- tests/test_dag.py extends with 3 _extract_preflight_hash_*
helper tests (cleaning #000009 §7.2 unfinished state).
- bench/emergent_log.jsonl adds new cycles from background run.
#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
Fan-out execution of the deferred-but-not-blocking pile from
prior status reports.
#000009 §7.2 — bench harness preflight_hash field:
- aborist/qa/query.py surfaces `preflight_hash` on result dict
(miss path, reject path, and cache-hit path via new helper
`_extract_preflight_hash_from_blob` that pulls the stage hash
out of persisted run_dag_blob).
- bench/qa_sweep.py adds 12-char preflight_hash prefix to bench
rows. Mirrors cache_key truncation pattern. Operators can
grep / SQL-filter bench JSONL by preflight policy state.
- 3 new tests in tests/test_dag.py for the extract helper.
#000009 §7.2 — `aborist providence --show-preflight CACHE_KEY_PREFIX`:
- New CLI flag pulls the preflight stage payload from a row's
run_dag_blob. Match by 12-char prefix. Renders preflight stage
hash + run-DAG stage list. Operator tool for inspecting which
policy state governed a cached row.
- Live verified on a real cache row (8a212fecb2a9 — current CEO
of OpenAI question, 10-stage CTI shape with preflight at idx 1).
- Legacy rows (predating #000009) report a clean fall-through
message: "run_dag has no preflight stage (predates #000009)".
#000010 §12.6 — reference-frame plumbing into QuestionState:
- Pre-retrieval preflight runs with reference_frames=()
(frame_detection needs source titles, not available yet).
Post-retrieval, query.py re-runs preflight_question() with
the detected frames so the result-dict + run-DAG QuestionState
carry frame-aware logical_statuses (specifically
`reference_frame_ambiguous` when 2+ frames match).
- Live verified on Orwell-style question; logical_statuses now
correctly includes `reference_frame_ambiguous` in the result.
Metacog-trigger bench fixture (#000010 §13.3):
- bench/qa_questions_metacog_subset.txt — 6 questions, one per
detector kind plus a well-formed control.
- Bench artifact 2026-05-04T02-18-42Z. Detector accuracy 6/6
on fixture; 2 of 5 trigger questions return STRICT on lattice
mode despite metacog warning (JSON STRICT on
George-Washington-stop-being-president-of-France false-premise
+ uploaded-contract out-of-corpus questions). Audit-line tails
correctly surface the warnings.
- qa-modes-bench.md Addendum 4 captures the per-question matrix
+ interpretation. #000010 §13.3 cross-references with bench
artifact stamp.
#000011 SOFT_PREFLIGHT_HINT design ticket opened:
- docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md
captures the design proposal per #000010 §18 / source doc.
Implementation deferred — design only.
- Sidecar would add model-assisted preflight as a soft signal
(`SOFT_FALSE_PREMISE_SUSPECTED` etc.) that NEVER enters the
verifier hard path. Strict guardrail: cannot create
PREFLIGHT_OK or PREFLIGHT_BLOCKED without deterministic
support.
- Validated by §13.3 finding: deterministic detectors flag
correctly; corpus-accidental grounding produces 2/5 STRICT
on trigger questions; soft sidecar would add independent
semantic skepticism.
- TICKETS.md index row added; Next ID bumped to 000012.
996 tests passing (3 new for the extract helper).
Cross-doc consistency:
- qa-modes-bench.md Addenda 1+2+3+4 chronological
- #000010 §13.1 (broad subset) + §13.2 (full bench) + §13.3
(metacog trigger subset)
- #000011 design captured but not implemented
Cross-references the 2026-05-03T23-30-12Z full 75-question
regression bench (Addendum 3 in qa-modes-bench.md) into:
- #000010 §13.2 — full-bench validation alongside the §13.1
broad-subset validation. Same verdict: defaults stay on.
Substrate-level wins beyond noise-bounded SR deltas:
pointer FORMAT_COLLAPSED: 0/225 (eliminated globally)
pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only)
JSON mean ratio: +3pp
- seven-point-program.md addendum — bench-validation paragraph
after the test-count line. Names the SR deltas + the
substrate-level wins; concrete evidence the post-landing
directive coverage claim survives full-corpus measurement.
No code changes; 993 tests still passing.
Addendum 3 to qa-modes-bench.md. Validates that flipping
quantifier_reminder_enabled=True for lattice modes (per #000010
§12.10 / §13.1) doesn't regress narrow-question performance.
Prior validation (Addendum 2) covered the 9-question broad subset
only. This run sweeps the full 75-question bench/qa_questions.txt
(~10% broad, ~89% narrow), 225 runs per mode, comparing against
the frozen 2026-05-02T15-07Z baseline.
Findings:
Mode | Pre-flip SR | Post-flip SR | Δ
----------+-------------+--------------+--------
quote | 0.54 | 0.52 | -2pp (within 5pp floor)
pointer | 0.20 | 0.21 | +1pp
JSON | 0.42 | 0.44 | +2pp
Mean ratio: -1/+2/+3pp — all within noise band.
Substrate-level wins beyond the headline metrics:
- pointer FORMAT_COLLAPSED: 0/225 across the full sweep.
Reminder eliminates collapse mode globally, not just on broad.
- pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when
reminder was off). Citation discipline propagates beyond the
rows where the reminder text actually fires — the model's
session attention reinforces.
- JSON mean ratio +3pp consistent with broad-subset finding.
Quote-mode is essentially unchanged because it's mode-gated off
the guard by default.
Latency 33-35s/call this evening (vs 17-19s in prior runs) is
endpoint contention, not a substrate regression — preflight adds
zero LLM calls.
Verdict: no regression. The default flip ships clean across the
full corpus. Defaults stay on. The substrate is strictly more
honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without
sacrificing throughput on non-broad questions.
Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}.
Post-landing validation cell. Same 9-question broad subset as
§12.6 reminder-only baseline; this run flips both
metacognition_enabled and quantifier_reminder_enabled to False so
we can isolate the preflight contribution.
Metric | OFF | ON | Δ
--------------------------+-----------+----------+----------
pointer mean ratio | 0.483 | 0.643 | +16pp
JSON mean ratio | 0.570 | 0.735 | +17pp
JSON UNGROUNDED rate | 7/27 | 1/27 | -22pp
pointer FORMAT_COLLAPSED | 2/27 | 0/27 | -100%
any-mode STRICT-rate | within ±11pp noise (27-sample n=3 floor)
Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all clear the
5pp signal floor on lattice modes. STRICT-rate moves are within
Hermes nondeterminism. The #000010 default flip is doing what
was claimed.
On this subset none of the metacognition detectors fire (no
temporal / contradiction / false-premise / out-of-corpus shapes),
so the delta effectively isolates the reminder contribution from
#000008. A metacog-trigger subset bench is deferred.
Documentation:
- docs/qa-modes-bench.md Addendum 2 captures the comparison
table + interpretation + verdict.
- docs/tickets/ticket-000010-... §13.1 cross-references with
bench artifact stamp.
Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}.
Three Explore agents fanned out in parallel for a docs/ + diagrams/
+ code-comment audit against the shipped state of the three
preflight tickets. This commit lands all the alignment fixes.
Core docs updates:
CLAUDE.md
- dag.py module description: stage counts now read
"7/8 quote · 9/10 CTI · 3 reject" reflecting #000009 preflight
stage + reject-broad early-return shape.
docs/cti-architecture.md §2.2 + §2.3
- §2.3 Merkle-AGI-DAG section rewritten: documents all five DAG
shapes (legacy 7/9, post-#000009 8/10, reject-broad 3),
describes the preflight stage's 5 nested CTI clauses
(classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs), pins
PREFLIGHT_NODE_VERSION = "preflight-node-v1", states the
audit-replay payoff.
- §2.2 CTI section: adds the four new modules
(quantifier, model_profiles, quantifier_reminder,
metacognition) as code anchors. Notes that pre-answer
preflight contract extends CTI upstream of retrieval.
docs/seven-point-program.md
- D3 status ½ → ¾ — pre-answer preflight contract landed via
#000008 + #000010. Code anchors + pinning tests updated.
- D4 status ½ → ¾ — preflight stage adds upstream control
commitment to the run-DAG. Code anchors include
build_reject_run_dag + preflight_node_hash.
- Status snapshot table: tickets column now references
#000008/#000009/#000010 against D1/D3/D4 directives.
- "Post-landing addendum (2026-05-03 / 2026-05-04)" subsection
summarises all three tickets + their commit shas + final
test count (993 passing, up from 734).
docs/modules.md
- Q&A pipeline table: added 4 new modules (quantifier.py,
model_profiles.py, quantifier_reminder.py, metacognition.py).
dag.py row updated to "7/8 quote · 9/10 CTI · 3 reject".
- dag.py subsection rewritten: documents all 5 DAG shapes,
describes the preflight payload's 5 clauses + question_state.
- 4 new module subsections (quantifier / model_profiles /
quantifier_reminder / metacognition) explaining each
module's purpose, signature, and how it feeds the run-DAG
preflight clause.
Diagram updates:
docs/diagrams/query-pipeline.dot + .svg
- New "PREFLIGHT (#000008 + #000010)" node inserted between
cache_check and concepts_lookup.
- New "REJECT-BROAD" node showing the 3-stage minimal DAG
escape path.
- render node label extended with the audit-line tail token
catalog.
docs/diagrams/aborist-modules.dot + .svg
- 4 new qa_* nodes in the retrieval & verifier cluster.
- 8 new edges: qa_query/qa_runner each call into all 4
preflight modules; qa_dag has dotted edges to qa_quantifier
+ qa_metacognition (preflight clause sources).
- qa_dag label updated to mention preflight_node_hash + 5 clauses.
docs/diagrams/verifier-ladder.dot + .svg
- Soft-demote violations list extended: BROAD_QUANTIFIER_RUNAWAY
/ CAP_APPLIED / SCOPE_UNBOUND, FORMAT_COLLAPSED, BARE_NAME_CLAIM.
- New "AUDIT-LINE TAILS" annotation node listing all 11 tail
tokens (#000008 broad-* + #000010 metacog + classic verifier).
- Dashed edges from each rung to tails note showing tails
compose onto labels.
Code-side stale-comment fixes (caught by 3rd Explore agent):
aborist/qa/keys.py:218
- "The four fields" → "The seven fields"; mention #000010 adds
six more for metacognition.
aborist/qa/query.py:2644
- 7-stage / 9-stage comment expanded to enumerate all four
base+preflight shapes plus the 3-stage reject path.
aborist/qa/runner.py:835
- same expansion as query.py for runner.ask() callsite.
mesh-*.dot, ingest-pipeline.dot, qa-modes-bench.md, bench-maxing.md,
bench-emergent-design.md, verifier-semantic-gap-design.md,
self-reference-design.md, concept-relations-design.md confirmed
orthogonal — no edits needed.
993 tests still passing (no behavior change). 7 files modified
across docs/ + 3 dot diagrams + 3 SVGs + 4 code-comment fixes.
Architectural feedback at ~/Downloads/RESPONSE-ticket-000009-... .txt
(2026-05-04) flagged five gaps in the c36e85c landing. Most
critical: reject-broad early-return path emitted no run_dag_blob,
so audit replay couldn't see that a rejection happened (let alone
under what policy state).
A — reject-path DAG (the critical gap):
aborist/qa/dag.py: build_reject_run_dag() — 3-stage minimal DAG
question → preflight → final_label. final_label payload carries
rejection_reason + answer_text_hash so two rejections under
different policy state produce different roots.
query.py reject path now wires it: returns run_dag_root +
run_dag_blob on the rejection result dict. Live-verified end-
to-end on `make query Q="winners of all major sports?"
REJECT_BROAD=1 BURN=1`.
Audit replay rule: 3 stages always means reject path. Operators
can read the stage list and tell instantly without parsing the
payload.
B — nested CTI clauses:
preflight_node_hash() payload restructured from flat 3-key to
nested 5-clause:
classifier — quantifier classifier output (#000008)
answer_contract — guard / cap / reject / metacog state (per-run)
prompt_contract — reminder enabled / injected / template_id
evidence_contract — exposure budget, line discipline
policy_refs — governance_policy_hash, model_profile_hash,
answer_mode (reference, not raw policy)
Plus question_state (metacog) as its own clause and top-level
stage + node_version. Single DAG stage; nested clauses inside
for diff legibility (feedback §3).
C — node_version field:
PREFLIGHT_NODE_VERSION = "preflight-node-v1" pinned in the
payload so legacy runs without the node can be unambiguously
labeled `unavailable_legacy_run` by audit tools (feedback §9).
D — reference hashes only:
policy_refs uses governance_policy_hash + model_profile_hash
rather than bundling raw policy booleans. Avoids
double-committing already-hashed state (feedback §4).
E — reminder_template_id:
prompt_contract.reminder_template_id = "broad-quantifier-bounded-v1"
or "broad-quantifier-unbounded-v1" depending on scope_bound_hint,
populated only when reminder actually fires.
F — stage name kept as `preflight` (not `quantifier_preflight`):
Node carries both #000008 quantifier AND #000010 metacognition
payloads. node_version disambiguates schema for audit tools.
G — docs/cti-architecture.md update deferred to a small follow-up.
Bug fixes:
- free-variable shadowing on verifier_policy_hash /
model_profile_hash / question_hash — local re-imports inside
the reject branch shadowed module-top imports used elsewhere
in query() / runner(); now use the module-top names.
- reject path question_hash signature: takes `mode=` not
`dedup_mode=` — fixed in the reject DAG builder caller.
Hash compatibility:
Rows written between c36e85c and this commit have hash payloads
matching the OLD flat 3-key shape. The persisted run_dag_blob
captures the actual payload that was hashed, so those rows
still verify via verify_run_dag(). New rows use the nested
5-clause shape.
7 new tests in tests/test_dag.py:
- hash sensitivity to answer_contract / prompt_contract /
policy_refs flips (audit-replay payoff demonstrations)
- PREFLIGHT_NODE_VERSION pinning
- reject DAG: 3-stage shape, root changes with preflight hash,
round-trips through verify_run_dag
993 tests passing (6 net new); 36 skipped.
Live verification:
make query Q="winners of all major sports?" REJECT_BROAD=1 BURN=1
→ status=broad_quantifier_rejected, run_dag_root populated,
blob carries 3-stage shape.
make query Q="winners of all major sports?" BURN=1
→ 10-stage shape preserved (question → preflight → retrieval
→ ... → final_label).
Ticket #000009 status: closed · re-landed 2026-05-04 with §8
corrections.
Closes ticket #000009 zero-shot. Scope expanded to cover BOTH
ticket #000008 (broad-quantifier) AND ticket #000010
(meta-cognition) preflight contracts in a single combined node —
both share the same audit-replay gap and inserting two separate
nodes between question and retrieval was operationally awkward.
aborist/qa/dag.py:
+ preflight_node_hash() — combines QuestionState +
quantifier classifier output + behavioral policy_state into
one canonical SHA-256 hex.
+ build_run_dag() gains optional preflight_hash parameter.
When supplied, inserts {"stage": "preflight", "hash": ...}
at position 1 (between question and retrieval).
Backward-compat: None → original 7/9-stage shapes preserved
for legacy run_dag_root re-validation.
Quote-mode: 7 → 8 stages with preflight.
Pointer-mode CTI: 9 → 10 stages with preflight.
aborist/qa/query.py + runner.py:
Both build the preflight payload from question_state +
quantifier dict + 10-field policy_state (guard_enabled,
guard_apply_caps, guard_apply_caps_mode_gated,
claim_cap_resolved, claim_cap_actually_applied,
reminder_enabled, reminder_eligible, reject_broad_active,
metacognition_enabled, block_on_contradiction).
This means two cache rows that share the same question + same
model output + same verifier verdict but DIFFERENT preflight
policy state now produce different run_dag_root values. Audit
replay can pin the policy decision per row.
9 new tests in tests/test_dag.py:
- preflight_node_hash determinism
- hash bumps on question_state change
- hash bumps on policy_state change (the audit-replay payoff)
- all-None defensive shape
- 7→8 stage transition (quote mode)
- 9→10 stage transition (pointer mode), preflight at index 1
- run_dag_root bumps when preflight_hash bumps
- verify_run_dag round-trips through preflight stage
Live verification: latest providence_cache row carries
['question', 'preflight', 'retrieval', 'evidence_map', 'prompt',
'raw_answer', 'parsed_claim_lattice', 'verify', 'render',
'final_label'] — preflight stage living in the persisted DAG.
987 tests passing (9 new); 36 skipped.
Tickets:
#000009 status: closed · landed 2026-05-03 (zero-shot)
#000010 cross-ref updated: "DAG binding shipped via #000009"
What's NOT in this ticket (logged in §7.2):
- CLI flag for inspecting preflight node from cache_key
- Bench harness preflight_hash field for cross-row comparison
- SOFT_PREFLIGHT_HINT (model-assisted preflight sidecar)
Closes ticket #000010 (Meta-Cognition Preflight Guard). Mechanism
complete; defaults preserve the dry-run discipline pattern from
#000008.
Phase 2 — wire preflight into query() and runner.ask():
- preflight_question() runs after policy resolution + quantifier
classification, before retrieval.
- QuestionState surfaces on miss path, cache-hit path, AND
reject-broad early-return path of query() — schema column-
aligned across all four returns.
- runner.ask() carries the same fields for `aborist ask` parity.
Phase 3 — policy fields + governance hash + CLI flags:
- 6 new policy fields, all default-on except
metacognition_block_on_contradiction (default False — label-
only by default; opt-in via --block-on-contradiction).
- All 6 folded into _VERIFIER_POLICY_FIELDS so flipping any
invalidates prior cache records on lookup.
- 2 new CLI flags on `aborist query`:
--no-preflight Level 2 master kill
--block-on-contradiction strict mode (hard-block on
lexical contradictions)
Phase 4 — audit-line labels + bench fields + tests:
- _render_warrant_tail extended with 5 metacog tail tokens:
· false premise
· contradictory
· stale risk
· out of corpus
· frame ambiguous
- Bench rows in qa_sweep.py gain 7 new bounded-size projection
fields (logical_statuses, question_shape, preflight_result,
temporal_sensitivity, has_false_premise, has_contradiction,
corpus_requirement). Full QuestionState stays on result dict
for CLI render only.
- tests/test_metacognition.py grew from 42 → 68 tests
(16 new: 6 governance + 6 audit-line tail + 4 default-policy
pinning).
Live verified end-to-end:
$ make query-dry Q="Who is the current CEO of OpenAI?" BURN=1
UNGROUNDED · via claim_lattice · stale risk
$ make query-dry Q="When did Mr. Burns become Homer's biological
father?" BURN=1
UNGROUNDED · via claim_lattice · false premise
978 tests passing; 36 skipped.
What's NOT shipped (deferred):
- Run-DAG node binding for metacognition_preflight stage —
joins ticket #000009 Phase 5 (same audit-replay gap; both
nodes can land together).
- Reference-frame plumbing — frame_detection runs post-retrieval,
preflight here is pre-retrieval; deferred until two-pass
or post-classification update lands.
- SOFT_PREFLIGHT_HINT (model-assisted sidecar) — source doc §18
reserves this label; hard rule preserved (no LLM in preflight
hard path).
- Bench A/B measuring preflight on vs off — quick to run once
stack settles.
Ticket #000010 status: closed · landed 2026-05-03.
Implements the Meta-Cognition Preflight Guard (M0 / MCTL) per
fox's directive at ~/Downloads/meta-cognition_for_hermes(1).txt
(2026-05-03).
aborist/qa/metacognition.py:
- QuestionState dataclass (frozen, JSON-serializable via to_dict)
- preflight_question() pure function: classifies a question
deterministically into a QuestionState before generation
- 4 new detectors:
detect_temporal_sensitivity() — current/latest/today/CEO/etc.
detect_contradiction() — lexical pairs (unmarried+spouse,
always+never, alive+dead, etc.)
detect_false_premise() — presupposition patterns:
when did X stop/become Y,
why did X cause Y,
how did X become Y
detect_out_of_corpus() — my-uploaded-X / file-I-sent shapes
- Reuses #000008 quantifier classifier (no duplication)
- Composes 8 LogicalStatus values:
well_formed, under_specified, false_premise_suspected,
contradictory_question, out_of_corpus_risk, stale_risk,
reference_frame_ambiguous, broad_quantifier_unbounded
- Three preflight results: PREFLIGHT_OK / _PARTIAL / _BLOCKED
- Per-detector enable switches in policy:
metacognition_enabled (master kill)
metacognition_temporal_check
metacognition_contradiction_check
metacognition_false_premise_check
metacognition_out_of_corpus_check
metacognition_block_on_contradiction (default False — label
only by default; opt-in
to hard-block)
- preflight_policy_hash for governance binding (Phase 3)
- PREFLIGHT_VERSION = "metacognition-v0.1"
Hard rule (D1): no LLM in this hard path. Pure regex + lexical
matching. Model-assisted preflight, if added later, labels itself
SOFT_PREFLIGHT_HINT (not implemented in this phase).
42 new tests cover the seven test cases from source doc §14
(false-premise, contradictory, broad-quantifier, reference-frame,
time-sensitive, out-of-corpus, model-cutoff) plus per-detector
unit tests, gating (master kill, per-detector disable,
block-on-contradiction opt-in), determinism (question_hash
stable, policy_hash bumps on flip), and serialization.
Ticket #000010 opened with status `open · in progress
(zero-shot 2026-05-03)`. TICKETS.md index updated; Next ID bumped
to 000011.
Phases 2-4 still queued (wire into query/runner, policy fields +
governance, audit-line labels + bench fields).
962 tests passing (42 new); 36 skipped.
Closes#000008 with status `closed · landed in 4f2b5a6` per the
docs/TICKETS.md convention. The preflight guard mechanism + bench
cycle + default flip all shipped 2026-05-03; the design log stays
in place.
Opens #000009 — Quantifier preflight run-DAG node binding. Splits
the Phase 5 follow-up out of #000008 §11.11 into its own ticket.
Scope: bind the classifier output + policy decision into
`run_dag_root` so audit replay can distinguish guard-on vs guard-off,
cap-applied vs not, reminder-injected vs skipped. Currently those
appear on the result dict but are NOT in the run-DAG hash. Required
to close the audit-replay gap that blocks the §9.5 Merkle-AGI-DAG
framing from fully holding. Estimated 3-4h. Awaiting go/no-go.
Makefile shortcuts for the #000008 CLI flags (operator ergonomics):
BROAD=1 → --apply-quantifier-caps (flip cap apply-gate)
REJECT_BROAD=1 → --reject-broad (preflight rejection)
ALLOW_BROAD=1 → --allow-broad (emergent search)
Available on both `make query` and `make query-dry`. Default
behavior unchanged: ANSWER_MODE=claim_lattice (JSON), reminder ON
for lattice modes, cap operator-opt-in.
Smoke-tested:
$ make query-dry Q="winners of all major sports?" BROAD=1
→ cap applies on JSON; classifier reports ALL/unbounded
$ make query-dry Q="winners of all major sports?" REJECT_BROAD=1
→ preflight rejection, exit-1 (consistent with UNGROUNDED)
TICKETS.md index:
#000008 closed · landed in `4f2b5a6`
#000009 open · awaiting go/no-go (D3, D4)
Next ID bumped 000009 → 000010
n=5 verification of cap+reminder cell (135 runs):
Metric | n=3 | n=5
--------------------+-----------+------------
JSON SR | 0.30 | 0.33 ← matches cap-only
JSON UNGROUNDED rate| 1/27 (4%) | 2/45 (4%) ← matches reminder-only
pointer SR | 0/27 | 0/45 ← unchanged across all cells
The §12.8 0.30 was Hermes nondeterminism. n=5 confirms cap+reminder
delivers cap-only's STRICT-rate AND reminder-only's UNGROUNDED-rescue.
§10.8 strict gate met at n=5:
vs reminder-only on JSON SR: +11pp (clears floor)
vs cap-only on JSON UNGROUNDED: -18pp (clears floor)
vs cap-only on ptr mean ratio: +12pp (clears floor)
Defaults flipped — Option A landing (per-mode tailored):
quantifier_reminder_enabled False → True
(load-bearing on both lattice modes)
NEW field: quantifier_apply_caps_modes = ["claim_lattice"]
(allowlist for which modes apply caps
when apply_caps=True; JSON-only since
cap-on-pointer is wasted noise per
§12.10 0/45 STRICT data)
quantifier_guard_apply_caps False → False (UNCHANGED)
(operator opts in via
--apply-quantifier-caps; preserves
§10.11.3 dry-run discipline)
Cap-application gate now reads:
if apply_caps AND mode in apply_caps_modes AND cap is not None:
effective_max_claims = looked_up_cap
quantifier_apply_caps_modes folded into _VERIFIER_POLICY_FIELDS so
flipping the allowlist invalidates prior cache records.
5 new tests pin: reminder default ON for both runner.DEFAULT_POLICY
and query.DEFAULT_QUERY_POLICY; apply_caps_modes default
["claim_lattice"]; governance-hash invalidation on allowlist flip;
apply_caps default still False (dry-run preserved).
920 tests passing (5 new); 36 skipped.
Operator behavior:
$ aborist query "winners of all major sports?"
→ reminder ON, cap OFF (default after this commit)
$ aborist query --apply-quantifier-caps "..."
→ cap applies on claim_lattice (JSON) only
$ aborist query --apply-quantifier-caps \
--policy quantifier_apply_caps_modes='["claim_lattice","claim_lattice_pointer"]' "..."
→ Option D for one call
Phase 5 (run-DAG node binding for quantifier_preflight) and
cross-model Qwen/GPT-4 verification remain as follow-ups per §11.11.
#000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs):
Metric | Base | Rem | Cap | Cap+Rem
--------------------+------+------+------+--------
JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR
JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue
pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best
pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven
§10.8 strict gate "compound beats either alone by ≥5pp on every metric"
NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR
(clears floor) and beats cap-only by +17pp on pointer mean ratio
(clears floor), but is -3pp vs cap-only on JSON SR (regression,
within noise).
#000008 §12.9 — Final verdict + recommendation:
Mechanism asymmetry (clean signal):
Reminder rescues UNGROUNDED → HYBRID
Cap rescues HYBRID → STRICT
Recommendation: Option A — single-knob defaults, per-mode tailored:
claim_lattice (JSON): apply_caps=True + reminder=True
claim_lattice_pointer: apply_caps=False + reminder=True
(cap can't rescue pointer-tag discipline
upstream of cap; cap fires 20× without
verdict gain)
quote: guard mode-gated off (already default)
Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5
verification on cap+reminder before flipping defaults.
Cross-doc updates:
- CLAUDE.md: architecture diagram now lists the three new Phase 1-3
modules (quantifier.py, model_profiles.py, quantifier_reminder.py).
New "Broad-quantifier preflight guard" conventions entry covers
the 7 policy fields, six-level disable hierarchy, dry-run
defaults, CLI flags, and §12 bench summary.
- docs/qa-modes-bench.md: addendum at end pointing at #000008 §12
four-cell A/B + 4 bench artifact stamps. Original 2026-05-02
journal frozen; 2026-05-03 broad-subset findings flagged as
question-mix-dependent narrative on top of the global
per-mode recommendation.
§12.7 captures the 2026-05-03T12-47-23Z cap-only A/B (apply_caps=
True, reminder=False) on the 9-question broad subset.
Mode | Baseline | Reminder | Cap-only
----------+----------+----------+----------
JSON SR | 0.19 | 0.22 | 0.33 ← +14pp
JSON U | 7 | 1 | 6 ← reminder dominates
JSON S | 5 | 6 | 9 ← cap dominates
ptr ratio | 0.473 | 0.643 | 0.516
JSON ratio| 0.524 | 0.735 | 0.643
§10.8 gate MET on JSON mode (+14pp STRICT-rate).
Headline insight: cap and reminder help in DIFFERENT ways.
- Reminder rescues UNGROUNDED → HYBRID (restates citation rule).
- Cap rescues HYBRID → STRICT (forces fewer-but-better claims).
The two are complementary, not redundant. If §12.8 (cap+reminder)
confirms the compound effect, that's the §10.8 trigger to land
Option D as default. Predicted JSON SR ~0.40 if effects compound.
Pointer mode still 0/27 STRICT under cap-only — TOO_MANY_CLAIMS
fires 20× (vs 7× baseline) but pointer-tag failures upstream of
the cap still gate the verdict.
Single-source-of-truth section for what was actually built in the
2026-05-03 implementation pass. Complements §8 (commit table) and
§9.6 (per-phase notes) with a full inventory:
- §11.1 New modules: aborist/qa/{quantifier,model_profiles,quantifier_reminder}.py
- §11.2 Modified modules: query.py, runner.py, keys.py, cli.py, qa_sweep.py
- §11.3 New test files: 4 new + 1 extended; 120 new tests total
- §11.4 Seven new policy fields in _VERIFIER_POLICY_FIELDS
- §11.5 Four new CLI flags on `aborist query`
- §11.6 Four new violation kinds (3 soft + 1 hard) with audit tails
- §11.7 New result-dict fields on miss + cache-hit paths
- §11.8 Eight new bench-row fields
- §11.9 Implementation-time decisions not in §9/§10:
RUNG_PRIORITY ordering, bounded-domain anchors, reminder templates,
reject answer_text format, render branch, EXPLICIT_COUNT fallback,
quote-mode opt-out
- §11.10 Live verification artifacts (Winners-of-all-major-sports
rejected; Beatles-bounded NOT rejected)
- §11.11 What was NOT shipped: quantifier_preflight run-DAG node,
three-clause CTI contract DAG binding, A/B/D bench measurements,
cross-model verification — all queued as Phase 5 / bench follow-up
Implementation-time decisions section is the most operationally
useful — captures judgment calls made during coding that aren't in
the design docs but are now binding via tests.
Updates §8 Status, §9.6 Phase details, §9.8 Test surface, §9.11
commit sequence to reflect actual implementation state:
§8 Status — Phase 0 through Phase 4 all landed 2026-05-03 across
six commits (2ffed00 → 5a60e85). 906 tests passing (120 new).
Defaults preserve §10.11.3 dry-run discipline:
quantifier_guard_apply_caps=False, reminder=False, reject=False.
Six-level disable hierarchy fully wired. Live verification recorded
for both reject-broad-fires (Winners of all major sports?) and
reject-broad-skips (name all members of the Beatles → bounded
universal, NOT rejected).
§9.6 Phase details — each phase tagged LANDED <sha>; description
trimmed to what actually shipped vs the original proposal.
§9.8 Test surface — every check box flipped to [x] with the
matching test file path and test count. The two reject-broad
integration tests remain skipped (exercised by live bench).
§9.11 Implementation commit sequence — actual SHAs replace planned
commit numbers. Notes that DAG/audit binding for quantifier_preflight
node ("commit 7" in plan) is NOT shipped — tracked as Phase 5
follow-up. Optional now; required for §9.5 Merkle-AGI-DAG framing
to fully hold.
Next steps are bench measurement, not code: full bench under dry-run,
classifier review, then flip apply_caps and measure §10.8 deltas.
Closes the bench-side gap surfaced in §5.2: JSONL was carrying summary
numbers only, blinding the harness to FORMAT_COLLAPSED rate and per-
violation-kind distributions. Without these, A/B/D bench measurements
on the broad-quantifier preflight guard would be guesses.
- query() result dict surfaces format_collapsed + raw_answer (lattice
modes only) so the bench can read them directly instead of re-deriving
from cache rows that --burn overwrites.
- Each bench row gains format_collapsed, violation_kinds (sorted unique
list — full payloads stay off the row to keep size bounded), and
answer_brackets (count of [E\d+] in raw_answer for lattice modes).
- _summarize aggregates per-mode FC count (only explicit True; None
means check didn't apply), per-kind tallies (each kind once per row),
and lattice-only bracket sum/n.
- Markdown renderer adds a `## format-collapse + violation kinds`
section with per-mode FC rate, mean raw brackets, and one column per
observed violation kind. Degrades gracefully when the sweep produces
no violations.
- 5 new bench-harness tests pin the aggregation rules.
Re-baseline (2026-05-02T20-58-57Z) sharpens §5.1 analysis dramatically:
NO_EVIDENCE_POINTER fires 3/3 in pointer mode and is the dominant gate,
not TITLE_MISMATCH (1/3) as §5.1 inferred from JSONL alone. FORMAT_
COLLAPSED actually fires 1/3 — not the rare corner the first baseline
called it. Implies Option B (prompt reminder) is the load-bearing fix
for the verdict; Option A (cap reduction) only moves secondary kinds.
§5.3 sub-investigation closed on first read — SCHEMA_INVALID:1 in
pointer mode is a legitimate kind emitted by verify_claim_lattice for
empty-claim-text (verify.py:1242) and bare-name-claim (verify.py:1270),
not a JSON-mode leak.
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.
Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.
verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict
Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
into verifier_policy_hash
- threaded through ask() and query() call sites
CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail
Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.
Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).
Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.
Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.
New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.
Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
gain `subject_tokens_absent_threshold` kwarg + per-claim check
block (mirrors TITLE_MISMATCH plumbing, sits right after it
in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
`verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value
Live verification (cache-split cleanly via policy-hash bump):
pre-fix cache_key 08dbd2c1… : STRICT (false positive)
post-fix cache_key 6a519636… : UNGROUNDED
Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT
Full suite: 776 passed, 34 skipped.
Re-aggregated bench/emergent_log.jsonl at 200 cycles. Distribution
holds (79.5% UNGROUNDED, 17.5% HYBRID, 3.0% STRICT). Six STRICTs
total; five defensible (known-truth or non-relationship grounding).
The sixth — `steer/reply/correcter` — is the FIRST CONFIRMED
EVIDENCE-WARRANTED false-positive in 200 cycles. Decompressed all
10 chunks of the cited document (Glossary of language teaching
terms and ideas, 33.5K chars) and confirmed the words `correcter`,
`steer`, and `reply` appear NOWHERE in the source. Yet STRICT 1/1
verified, no violations.
Mechanism: per-claim citation-coverage Rule 5 (≥30% claim tokens
in cited span) passed on GENERIC linguistic vocabulary (language,
communication, grammar, exchange, relationships) which appears in
any linguistics article. The question-distinctive tokens
(correcter, steer, reply) rode along unverified. PREMISE_PARROTING
shape predicted in original ticket §D, exhibit A.
Headline correction: from "zero false-positives" to "one confirmed
false-positive (0.5%)."
Action item: opens design space for #000008 — PREMISE_PARROTING
detector. Lexical signal: question-distinctive tokens absent from
cited chunk while Rule 5 still passes on generic vocabulary →
SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays
binary, stays lexical, folds into verifier_policy_hash.
The ticket file was at docs/ticket-000006-... at top-level; the
tickets/ subdir convention landed before #6 was opened. Move into
docs/tickets/ to match every other ticket.
Amend with the 62-entry post-ticket delta. bench/emergent_log.jsonl
now holds 134 cycles total. Distribution stable vs original 72:
106 UNGROUNDED (79.1%), 22 HYBRID (16.4%), 6 STRICT (4.5%). Ten
new grounded cases all carry appropriate violation labels — zero
EVIDENCE-WARRANTED false-positives across all 134 cycles. Original
tuning candidates: Tomas-deflection resolved by a1dd330,
xxviii-STRICT kept as designed, metaphor sidecar calibration
deferred (still rare).
Reframe ticket as a rolling research log. Emergent stress-testing
is an ongoing thread, not a one-shot defect-fix; future
bench-emergent re-runs append new ## Amend sections here, and any
code-level tunings open their own tickets linking back.
TICKETS.md row reflects the rolling-research framing. Stale
"## Next ID" footer scrubbed from the ticket body (TICKETS.md is
canonical for the next-id counter).
Closes the FTS5 hyphen-tokenization asymmetry: `bi-polar is rare?`
retrieved only the Bi-Polar album/disambiguation cluster while the
medical-condition cluster (Bipolar disorder, Bipolar I/II disorder,
etc.) sat in the same shards untouched. `unicode61` splits hyphens
at index AND query time; `Bi-Polar Blues` indexes as [bi, polar, ...]
while `Bipolar disorder` indexes as [bipolar] — non-overlapping
token sets that never met.
Fix is query-layer only — no canonicalization_version bump, no
re-index, existing cache_keys stay valid:
- _hyphen_fold_variants(s): emit joined-no-hyphen variants for
every hyphenated run.
- _title_query_tokens(s): additively merges variants symmetrically
(queries AND titles when called on either).
- _filter_by_title_relevance: accept-path 5 — title stem-overlap
with hyphen-fold anchors passes the breadth gate. Rescues
`Bipolar disorder` (1-of-N qtoken match) without disrupting
non-hyphen queries (anchors empty → zero side effect).
- DEFAULT_QUERY_POLICY / DEFAULT_POLICY: hyphen_fold_v1: True
marker folds into governance_policy_hash; new records
cache-split cleanly from pre-fold records.
Live verification on /home/fox/.aborist/shards: same query now
retrieves `Bipolar disorder` (#5) and `Bipolar` disambiguation
(#7); model cites both, answer reads "Bi-polar disorder is not
rare; it affects approximately 2.8% of the U.S. population".
EVIDENCE-WARRANTED 2/2, properly grounded.
Tests: 4 new (3 unit, 1 integration with regression-pinned
Bipolar-disorder retrieval). Full suite 760 passed, 34 skipped.
Also: CLAUDE.md gains a close-when-complete hint for tickets — an
open ticket whose code already shipped is a stale map.
72 random-word triangulation cycles ran on 2026-05-02. Aggregate:
- 54 UNGROUNDED (75%)
- 14 HYBRID (19%)
- 4 STRICT (6%)
- 18 grounded total (25%)
Catalogs five failure/success shapes the curated bench-qa fixture
set can't surface, with one-line per category. Key finding: the
verifier-ladder + soft-demote stack is doing its job — every
HYBRID demoted via TITLE_MISMATCH / DEFLECTION_DETECTED /
CITATION_MISMATCH / TOO_MANY_EVIDENCE_IDS appropriately. Zero
EVIDENCE-WARRANTED false-positives in the sample.
Tuning candidates queued for the parallel-agent work:
1. Tomas-pulls-transceivers case escaped DEFLECTION_DETECTED
(subject anchor buried mid-paragraph; consider last 2-3
content tokens vs just last 1)
2. Roman-numeral / archaic-word triplets — accept gracefully
(system already does)
3. metaphor_deflection sidecar (added in parallel work) only
fired 1×/72; calibration pass when more samples land
Ticket is OPEN and awaiting follow-up tunings; re-run
`make bench-emergent EMERGENT_N=50` after parallel work commits
land and amend with the delta.
Status table + Next ID bumped to 000007.
12 per-module files + index = 13 files of 50-185 lines each =
1,064 lines of API reference scattered across a directory.
Each per-module file had real meat (API examples, ASCII tree,
conventions) but the cognitive cost of 'which file is this in?'
outweighed the navigation benefit.
Built via concatenation + patch-fix:
- cat index.md + per-module files in topological order
- rewrite ../diagrams/ -> diagrams/ (relative to docs/modules.md)
- rewrite ../../aborist/ -> ../aborist/
- rewrite ../TICKETS.md -> TICKETS.md, ../mesh.md -> mesh.md, etc.
- inter-module links (./<name>.md, <name>.md) -> #<name>-py anchors
- demote per-module H1 -> H2, H2 -> H3, etc., so the wrapper H1
is the only top-level heading
- de-dup the index.md's (now-H2) 'Aborist module reference'
header against the wrapper, replace with 'Diagrams index'
- inject explicit <a id="<name>-py"></a> anchors after each
module's H2 so the TOC links resolve regardless of GitHub's
auto-slug rules
- polish TOC link text: '[merkle.md](#merkle-py)' -> '[↓](#...)'
(the '.md' suffix made no sense once it's an in-doc anchor)
References updated:
- README.md (×2)
Net: 1,124 single-file lines vs 1,064 across 13 files. Slightly
larger because of the patch-fix scaffolding (anchors + section
markers), but one Cmd-F covers everything.
751/34 tests still pass.
Two cleanup operations bundled (separate scopes, single commit
since they share the doc-tree settle):
1. Move ticket-NNNNNN-<slug>.md files into docs/tickets/. The
directory makes browsing the design log easier; the index
stays at docs/TICKETS.md (top-level pointer). Convention text
in TICKETS.md updated to spell the new path.
2. Delete three docs whose load-bearing content has either been
absorbed into the codebase or distilled into closed tickets:
- docs/naming-deferral.md (147 lines) — explained why we
don't rename claim_lattice → CTI/PROMETHEUS-Σ. Decision
stays in place; the rationale is no longer worth a
dedicated doc. Inline citations removed from
cti-architecture.md (4 refs), warrant.py, ticket-000003
(closed-ticket internal ref).
- docs/reference-frame-failure-class.md (169 lines) — Orwell
case journal that motivated the phrase-pattern retrieval
route. The route shipped; the analysis is now duplicate
with the closed Ticket #000002. Inline citation removed
from CLAUDE.md retrieval pipeline section + frame.py.
- docs/test-coverage-audit-2026-05-01.md (46 lines) —
point-in-time audit checking 16/16 of fox's §11 list. Tests
themselves live in tests/; the audit was a one-shot
checkmark exercise.
References updated:
CLAUDE.md, aborist/qa/frame.py, aborist/qa/retrieval_plan.py,
aborist/qa/warrant.py, docs/cti-architecture.md, docs/TICKETS.md,
docs/tickets/ticket-000003 + ticket-000004 (internal links).
Net: -362 lines + tickets/ subdir. 751/34 tests still pass.
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
The journal had two stapled sections — the morning 11:31Z bench
(n=2 cell-grouped) followed by an 'Update' divider and the
afternoon 15:07Z bench (n=3 sample-shuffled). Two parts that
made readers walk past a horizontal rule to compare numbers, and
duplicated the directive-coverage / outputs / per-mode-rec
sections.
Rewrote as one coherent narrative:
- Frontmatter mentions both stamps in one table (when/what/wall)
- 'Hardening' section walks the chronology in two beats:
* pre-11:31Z (Rule 8, warrant gen, ladder, etc.)
* 11:31Z → 15:07Z (Sprint 1b, Sprint 2, DRY, keep-alive,
sample-shuffle, --resume, concurrency-sweep, surrogate v2)
- 'Aggregate' = authoritative 15:07Z table
- 'Δ across the day' = 3-column comparison
(2026-04-30 → 11:31Z → 15:07Z) with net deltas
- 'Per-bucket strict-rate' = 15:07Z bucket data
- 'Recommended context budget' = final
- 'Pointer-mode signal' kept (lazy-anchor analysis still valid)
- 'Wall-clock & throughput evolution' = both benches in one
table + concurrency sweep table
- 'Errors' = surrogate story across both benches
- 'Verdict' = final recommendation
- 'Outputs' = both jsonl files
212 lines → 175 lines (-36). One source of truth, one read.
bench-maxing.md gains a 'Bench harness — the speed playbook'
section capturing the 2026-05-02 speed wins as durable doctrine:
- Sample-level shuffled scheduling vs cell-grouped (+58%
throughput, true i.i.d. variance for n>=3)
- Persistent httpx client (TLS handshake amortization)
- Concurrency tuning (vLLM peak at c=3-4, brutal past c=4)
- Per-mode max_context_chars from bench feedback (the bench
is the substrate's voice; let it drive policy)
- --seed for reproducibility
- --resume for stop/start-able bench
- Smoke fixture for inner-loop iteration
- pytest -n auto (3.6× speedup on test suite)
README:
- Updated whitepaper section refs (§6/7/8/9/13) — old refs
pointed at §13.4.11/13.8/13.9 which no longer exist after
the whitepaper rewrite landed.
- 'Tests' section renamed 'Tests & bench' with make targets
for bench-qa, bench-qa-smoke, test-live. Resume + concurrency
semantics surfaced.
The c=4 sample-shuffled bench at 15:07Z lands the post-Sprint-1b
+ post-Sprint-2 + post-DRY + post-keep-alive + post-shuffle
state.
Headlines:
quote 0.50 → 0.54 (+4pp)
claim_lattice_pointer 0.23 → 0.20 (-3pp)
claim_lattice (JSON) 0.44 → 0.42 (-2pp)
Quote's +4pp is the cleanest lift of the sprint set: the per-mode
24KB cap (Sprint 1b) surfaces tighter retrievals that quote can
ground verbatim, and the bucket data confirms quote peaks at
8-16KB (0.58 strict-rate). JSON's peak migrated to its targeted
32-64KB bucket (0.48 strict-rate, vs 0.38 at 16-32KB) — Sprint
1b's intent confirmed at the per-bucket level even though the
aggregate slipped 2pp.
Pointer's slight drop is consistent with Sprint 2's smoke result
— the chunk-specificity Rule 9 didn't lift Hermes-3-8B's
lazy-anchoring at n=3. The structural fix will need a stronger
intervention than a prompt nudge.
Wall-clock & throughput:
11:31Z: cell-grouped, c=4, n=2, 426 tasks, 51 min, 8.4/min
15:07Z: sample-shuffled, c=4, n=3, 639 tasks, 48 min, 13.3/min
Sample-shuffled scheduling delivers +58% throughput at same
concurrency. n=3 (50% more work) ran in 6% LESS wall-clock.
Per-call mean latency dropped 35-42% across all modes — vLLM's
continuous batcher fills better when fed a diverse request
stream instead of cache_key-correlated cells.
Concurrency sweep: c=3 peak, c=4 within 4% (chosen), c=5 12%
slower, c=6 brutal (45% slower). vLLM saturates at c=3-4 on
this endpoint.
Errors: 6, all on 'tell me about the roman empire' question.
Root cause traced & fixed in 41d1d9b (lone UTF-16 surrogates
in Wikipedia chunk content broke httpx's outbound JSON encode
— different path from the 3b91223 SHA-256 hashers fix which
hardened the OUTPUT side). Next bench: 0 errors.
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
Captures the post-Rule-8 / post-warrant-generalization /
post-frame-detector bench taken with --concurrency 4 (~51 min
wall-clock on 426 runs).
Headlines (vs 2026-04-30 post-retry rerun):
quote 0.47 → 0.50 (+3pp)
claim_lattice_pointer 0.24 → 0.23 (−1pp)
claim_lattice (JSON) 0.50 → 0.44 (−6pp)
Pointer mode is essentially flat. The earlier 'pointer regressed
13pp' alarm was wrong — that compared to the pre-retry
2026-04-30 numbers (0.36) instead of the post-retry rerun (0.24)
which is the correct reference.
Real news: JSON's 6pp drop is the honesty cost of Rule 8
title-relevance promotion + warrant-class generalization. Lattice
modes now hit 99% directive coverage (D2/D3/D4/D6/D7); the 1% gap
is 2 UTF-16 surrogate errors per mode on one question.
Pointer-mode failure pattern shifted to lazy-anchor: 49/142 rows
have lazy_anchor_ratio >= 0.75. JSON's structured-per-claim
linkage absorbs Rule 8's pressure better than pointer's
prose-with-tags. 19/71 questions show JSON > pointer by 50pp+.
Recommended context budget surfaced for the first time:
- claim_lattice (JSON) peaks at 32-64KB (0.51)
- pointer + quote peak at 16-32KB
CLAUDE.md headline numbers refreshed; docs index points to the
new journal alongside the prior 2026-04-30 reference.
New `docs/concept-relations-design.md`: architecture reference for
the per-shard concept_relations layer that replaced the legacy
frozenset module (commit 5fd458a). Covers:
- Why phase 1 (hand-curated frozensets) didn't scale.
- Append-only schema + the three by-construction properties (idempotent
re-derivation via UNIQUE, per-shard storage, Merkle-orthogonal).
- Built-in `link_reciprocity_synonym` extractor reading the existing
`edges` table — no new crawler, works for Wikipedia AND HTML sites.
- Measured storage: 95.58 MB across 4 wiki shards (3.47M docs,
10.75M resolved edges, 55,148 reciprocal pairs, 289,848 synonyms),
4m16s wall-clock backfill. 1.6% tax on the 6 GB corpus.
- Three storage compactions considered & rejected, each with the
specific trade-off it loses on (drop idx_concept_evid → painful
purge debugging; BLOB source_root → schema inconsistency; FK
normalization → JOIN in retrieval hot path).
- How-to: backfill, manual add, purge.
- Adding new extractors.
- Deferred follow-ons (CLI commands, Wikipedia See-also extractor,
category extractor, hatnote extractor).
CLAUDE.md item 5 in the retrieval-pipeline list updated to point at
the new module path (aborist/concepts/) and the design doc.
TICKETS.md reference list updated to mention the new design doc.
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
#000004 — Directive coverage in bench summary. Filed as closed
(landed in commit acd1f9c) for design-log traceability. Captures
the v1 substrate: per-row directive_compliance helper, _summarize
aggregation, markdown directive-coverage section, 5 unit tests.
#000005 — Label ladder migration (POINTER-LINKED / ANCHOR-WARRANTED
/ EVIDENCE-WARRANTED / ENTAILMENT-VERIFIED). Status open. Migration
from today's two-rung EVIDENCE-LINKED to a four-rung ladder where
each rung names a strictly stronger property. Schema column stays
{STRICT, HYBRID, UNGROUNDED} so v9.8 cache_key invariants hold;
renderer-level mapping only. Reserves ENTAILMENT-VERIFIED for a
future committed entailment engine without forcing premature
implementation.
TICKETS.md index gains both rows. Next ID bumps to 000006.
seven-point-program.md status snapshot updated:
- D7 row links #000005 (ladder migration proposal).
- D8 row links #000004 (closed substrate).
- New "Anti-regression test layer" section maps each directive
to its structural pin in tests/test_directives.py.
Note on fox's roadmap item #000006 (Internal CTI Expansion / frame
lattice / multi-frame answers): scope overlaps with #000002
(Reference-Frame Polarity Contract / Module L). NOT opening as
a separate ticket — Module L already covers it.
Today's warrant-lite (aborist/qa/warrant.py) covers two anchor
classes: proper-noun (relation-shape questions) and date (any
claim with a 4-digit year). Three failure shapes pass the existing
seven hard checks unchecked:
- entity-list: 'name the simpsons family + pets' — claim
enumerates entities, but the warrant doesn't demand all named
entities appear in some cited span.
- count: 'how many wives did henry the eighth have?' — claim
says 'six', but the warrant doesn't demand the count token
appear in some cited span.
- why-cause: 'why did the titanic sink?' — claim names a cause
noun (iceberg, asteroid, propaganda), but the warrant doesn't
demand the cause token appear in some cited span.
Ticket proposes per-shape anchor extractors dispatched by question-
shape detector (existing _question_is_relation_shape pattern).
Conservative scope on cause anchors: gate by why-shape only since
lowercase common-noun extraction has higher false-positive risk.
Folds into governance_policy_hash via per-class policy dict so an
operator can disable individual classes per-corpus when they
over-fire.
Out of scope per docs/naming-deferral.md: typed-contract framework
(per-question-type rule libraries). General anchor-class primitive
catches the failures listed without per-type rule books; typed
contracts earn slots only when bench evidence shows the general
primitive misses cases.
Forecast cost ~3-4 hours; risk medium (false-positive risk on
why-shape cause-noun extraction). Closes the lazy-anchor gap on
the question shapes the bench question set already exercises.
The 2026-05-01 distilled architectural directive becomes the
audit lens applied before every new ticket / feature / prompt
edit. Each of the seven directives gets:
- status (✓ / ½ / ✗)
- code anchors (verifier paths, renderer hooks)
- pinning tests (per-rule coverage in tests/)
- tickets covering remaining work
- bench signal (which row column reports it)
Status snapshot at landing:
D1 (no LLM in verifier) ✓
D2 (pointer clauses) ✓
D3 (CTI internally) ½ → ticket #000002
D4 (retrieval map AND evidence map) ½ → ticket #000001
D5 (deterministic pointer verify) ✓
D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair)
D7 (honest labels) ✓
D8 (test-pinning before automation) discipline
CLAUDE.md docs index gains a "North-star" section pointing at the
program; ticket entries gain a "Directive" column threading them
back to the program.
Walked the five-step algorithm on CLAUDE.md itself.
Step 1 — requirements check: every section earned its place via a
specific operator-failure context. Sections that had grown into
prose essays got compressed back to load-bearing rules + pointers.
Step 2 — delete:
- Bench-maxing prose section (~100 lines) → moved verbatim to
docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list
pointing at the doc.
- Conventions section: each rule tightened to one paragraph max,
with a `See aborist/qa/verify.py` (or similar) trailing pointer
so the operator can jump straight to the rationale in code.
Verbose duplications between conventions and retrieval-pipeline
sections (e.g. wikitext base prose explained twice) collapsed.
- Retrieval pipeline: each of the 9 stages now references the
relevant module (`qa/concepts.py`, `query.py`) instead of
re-narrating the failure case in prose.
- Architecture tree: minor trim, removed redundant comments
where the filename already names the role.
Step 3 — simplify: source papers section dropped one item that was
duplicate (PDF + RST point to same content); kept the canonical
source.
Steps 4 + 5 — n/a (this is a doc, not a process).
New section: Docs index. Lists every architectural / design doc
in docs/ plus a ticket sub-list (open tickets with their files).
Closes the discoverability gap where TICKETS.md existed but agents
didn't know to look at it.
Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information
density is up — bench prose doesn't bloat the entry-point doc, and
each convention now points to its source. The `docs/bench-maxing.md`
extraction is the real win: separation of "rules I must not break"
(CLAUDE.md) from "discipline I should internalize" (bench-maxing.md).
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Fox-supplied review expanding §2-§4 with axiomatic framing, CTI /
Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete implementation
diff, and a strengthened test list. Captured verbatim as Appendix A.
Key refinements over the §2-§4 sketch:
- Splits retrieval-stage hash into retrieval_plan_hash +
retrieval_result_hash + retrieval_stage_hash (plan vs result
separation, axiomatically cleaner than embedding plan in the
retrieval-stage payload directly).
- Adds two new audit events: retrieval_plan_built and
retrieval_result_selected. Surfaces the retrieval inputs in the
audit chain (not just the run_dag), so cache-hits also commit
the plan via cache_hit_with_retrieval_plan.
- Distinguishes three keyword-handling cases for cache identity
(Case A/B/C) and lays out a clean boundary rule: keywords stay
in retrieval_plan unless they reach the model (then prompt_hash)
or affect routing policy (then retrieval_policy_hash).
- Promotes 'the map must be committed, not only the territory
reached' as a core principle. Operator-supplied maps (keywords,
routing modes, ranking policies) become first-class clauses
upstream of retrieval_result.
- Lists 12 concrete tests including the critical 'same sources,
different keywords → different run_dag_root' assertion that
proves the gap is closed.
Header note: Appendix A is authoritative when it disagrees with
§2-§4 above. The original §2-§4 sketch stays in place for
chronological context but is superseded by the appendix.
