CapitalProfile (8 forms: living, material, financial, intellectual,
experiential, social, cultural, spiritual) attached per state-changing
op as a sibling-table row in capital_ledger. Sibling semantics: ledger
rows reference an audit_event_hash but do NOT enter the audit-event
preimage, so retroactive cost re-estimation cannot break the chain.
Surface:
- arborist.capital.{profile,store}
- profile_for_op dispatch with per-op estimators (ingest/qa/distill)
- record/summary/op_cost/top_by_form
- CLI: arborist capital summary|op-cost|top
Wire-up at three op sites:
- ingest.py — one row per batch (doc_count + total_bytes)
- qa/runner.py — one row per cache-miss (answer_chars + llm_seconds)
- distill/runner.py — one row per derivation (positive intellectual)
Estimator constants are heuristic v1 (ESTIMATOR_VERSION pin in the
schema). Re-estimation is supported by re-running estimators against
the recorded inputs_blob and writing a new row with a bumped version
pin; old rows stay queryable.
Tests: tests/test_capital.py (13 cases). Sibling-table invariant
verified: audit chain stays intact across capital writes.
Full suite: 1025 passed, 36 skipped.
SelfModel binds an arborist agent's identity to bytes a verifier can
recompute: model_profile_hash, verifier_method_root, governance hash,
canonicalization/chunking versions, optional patch + memory roots,
sorted capability-claim hashes. Hard-hash committed; no soft state in
preimage. State transitions live on the row, not the body, so the
selfmodel_root stays stable across live → stale → falsified.
Surface:
- arborist.selfmodel.{canonical,snapshot,store,falsify}
- CLI: arborist selfmodel snapshot|show|falsify|list
- Schema: selfmodel_records + selfmodel_capability_claims (additive)
- Audit events: selfmodel_snapshot_landed,
selfmodel_capability_claim_added, selfmodel_falsified,
selfmodel_marked_stale (all chain via existing append_audit)
Also folds in:
- CLAUDE.md operational rule: arborist stays Python-only; non-Python
toolchains live in sibling repos. Forks/clients/servers in any
language follow our schemas + canonical encodings.
- Ticket #000016 update: ZK lives in sibling repo arborist-zk-bench;
arborist gains at most a wire-format consumer, never a Rust dep.
- Schema migrations also stub capital_ledger and memory_records
tables for tickets #000020 and #000017 respectively (additive,
empty until those modules land).
Tests: tests/test_selfmodel.py (14 cases; canonical-JSON stability,
root order-invariance, snapshot determinism, store idempotency,
audit events, falsify/mark_stale semantics, audit-chain integrity).
Full suite: 1012 passed, 36 skipped.
Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.
#000025 — Metacog test fixture expansion:
bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
covering edge cases per detector kind: temporal (4 cases),
contradiction (4), false-premise (5), out-of-corpus (3), multi-
trigger (2), well-formed controls (5). Documents two known
detector ceilings: Q11 over-fires on past-tense factoid
("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
NASA-fake) miss false premises that lack a presupposition
pattern match. Fixture now serves as long-term regression suite.
#000026 — --show-preflight full clause render:
build_run_dag() and build_reject_run_dag() gain optional
preflight_payload kwarg. When supplied, the canonical 5-clause
CTI payload (classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs + question_state + node_version)
persists alongside the leaf hash in run_dag_blob.
aborist providence --show-preflight CACHE_KEY now renders the
full payload + verifies the persisted hash matches the
recomputed canonical hash (audit-replay tamper detection).
Legacy rows fall through cleanly: payload_hash_check reports
"unavailable: legacy row predates preflight_payload persistence".
#000027 — Latency profile:
Microbenched preflight: 0.46ms/question (negligible). Single
fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
33-35s in Addendum 3 was vLLM concurrency contention at c=4
(per qa-modes-bench.md saturation note), not substrate
overhead. Added preflight_ms + soft_preflight_ms to timings
dict for explicit confirmation in future cycles.
#000028 — Auto-quality-check sweep revival:
scripts/bench_emergent.py running with EMERGENT_N=100 in
background (PID 125680). Will accumulate cycles into
bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
Async — not blocking on completion.
#000029 — #000011 SOFT_PREFLIGHT_HINT implementation:
aborist/qa/soft_preflight.py — new module. SoftPreflightHint
dataclass + soft_preflight_question() pure function. 9
canonical labels mapping to soft analogues of #000010 hard
detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
Constrained-generation prompt (max_tokens=128, temp=0.0) asks
the model to pick ONE label + one-line rationale.
Fail-closed across every parse path:
- chat_client raises → SOFT_PARSE_FAIL
- response unparseable → SOFT_PARSE_FAIL
- label outside enum → SOFT_PARSE_FAIL
Sidecar enforces SOFT_ prefix at the normalize step so a
model that drops the prefix still gets caught.
Wired into query() between preflight & retrieval. Default
OFF (`soft_preflight_enabled: False`). NOT folded into
_VERIFIER_POLICY_FIELDS — soft hints don't gate cache
identity (#000011 §4). Audit-line tail renders as
"· soft: <label>" (e.g. "· soft: time sensitive") so the
signal is visually distinct from hard tails.
--soft-preflight CLI flag opts in per-call. End-to-end
live-verified on "When did Mr. Burns become Homer's biological
father?" — produces:
EVIDENCE-WARRANTED · via claim_lattice
· false premise · soft: time sensitive
1/1 16.4s
Hard `· false premise` (from #000010 deterministic detector)
composed with soft `· soft: time sensitive` (from #000011
sidecar). The model classified a different shape than the hard
detector — by design; soft hints are independent advisory
signals, not redundant with the hard layer.
25 new tests pin: default-OFF behavior, parse-failure modes,
label normalization (SOFT_ prefix enforced), all 8 actionable
labels round-trip, fail-closed on client exceptions, dataclass
JSON round-trip, rationale-length cap.
Other:
- #000010 §13.3 documents 2/5 metacog-trigger questions return
STRICT despite hard-detector warning — direct empirical
motivation for #000011 design.
- tests/test_dag.py extends with 3 _extract_preflight_hash_*
helper tests (cleaning #000009 §7.2 unfinished state).
- bench/emergent_log.jsonl adds new cycles from background run.
#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
Fan-out execution of the deferred-but-not-blocking pile from
prior status reports.
#000009 §7.2 — bench harness preflight_hash field:
- aborist/qa/query.py surfaces `preflight_hash` on result dict
(miss path, reject path, and cache-hit path via new helper
`_extract_preflight_hash_from_blob` that pulls the stage hash
out of persisted run_dag_blob).
- bench/qa_sweep.py adds 12-char preflight_hash prefix to bench
rows. Mirrors cache_key truncation pattern. Operators can
grep / SQL-filter bench JSONL by preflight policy state.
- 3 new tests in tests/test_dag.py for the extract helper.
#000009 §7.2 — `aborist providence --show-preflight CACHE_KEY_PREFIX`:
- New CLI flag pulls the preflight stage payload from a row's
run_dag_blob. Match by 12-char prefix. Renders preflight stage
hash + run-DAG stage list. Operator tool for inspecting which
policy state governed a cached row.
- Live verified on a real cache row (8a212fecb2a9 — current CEO
of OpenAI question, 10-stage CTI shape with preflight at idx 1).
- Legacy rows (predating #000009) report a clean fall-through
message: "run_dag has no preflight stage (predates #000009)".
#000010 §12.6 — reference-frame plumbing into QuestionState:
- Pre-retrieval preflight runs with reference_frames=()
(frame_detection needs source titles, not available yet).
Post-retrieval, query.py re-runs preflight_question() with
the detected frames so the result-dict + run-DAG QuestionState
carry frame-aware logical_statuses (specifically
`reference_frame_ambiguous` when 2+ frames match).
- Live verified on Orwell-style question; logical_statuses now
correctly includes `reference_frame_ambiguous` in the result.
Metacog-trigger bench fixture (#000010 §13.3):
- bench/qa_questions_metacog_subset.txt — 6 questions, one per
detector kind plus a well-formed control.
- Bench artifact 2026-05-04T02-18-42Z. Detector accuracy 6/6
on fixture; 2 of 5 trigger questions return STRICT on lattice
mode despite metacog warning (JSON STRICT on
George-Washington-stop-being-president-of-France false-premise
+ uploaded-contract out-of-corpus questions). Audit-line tails
correctly surface the warnings.
- qa-modes-bench.md Addendum 4 captures the per-question matrix
+ interpretation. #000010 §13.3 cross-references with bench
artifact stamp.
#000011 SOFT_PREFLIGHT_HINT design ticket opened:
- docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md
captures the design proposal per #000010 §18 / source doc.
Implementation deferred — design only.
- Sidecar would add model-assisted preflight as a soft signal
(`SOFT_FALSE_PREMISE_SUSPECTED` etc.) that NEVER enters the
verifier hard path. Strict guardrail: cannot create
PREFLIGHT_OK or PREFLIGHT_BLOCKED without deterministic
support.
- Validated by §13.3 finding: deterministic detectors flag
correctly; corpus-accidental grounding produces 2/5 STRICT
on trigger questions; soft sidecar would add independent
semantic skepticism.
- TICKETS.md index row added; Next ID bumped to 000012.
996 tests passing (3 new for the extract helper).
Cross-doc consistency:
- qa-modes-bench.md Addenda 1+2+3+4 chronological
- #000010 §13.1 (broad subset) + §13.2 (full bench) + §13.3
(metacog trigger subset)
- #000011 design captured but not implemented
Architectural feedback at ~/Downloads/RESPONSE-ticket-000009-... .txt
(2026-05-04) flagged five gaps in the c36e85c landing. Most
critical: reject-broad early-return path emitted no run_dag_blob,
so audit replay couldn't see that a rejection happened (let alone
under what policy state).
A — reject-path DAG (the critical gap):
aborist/qa/dag.py: build_reject_run_dag() — 3-stage minimal DAG
question → preflight → final_label. final_label payload carries
rejection_reason + answer_text_hash so two rejections under
different policy state produce different roots.
query.py reject path now wires it: returns run_dag_root +
run_dag_blob on the rejection result dict. Live-verified end-
to-end on `make query Q="winners of all major sports?"
REJECT_BROAD=1 BURN=1`.
Audit replay rule: 3 stages always means reject path. Operators
can read the stage list and tell instantly without parsing the
payload.
B — nested CTI clauses:
preflight_node_hash() payload restructured from flat 3-key to
nested 5-clause:
classifier — quantifier classifier output (#000008)
answer_contract — guard / cap / reject / metacog state (per-run)
prompt_contract — reminder enabled / injected / template_id
evidence_contract — exposure budget, line discipline
policy_refs — governance_policy_hash, model_profile_hash,
answer_mode (reference, not raw policy)
Plus question_state (metacog) as its own clause and top-level
stage + node_version. Single DAG stage; nested clauses inside
for diff legibility (feedback §3).
C — node_version field:
PREFLIGHT_NODE_VERSION = "preflight-node-v1" pinned in the
payload so legacy runs without the node can be unambiguously
labeled `unavailable_legacy_run` by audit tools (feedback §9).
D — reference hashes only:
policy_refs uses governance_policy_hash + model_profile_hash
rather than bundling raw policy booleans. Avoids
double-committing already-hashed state (feedback §4).
E — reminder_template_id:
prompt_contract.reminder_template_id = "broad-quantifier-bounded-v1"
or "broad-quantifier-unbounded-v1" depending on scope_bound_hint,
populated only when reminder actually fires.
F — stage name kept as `preflight` (not `quantifier_preflight`):
Node carries both #000008 quantifier AND #000010 metacognition
payloads. node_version disambiguates schema for audit tools.
G — docs/cti-architecture.md update deferred to a small follow-up.
Bug fixes:
- free-variable shadowing on verifier_policy_hash /
model_profile_hash / question_hash — local re-imports inside
the reject branch shadowed module-top imports used elsewhere
in query() / runner(); now use the module-top names.
- reject path question_hash signature: takes `mode=` not
`dedup_mode=` — fixed in the reject DAG builder caller.
Hash compatibility:
Rows written between c36e85c and this commit have hash payloads
matching the OLD flat 3-key shape. The persisted run_dag_blob
captures the actual payload that was hashed, so those rows
still verify via verify_run_dag(). New rows use the nested
5-clause shape.
7 new tests in tests/test_dag.py:
- hash sensitivity to answer_contract / prompt_contract /
policy_refs flips (audit-replay payoff demonstrations)
- PREFLIGHT_NODE_VERSION pinning
- reject DAG: 3-stage shape, root changes with preflight hash,
round-trips through verify_run_dag
993 tests passing (6 net new); 36 skipped.
Live verification:
make query Q="winners of all major sports?" REJECT_BROAD=1 BURN=1
→ status=broad_quantifier_rejected, run_dag_root populated,
blob carries 3-stage shape.
make query Q="winners of all major sports?" BURN=1
→ 10-stage shape preserved (question → preflight → retrieval
→ ... → final_label).
Ticket #000009 status: closed · re-landed 2026-05-04 with §8
corrections.
Closes ticket #000009 zero-shot. Scope expanded to cover BOTH
ticket #000008 (broad-quantifier) AND ticket #000010
(meta-cognition) preflight contracts in a single combined node —
both share the same audit-replay gap and inserting two separate
nodes between question and retrieval was operationally awkward.
aborist/qa/dag.py:
+ preflight_node_hash() — combines QuestionState +
quantifier classifier output + behavioral policy_state into
one canonical SHA-256 hex.
+ build_run_dag() gains optional preflight_hash parameter.
When supplied, inserts {"stage": "preflight", "hash": ...}
at position 1 (between question and retrieval).
Backward-compat: None → original 7/9-stage shapes preserved
for legacy run_dag_root re-validation.
Quote-mode: 7 → 8 stages with preflight.
Pointer-mode CTI: 9 → 10 stages with preflight.
aborist/qa/query.py + runner.py:
Both build the preflight payload from question_state +
quantifier dict + 10-field policy_state (guard_enabled,
guard_apply_caps, guard_apply_caps_mode_gated,
claim_cap_resolved, claim_cap_actually_applied,
reminder_enabled, reminder_eligible, reject_broad_active,
metacognition_enabled, block_on_contradiction).
This means two cache rows that share the same question + same
model output + same verifier verdict but DIFFERENT preflight
policy state now produce different run_dag_root values. Audit
replay can pin the policy decision per row.
9 new tests in tests/test_dag.py:
- preflight_node_hash determinism
- hash bumps on question_state change
- hash bumps on policy_state change (the audit-replay payoff)
- all-None defensive shape
- 7→8 stage transition (quote mode)
- 9→10 stage transition (pointer mode), preflight at index 1
- run_dag_root bumps when preflight_hash bumps
- verify_run_dag round-trips through preflight stage
Live verification: latest providence_cache row carries
['question', 'preflight', 'retrieval', 'evidence_map', 'prompt',
'raw_answer', 'parsed_claim_lattice', 'verify', 'render',
'final_label'] — preflight stage living in the persisted DAG.
987 tests passing (9 new); 36 skipped.
Tickets:
#000009 status: closed · landed 2026-05-03 (zero-shot)
#000010 cross-ref updated: "DAG binding shipped via #000009"
What's NOT in this ticket (logged in §7.2):
- CLI flag for inspecting preflight node from cache_key
- Bench harness preflight_hash field for cross-row comparison
- SOFT_PREFLIGHT_HINT (model-assisted preflight sidecar)
Closes ticket #000010 (Meta-Cognition Preflight Guard). Mechanism
complete; defaults preserve the dry-run discipline pattern from
#000008.
Phase 2 — wire preflight into query() and runner.ask():
- preflight_question() runs after policy resolution + quantifier
classification, before retrieval.
- QuestionState surfaces on miss path, cache-hit path, AND
reject-broad early-return path of query() — schema column-
aligned across all four returns.
- runner.ask() carries the same fields for `aborist ask` parity.
Phase 3 — policy fields + governance hash + CLI flags:
- 6 new policy fields, all default-on except
metacognition_block_on_contradiction (default False — label-
only by default; opt-in via --block-on-contradiction).
- All 6 folded into _VERIFIER_POLICY_FIELDS so flipping any
invalidates prior cache records on lookup.
- 2 new CLI flags on `aborist query`:
--no-preflight Level 2 master kill
--block-on-contradiction strict mode (hard-block on
lexical contradictions)
Phase 4 — audit-line labels + bench fields + tests:
- _render_warrant_tail extended with 5 metacog tail tokens:
· false premise
· contradictory
· stale risk
· out of corpus
· frame ambiguous
- Bench rows in qa_sweep.py gain 7 new bounded-size projection
fields (logical_statuses, question_shape, preflight_result,
temporal_sensitivity, has_false_premise, has_contradiction,
corpus_requirement). Full QuestionState stays on result dict
for CLI render only.
- tests/test_metacognition.py grew from 42 → 68 tests
(16 new: 6 governance + 6 audit-line tail + 4 default-policy
pinning).
Live verified end-to-end:
$ make query-dry Q="Who is the current CEO of OpenAI?" BURN=1
UNGROUNDED · via claim_lattice · stale risk
$ make query-dry Q="When did Mr. Burns become Homer's biological
father?" BURN=1
UNGROUNDED · via claim_lattice · false premise
978 tests passing; 36 skipped.
What's NOT shipped (deferred):
- Run-DAG node binding for metacognition_preflight stage —
joins ticket #000009 Phase 5 (same audit-replay gap; both
nodes can land together).
- Reference-frame plumbing — frame_detection runs post-retrieval,
preflight here is pre-retrieval; deferred until two-pass
or post-classification update lands.
- SOFT_PREFLIGHT_HINT (model-assisted sidecar) — source doc §18
reserves this label; hard rule preserved (no LLM in preflight
hard path).
- Bench A/B measuring preflight on vs off — quick to run once
stack settles.
Ticket #000010 status: closed · landed 2026-05-03.
Implements the Meta-Cognition Preflight Guard (M0 / MCTL) per
fox's directive at ~/Downloads/meta-cognition_for_hermes(1).txt
(2026-05-03).
aborist/qa/metacognition.py:
- QuestionState dataclass (frozen, JSON-serializable via to_dict)
- preflight_question() pure function: classifies a question
deterministically into a QuestionState before generation
- 4 new detectors:
detect_temporal_sensitivity() — current/latest/today/CEO/etc.
detect_contradiction() — lexical pairs (unmarried+spouse,
always+never, alive+dead, etc.)
detect_false_premise() — presupposition patterns:
when did X stop/become Y,
why did X cause Y,
how did X become Y
detect_out_of_corpus() — my-uploaded-X / file-I-sent shapes
- Reuses #000008 quantifier classifier (no duplication)
- Composes 8 LogicalStatus values:
well_formed, under_specified, false_premise_suspected,
contradictory_question, out_of_corpus_risk, stale_risk,
reference_frame_ambiguous, broad_quantifier_unbounded
- Three preflight results: PREFLIGHT_OK / _PARTIAL / _BLOCKED
- Per-detector enable switches in policy:
metacognition_enabled (master kill)
metacognition_temporal_check
metacognition_contradiction_check
metacognition_false_premise_check
metacognition_out_of_corpus_check
metacognition_block_on_contradiction (default False — label
only by default; opt-in
to hard-block)
- preflight_policy_hash for governance binding (Phase 3)
- PREFLIGHT_VERSION = "metacognition-v0.1"
Hard rule (D1): no LLM in this hard path. Pure regex + lexical
matching. Model-assisted preflight, if added later, labels itself
SOFT_PREFLIGHT_HINT (not implemented in this phase).
42 new tests cover the seven test cases from source doc §14
(false-premise, contradictory, broad-quantifier, reference-frame,
time-sensitive, out-of-corpus, model-cutoff) plus per-detector
unit tests, gating (master kill, per-detector disable,
block-on-contradiction opt-in), determinism (question_hash
stable, policy_hash bumps on flip), and serialization.
Ticket #000010 opened with status `open · in progress
(zero-shot 2026-05-03)`. TICKETS.md index updated; Next ID bumped
to 000011.
Phases 2-4 still queued (wire into query/runner, policy fields +
governance, audit-line labels + bench fields).
962 tests passing (42 new); 36 skipped.
n=5 verification of cap+reminder cell (135 runs):
Metric | n=3 | n=5
--------------------+-----------+------------
JSON SR | 0.30 | 0.33 ← matches cap-only
JSON UNGROUNDED rate| 1/27 (4%) | 2/45 (4%) ← matches reminder-only
pointer SR | 0/27 | 0/45 ← unchanged across all cells
The §12.8 0.30 was Hermes nondeterminism. n=5 confirms cap+reminder
delivers cap-only's STRICT-rate AND reminder-only's UNGROUNDED-rescue.
§10.8 strict gate met at n=5:
vs reminder-only on JSON SR: +11pp (clears floor)
vs cap-only on JSON UNGROUNDED: -18pp (clears floor)
vs cap-only on ptr mean ratio: +12pp (clears floor)
Defaults flipped — Option A landing (per-mode tailored):
quantifier_reminder_enabled False → True
(load-bearing on both lattice modes)
NEW field: quantifier_apply_caps_modes = ["claim_lattice"]
(allowlist for which modes apply caps
when apply_caps=True; JSON-only since
cap-on-pointer is wasted noise per
§12.10 0/45 STRICT data)
quantifier_guard_apply_caps False → False (UNCHANGED)
(operator opts in via
--apply-quantifier-caps; preserves
§10.11.3 dry-run discipline)
Cap-application gate now reads:
if apply_caps AND mode in apply_caps_modes AND cap is not None:
effective_max_claims = looked_up_cap
quantifier_apply_caps_modes folded into _VERIFIER_POLICY_FIELDS so
flipping the allowlist invalidates prior cache records.
5 new tests pin: reminder default ON for both runner.DEFAULT_POLICY
and query.DEFAULT_QUERY_POLICY; apply_caps_modes default
["claim_lattice"]; governance-hash invalidation on allowlist flip;
apply_caps default still False (dry-run preserved).
920 tests passing (5 new); 36 skipped.
Operator behavior:
$ aborist query "winners of all major sports?"
→ reminder ON, cap OFF (default after this commit)
$ aborist query --apply-quantifier-caps "..."
→ cap applies on claim_lattice (JSON) only
$ aborist query --apply-quantifier-caps \
--policy quantifier_apply_caps_modes='["claim_lattice","claim_lattice_pointer"]' "..."
→ Option D for one call
Phase 5 (run-DAG node binding for quantifier_preflight) and
cross-model Qwen/GPT-4 verification remain as follow-ups per §11.11.
Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):
intensity pre-fix post-fix
SINGULAR 61 (84%) 65 (89%)
MANY 4 ( 5%) 0 ( 0%) ← all 4 were `how many X?`
ALL 1 ( 1%) 1 ( 1%)
COMPREHENSIVE 1 ( 1%) 1 ( 1%)
OPEN_REQUEST 5 ( 7%) 5 ( 7%)
SMALL_NUM 1 ( 1%) 1 ( 1%)
Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).
Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.
9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).
Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:
name all members of the beatles
list all planets in the solar system
Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.
915 tests passing (9 new); 36 skipped.
CLI flags on `aborist query`:
--no-quantifier-guard Level 2 disable: kills the guard for
one call. Telemetry → None.
--allow-broad Emergent-search: classifier on, caps
off. For exploratory enumeration.
--reject-broad Strict reject: ALL/COMPREHENSIVE/
OPEN_REQUEST + scope_bound_hint==
"unbounded" returns UNGROUNDED before
the LLM call (saves ~10-15s). Bounded
universals (Beatles, year-anchored)
are NOT rejected per §10.1.
--apply-quantifier-caps Flip Phase 2 dry-run gate per-call.
Path from dry-run to live cap.
Three new soft-demote violation kinds (§10.3) — no new audit_mode
token; tails on the existing audit-line:
BROAD_QUANTIFIER_RUNAWAY "broad runaway"
BROAD_QUANTIFIER_CAP_APPLIED "broad cap N" (cap value rendered)
BROAD_QUANTIFIER_SCOPE_UNBOUND "broad unbounded"
All three cap the ladder at ANCHOR-WARRANTED. Plus one HARD demote
(early-return UNGROUNDED):
BROAD_QUANTIFIER_REJECTED "broad rejected" (preflight rejection)
The reject-broad path early-returns from query() before the LLM
call when policy enables quantifier_reject_broad AND the question
is broad-unbounded. Result schema mirrors a normal UNGROUNDED row
(answer_text carries the rejection rationale + actionable narrowing
hints). _render_query_human gets a dedicated branch for the new
status so operators see the rejection without --json.
Live verification (post-commit):
$ aborist query --reject-broad "Winners of all major sports?"
UNGROUNDED · via BROAD_QUANTIFIER_REJECTED · ALL ("all") · cap was 8
0/0 0.0s (preflight)
BROAD-QUANTIFIER PREFLIGHT REJECTED · scope unbounded
Question matched ALL intensity ("all") with an under-specified
universe. Narrow ... or run with --allow-broad for exploratory
enumeration.
$ aborist query --reject-broad "name all members of the Beatles"
UNGROUNDED · via claim_lattice · title mismatch 4/4 20.9s
[Beatles enumerated, scope_bound_hint=bounded → not rejected]
`quantifier_reject_broad` folded into _VERIFIER_POLICY_FIELDS so
flipping reject default invalidates prior cache records.
16 new tests cover: soft-demote registration, hard-demote NOT in
soft-demote set, ladder rung mapping for each kind, tail rendering
(including cap value interpolation), tail combination with
existing kinds, end-to-end render through _render_query_human,
governance-hash binding. Two skipped placeholders mark the
integration paths exercised by live bench.
Lands aborist/qa/quantifier_reminder.py with broad_quantifier_
reminder(): one-line user-turn message restating the cap and the
[E\d+] citation rule for broad-intensity questions. Two templates:
bounded universe:
"This is a broad-quantifier query with a bounded universe.
Return at most N pointer-linked claim lines. Each claim must
cite an evidence id like [E5]; do not write claim lines
without bracket citations."
unbounded universe:
"This is a broad-quantifier query with an under-specified scope.
Return at most N pointer-linked claim lines. If you cannot
ground N claims with evidence IDs, return fewer grounded
claims. Do not enumerate from training prior. Each claim must
cite an evidence id like [E5]; do not write claim lines
without bracket citations."
The bounded template skips the "do not enumerate from training
prior" clause — the corpus has the answer set. Unknown scope falls
through to the stricter unbounded template (over-warn rather than
under-warn).
Wired into both query() and runner.ask() at the same insertion
point as the existing grounding_reminder — between
grounding_reminder and the evidence/question payload, where
Hermes-3-8B's most-recent-token attention catches it.
Default OFF (`quantifier_reminder_enabled: false`). Empirical
justification: ticket §3 Option B con notes Hermes already ignores
parts of the existing reminder under enumeration pressure. The
mechanism lands so an operator can A/B test cap-only vs cap+reminder
without code changes; default flips on after bench shows ≥5pp delta
on FORMAT_COLLAPSED or pointer-loss rate per §10.8.
`quantifier_reminder_enabled` folded into _VERIFIER_POLICY_FIELDS
so flipping the switch invalidates prior cache records.
19 new tests cover: gating (non-broad → None, missing cap → None,
None intensity → None), bounded-vs-unbounded template selection,
unknown scope falls back to unbounded, cap interpolation,
[E\d+] citation rule restatement, governance-hash invalidation.
Lands aborist/qa/model_profiles.py with two profiles:
- adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic
tight caps for broad intensities (ALL=8, COMPREHENSIVE=5,
OPEN_REQUEST=5) reflecting the 2026-05-02 runaway case.
- default
large-reasoner-class fallback (ALL=12, COMPREHENSIVE=15,
OPEN_REQUEST=12). Used when model_profile_id has no entry.
cap_for_intensity() resolves caps via three-source chain:
1. policy_overrides (per-call dict, highest priority)
2. per-model profile from PROFILES
3. "default" profile fallback
EXPLICIT_COUNT sentinel handles SMALL_NUM_EXPLICIT and
COMPARATIVE_BOUND — cap is the question's explicit count, not a
profile-set value. Defensive fallback to MANY cap if classifier
fired the rung without extracting a count.
Four new policy fields, all folded into governance_policy_hash via
_VERIFIER_POLICY_FIELDS:
- quantifier_guard_enabled master kill (default True)
- quantifier_guard_apply_caps dry-run gate (default False per
§10.11.3 — cap LOOKED UP and reported
on result, but NOT applied to the
verifier until operator flips True)
- quantifier_caps_by_intensity per-call override dict
- quantifier_guard_modes per-mode opt-in list (default
["claim_lattice_pointer",
"claim_lattice"]; quote opts out)
Six-level disable hierarchy (§10.11.2) implemented:
- Per-test: policy={"quantifier_guard_enabled": False}
- Per-call: --no-quantifier-guard (Phase 4)
- Per-phase: each policy switch is independent
- Per-mode: quantifier_guard_modes filter
- Per-model: model_profiles.py lookup
- Master: governance_policy_hash invalidation on flip
Wired through both query() and runner.ask() — both compute
effective_max_claims from the (classifier_intensity, model_profile,
policy_overrides) triple and pass it as max_claims_per_answer to
the verifier. Dry-run mode keeps effective_max_claims at the policy
default (12) until apply_caps flips True.
Result dict surfaces claim_cap_applied (the LOOKED-UP cap, even in
dry-run) plus all Phase-1 quantifier fields on miss-path AND
cache-hit path so bench rows stay column-aligned.
19 new tests pin: per-model selection, EXPLICIT_COUNT sentinel,
override precedence, governance-hash invalidation on every cap
field, profile shape (all ten rungs covered), default profile
presence.
Lands aborist/qa/quantifier.py with classify_question_quantifier(),
a pure function mapping a question string onto the ten-rung
intensity ladder (ticket #000008 §2):
ABSENT < SINGULAR < PROPORTIONAL < SMALL_NUM_EXPLICIT
< COMPARATIVE_BOUND < FEW < MANY < ABSENT < ALL
< OPEN_REQUEST < COMPREHENSIVE
Returns intensity, matched_token, explicit_count, is_broad,
operational_shape, scope_bound_hint, classifier_version. Pure: no
I/O, no model call, no retrieval call.
Highest-intensity-wins arbitration: COMPREHENSIVE strictly stronger
than OPEN_REQUEST (both > ALL). Catches "tell me everything about
all wars" → COMPREHENSIVE rather than dropping to one of the softer
shape detectors.
Scope_bound_hint heuristic (§10.1): bounded vs unbounded universals.
"All members of the Beatles" → bounded (corpus-known finite set).
"Winners of all major sports" → unbounded (scope undefined). Year-
anchored questions ("…in 2024") bound the universe to one event.
Heuristic only — corpus-arity check left for future refinement.
Wired into query() right after policy resolution. Both miss and
cache-hit paths surface quantifier_intensity, quantifier_matched
_token, scope_bound_hint, quantifier_explicit_count on the result
dict. claim_cap_applied is None until Phase 2 lands the cap-
application gate (default-off per §10.11.3 dry-run discipline).
61 new tests cover every rung, scope-bound detection (bounded /
unbounded / unknown), highest-wins arbitration, and regression
fixtures (factoid/wh-questions don't over-classify as broad).
Adds the §10.6 + §9.6 Phase-0.x bench-row fields. Pure additive —
no row-schema renames, no policy effects yet. Phase 1 classifier
will fill the quantifier_* slots; Phase 2 cap-table will fill
claim_cap_applied. Keeping the keys present here makes the JSONL
schema stable across the rollout so post-Phase-1 markdown can
re-render against pre-Phase-1 rows without column-misalignment.
New per-row fields:
- answer_pointer_count distinct E\d+ ids in raw_answer
- answer_chars_with_brackets chars inside [E\d+,...] regions
- raw_meaningful_line_count >20-char lines in raw_answer
(matches verifier FORMAT_COLLAPSED
denominator)
- quantifier_intensity slot for Phase 1 classifier
- quantifier_matched_token slot for Phase 1 classifier
- scope_bound_hint slot for Phase 1 classifier
- claim_cap_applied slot for Phase 2 cap-table
- model_profile_id configured model id verbatim
New helper _bracket_diagnostics() bundles the bracket/pointer/line
extraction in one place; module-level regexes (_BRACKET_RE,
_BRACKET_REGION_RE, _POINTER_ID_RE) avoid per-row recompilation.
5 new tests cover the helper: empty input, single pointer, multi-
pointer-in-one-bracket, separate brackets with shared id, format-
collapsed shape (5+ meaningful lines, 0 brackets — the
2026-05-02 winners-of-all-major-sports case).
Closes the bench-side gap surfaced in §5.2: JSONL was carrying summary
numbers only, blinding the harness to FORMAT_COLLAPSED rate and per-
violation-kind distributions. Without these, A/B/D bench measurements
on the broad-quantifier preflight guard would be guesses.
- query() result dict surfaces format_collapsed + raw_answer (lattice
modes only) so the bench can read them directly instead of re-deriving
from cache rows that --burn overwrites.
- Each bench row gains format_collapsed, violation_kinds (sorted unique
list — full payloads stay off the row to keep size bounded), and
answer_brackets (count of [E\d+] in raw_answer for lattice modes).
- _summarize aggregates per-mode FC count (only explicit True; None
means check didn't apply), per-kind tallies (each kind once per row),
and lattice-only bracket sum/n.
- Markdown renderer adds a `## format-collapse + violation kinds`
section with per-mode FC rate, mean raw brackets, and one column per
observed violation kind. Degrades gracefully when the sweep produces
no violations.
- 5 new bench-harness tests pin the aggregation rules.
Re-baseline (2026-05-02T20-58-57Z) sharpens §5.1 analysis dramatically:
NO_EVIDENCE_POINTER fires 3/3 in pointer mode and is the dominant gate,
not TITLE_MISMATCH (1/3) as §5.1 inferred from JSONL alone. FORMAT_
COLLAPSED actually fires 1/3 — not the rare corner the first baseline
called it. Implies Option B (prompt reminder) is the load-bearing fix
for the verdict; Option A (cap reduction) only moves secondary kinds.
§5.3 sub-investigation closed on first read — SCHEMA_INVALID:1 in
pointer mode is a legitimate kind emitted by verify_claim_lattice for
empty-claim-text (verify.py:1242) and bare-name-claim (verify.py:1270),
not a JSON-mode leak.
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.
Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.
verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict
Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
into verifier_policy_hash
- threaded through ask() and query() call sites
CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail
Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.
Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).
Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.
Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.
New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.
Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
gain `subject_tokens_absent_threshold` kwarg + per-claim check
block (mirrors TITLE_MISMATCH plumbing, sits right after it
in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
`verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value
Live verification (cache-split cleanly via policy-hash bump):
pre-fix cache_key 08dbd2c1… : STRICT (false positive)
post-fix cache_key 6a519636… : UNGROUNDED
Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT
Full suite: 776 passed, 34 skipped.
aborist now writes one JSONL session per `make query` invocation
and per `bench-emergent` cycle to:
~/.aborist/unfirehose/{project-slug}/{session-uuid}.jsonl
Unfirehose's native-harness auto-discovery picks up any
~/.{name}/unfirehose/ directory (see ingest.ts:discoverNativeHarnesses)
without registration — once a session lands, the unfirehose watcher
debounces, ingests, and exposes it in the dashboard alongside
Claude Code / Fetch / uncloseai sessions.
Schema: unfirehose/1.0 (per ~/git/unfirehose-nextjs-logger/docs/
unfirehose-schema.md). Each session file:
line 1 type=session (header — id, projectId, firstPrompt,
harness="aborist", harnessVersion)
line 2 type=message role=user
line 3 type=message role=assistant
content=[text]
model=hermes-3-llama-3.1-8b-fp8-dynamic
provider=hermes
durationMs=<wall>
aborist_meta={audit_mode, n_verified/n_quotes,
cache_key, cache_status, lookup_path,
violations, sources, timings_ms, answer_mode}
line 4 type=message role=system subtype=session_end durationMs
aborist-specific extras (verifier verdict, sources, timings) ride
under namespaced ``aborist_meta`` so the canonical fields stay clean
for off-the-shelf consumers; per the spec, unknown fields are
ignored downstream.
Bench-emergent cycles emit an additional system init message at
the start of each session noting the 3 random words, marking the
session as a generator-driven cycle vs a normal user query.
Failure-isolation: journal write is wrapped in a broad try/except
at every call site. A journaling bug must NEVER break the query
or bench loop.
Tests: 10 new in tests/test_journal.py (slug encoding, session
header, parent-id chain, session_end on close, aborist_meta
passthrough, usage block, idempotent close). Full suite: 663 passed.
Live verified: `make query Q="what is photosynthesis?"` produced
a 4-line JSONL with STRICT 3/3, all sources + timings populated,
ready for unfirehose ingestion.
Three changes that shape the same lever:
(1) The metaphor-cue wordlist now unions /usr/share/dict/words +
/usr/share/dict/american-english + /usr/share/dict/british-english.
The Debian split made the prior 'just symlink to american-english'
miss British spellings (colour, organisation, realise) which
silently became false negatives on British-speaker questions.
Union: 102,485 → 104,305 entries on this machine. ~1,820 added
British-specific entries.
(2) Supplemental dictionary support: operators can layer
domain-specific vocabulary into the morphological substrate.
Two paths:
- Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b
- Programmatic: register_metaphor_dictionary(path)
Each supplemental dict is one word per line. The cue suffix
tests (-ly stem, -ing stem, -est stem) then resolve domain
stems automatically — adding 'aerodynamic' to a custom dict
makes 'aerodynamically' classify as adverbial without code
changes.
Use case: 'a tree with its own vocabulary' — an aviation
forest, a medical corpus, a legal-domain shard each carries
jargon the standard wordlist doesn't cover. Register once,
suffix tests pick up domain stems forever.
(3) README gains a 'Sidecar diagnostics' section with a table of
the three sidecars (deflection, title-relevance, metaphor-
deflection) plus a 'Metaphor-deflection cue dictionary' subsection
explaining the derivation rule, the load order, and the per-
forest vocabulary configurability. Architecturally documents
why the rule is *derived* from the union (Phase-2 lesson) and
not hand-curated.
3 new tests in tests/test_inspect.py:
- register_metaphor_dictionary unions a custom path's words
- ABORIST_METAPHOR_DICTS env var supplements with two paths
- re-registering same path is idempotent
763/34 tests pass.
Closes the FTS5 hyphen-tokenization asymmetry: `bi-polar is rare?`
retrieved only the Bi-Polar album/disambiguation cluster while the
medical-condition cluster (Bipolar disorder, Bipolar I/II disorder,
etc.) sat in the same shards untouched. `unicode61` splits hyphens
at index AND query time; `Bi-Polar Blues` indexes as [bi, polar, ...]
while `Bipolar disorder` indexes as [bipolar] — non-overlapping
token sets that never met.
Fix is query-layer only — no canonicalization_version bump, no
re-index, existing cache_keys stay valid:
- _hyphen_fold_variants(s): emit joined-no-hyphen variants for
every hyphenated run.
- _title_query_tokens(s): additively merges variants symmetrically
(queries AND titles when called on either).
- _filter_by_title_relevance: accept-path 5 — title stem-overlap
with hyphen-fold anchors passes the breadth gate. Rescues
`Bipolar disorder` (1-of-N qtoken match) without disrupting
non-hyphen queries (anchors empty → zero side effect).
- DEFAULT_QUERY_POLICY / DEFAULT_POLICY: hyphen_fold_v1: True
marker folds into governance_policy_hash; new records
cache-split cleanly from pre-fold records.
Live verification on /home/fox/.aborist/shards: same query now
retrieves `Bipolar disorder` (#5) and `Bipolar` disambiguation
(#7); model cites both, answer reads "Bi-polar disorder is not
rare; it affects approximately 2.8% of the U.S. population".
EVIDENCE-WARRANTED 2/2, properly grounded.
Tests: 4 new (3 unit, 1 integration with regression-pinned
Bipolar-disorder retrieval). Full suite 760 passed, 34 skipped.
Also: CLAUDE.md gains a close-when-complete hint for tickets — an
open ticket whose code already shipped is a stale map.
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
Emergent stress-test log surfaced two failure shapes the curated
bench can't reach. Reviewer (claude-opus-4-7 + fox) appended
teacher: blocks to four log entries; two of them named real
defects in the demote chain. This commit ships those fixes plus
the teacher-block annotations.
Cases reviewed (2026-05-02 emergent_log.jsonl):
✅ Menkar/sterilizers/trifle — working as designed
🟡 swallowtail/upbraided/rockiest — metaphorical deflection
🔴 cashback/widescreens/within — warrant gap on inferential claims
🔴 comeliness/fetish/investitures — topic shift earned ANCHOR-WARRANTED unfairly
Two code changes:
1) verify_claim_lattice + verify_claim_lattice_json: when EVERY
resolving claim has TITLE_MISMATCH, demote audit_mode →
UNGROUNDED. Previously HYBRID. Catches the cashback case
('widescreens offer cashback' cited to a generic Coupon
article — n_verified=1 overclaimed when the citation was
meaningless).
2) cli._ladder_rung_for_lattice: DEFLECTION_DETECTED moves from
soft-demote (cap at ANCHOR-WARRANTED) to hard-demote (cap at
POINTER-LINKED) alongside WARRANT_MISSING and TITLE_MISMATCH.
Catches the comeliness case (model fully shifted topic to
'Rock & Chips'; pre-fix the verified Rock & Chips claims
landed at ANCHOR-WARRANTED unfairly).
Test updates:
- tests/test_verify_json.py:test_verify_json_title_mismatch_demotes_to_ungrounded
(renamed; expected outcome flipped HYBRID → UNGROUNDED)
- 4 fixtures in tests/test_claim_lattice.py and 1 in
tests/test_verify_json.py: claim text augmented with 'in the
film' so the Jurassic-Park-titled fixture's title shares a
stem with the claim. These tests were testing other paths
(NO_EVIDENCE_POINTER, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID)
and the title-mismatch was incidental to the fixture choice.
Teacher blocks on the four log entries record the verdict +
failure_class + recommendation + notes, with reviewer + date.
The two ✅/🟡 entries have no recommended code change; the two 🔴
entries reference this commit as the fix.
751/34 tests pass.
Two compounding things in one commit because they're tightly coupled
(the new violation is what makes the new fixtures meaningful):
(1) DEFLECTION_DETECTED — new soft-demote violation kind.
`aborist.qa.inspect.diagnose_deflection` was a read-only sidecar:
when the question's subject anchor (last content token after
stopword strip) is missing from the answer, it returned
`kind="deflection"`. Useful telemetry but never affected the
verdict. 2026-05-02 fox surfaced two cases where the missing
signal mattered:
Q: "who burns the amazon river?"
A: "Deforestation in the Amazon Rainforest is primarily caused
by human activities..." [E10 | Deforestation of the Amazon
Rainforest]
Pre: EVIDENCE-WARRANTED 1/1 ← silent river→rainforest topic shift
Post: ANCHOR-WARRANTED-PARTIAL 1/2 with DEFLECTION_DETECTED
The verifier now runs diagnose_deflection() after all hard checks.
On `kind == "deflection"` it appends a DEFLECTION_DETECTED
violation (rationale + subject_anchor + overlap_ratio) and caps
audit_mode at HYBRID. Renderer's four-rung ladder picks it up via
`_SOFT_DEMOTE_VIOLATION_KINDS` (cli.py) and demotes EVIDENCE-
WARRANTED → ANCHOR-WARRANTED automatically.
Folds into verifier_policy_hash via new
`claim_lattice_deflection_check_enabled: True` policy field
(defaults on). Bumps the 9-dim cache_key when toggled.
Wired into both verify_claim_lattice (pointer) and
verify_claim_lattice_json (JSON). Deferred import keeps verify.py
module-load clean.
(2) Three live fixtures gating the river/rainforest/deflection family:
- test_amazon_river_burning_who_deflects_or_grounds_river
Gates the silent topic-shift pattern: if model deflects, the
verifier MUST catch it (DEFLECTION_DETECTED present OR
UNGROUNDED OR "river" actually in answer). Failure mode this
catches: EVIDENCE-WARRANTED with rainforest answer to a river
question and no soft-demote.
- test_amazon_river_burning_culture_refused_or_ungrounded
Adversarial-impossible mirror of Mars-BDFL: rivers don't burn,
so model must refuse / deflect / refute. Failure mode:
STRICT/HYBRID with affirmative culture-burns-river claim.
- test_amazon_rainforest_burning_culture_grounds_or_deflection_caught
Feasible-but-framing-specific: corpus has deforestation
content but no "culture" framing. Either grounds with rainforest
content OR a soft-demote/UNGROUNDED catches the framing gap.
Live verified (2026-05-02): all 3 pass against the real Hermes
endpoint. 641 unit tests still pass (deflection check skipped on
fixtures lacking question or rendered_text).
Captures the canonical "model knows it but corpus doesn't" case fox
surfaced 2026-05-02:
Q: what is boss baby?
Expected: UNGROUNDED (the 2010-11 wiki snapshot has no Boss Baby
coverage — the 2017 DreamWorks film didn't exist; the 2010 Marla
Frazee picture book had just been published & no article)
Model: emits training-prior content about the film/book
Verifier: 0/N verified — no evidence chunk anchors the claims
This is the failure-by-honest-admission case the verifier must
preserve. Two compounding false-positive risks the test guards:
1. Verifier accepting the claim against an unrelated chunk that
happens to share content tokens (Death of Baby P, Cake Boss,
Beanie Baby — all adjacent-token noise observed at top-K).
This is the warrant-failure class warrant-lite was built to
catch — if it ever passes here, anchor-class warrant has a hole.
2. Verifier accepting via OR-mode FTS5 noise — a token-level match
of "baby" or "boss" in unrelated content surfacing as positive
grounding.
Acceptable behaviors:
A. UNGROUNDED (verifier returns 0/N — corpus-truth over prior)
B. STRICT/HYBRID without "boss baby" anywhere in the answer
(deflection — model declined the false-corpus question and
answered an adjacent grounded fact, like Mars-BDFL → Guido
pattern).
Failure: STRICT/HYBRID + "boss baby" in answer = false-positive
grounding. Test fails by name pointing at the warrant gap.
Live verified (2026-05-02): UNGROUNDED 0/2, model emitted both the
2017-film claim AND the 2010-book claim, verifier rejected both
honestly. Test passes in 10s.
Regression fox surfaced 2026-05-02:
Q: "what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without
speaking or sign language."
→ sqlite3.OperationalError: Expression tree is too large
(maximum depth 1000)
Root cause: the v1 of _search_titles (commit 0052845) chained N
``CASE WHEN ... THEN 1 ELSE 0 END + ...`` expressions for the
title_score column. Each CASE WHEN is multiple tree nodes; +-
chained N times exceeded SQLite's default 1000-depth bound on
question texts with ~30+ content tokens.
Fix: simplified SQL — OR-chain WHERE + ORDER BY LENGTH(title) ASC
+ LIMIT bumped 4x to compensate for the lost smart sorting. The
caller's post-filter (word-boundary stem-aware token-set
intersect) does the actual title-relevance ranking; SQL just
needs to surface enough candidates for the post-filter to grade.
Also: cap the OR-chain at MAX_TITLE_LIKE_TOKENS=24 so pathological
200-token queries don't cascade SQL expression growth even
defensively. Beyond ~24 tokens the post-filter is doing all the
work; extra LIKEs just inflate candidate sets without signal.
5 new tests in tests/test_query.py covering the regression at
unit / integration / functional layers:
- test_unit_search_titles_handles_long_question_without_crash
50-token query through _search_titles directly. Pre-fix raised
sqlite3.OperationalError; post-fix returns row list.
- test_unit_search_titles_handles_zero_tokens
Defensive: empty token list → empty result, no SQL executed.
- test_unit_search_titles_caps_or_chain_at_max_tokens
200-token pathological query — bounded by MAX_TITLE_LIKE_TOKENS,
doesn't crash.
- test_integration_query_completes_on_long_question
End-to-end query() with StubClient + long question completes
without the SQLite error. Pre-fix raised before reaching the
LLM call.
- test_functional_long_question_returns_sources
The neurotech doc (richest body match) appears in top-K despite
the long-question retrieval path.
Test_query.py: 35 → 40 passing. Full suite (excluding parallel
test_concepts churn from concepts/query.py rewrite): no other
regressions.
Two paired changes addressing fox's "learn this hyperparameter
from model use, not hard coding" + the 1M-context-window
caveat:
(1) Log-scale prompt-size buckets extend from 8KB through 1M+:
<8KB / 8-16KB / 16-32KB / 32-64KB / 64-128KB / 128-256KB /
256-512KB / 512K-1M / >=1M
The same bench harness now covers 8B-class models (Hermes 82K
context, max useful prompt ~32-64KB) through 1M-context models
(Gemini 1.5 Pro, Claude with extended context, Llama 4) without
code change. A model whose context window stops at 82K simply
never populates the giant buckets; a 1M-context model fills
them and finds its own sweet spot.
(2) New "recommended context budget (learned from this bench)"
section — per-mode peak-strict-rate bucket. Operator-driven
landing per the five-step algorithm step 5: surfaced, not
auto-applied. Minimum sample size of 5 runs per bucket so
statistical noise doesn't masquerade as signal. Tie-break on
smaller-bucket-wins so equivalent strict-rates favor the
cheaper choice.
The substrate is now self-tuning at the OBSERVATION layer: bench
records what budget actually grades best per model. Per-model
profile JSON (storing the recommended budget back into
~/.aborist/model_profiles/<model>.json) is the next beat once
this surface is observable in real bench runs.
3 new bench tests cover the recommended-budget section, the
sample-size floor, and the giant-context bucket coverage.
Full suite: 746 passed (was 738, +8).
Connects to D8 (automate after test-pinning): the bench tells us;
we don't guess.
Phase 2 of the concepts/ layer. The Apr 27 commit (c6182ae) shipped
hand-curated frozensets in aborist/qa/concepts.py with a TODO to
"derive from Wikipedia's category graph or 'See also' sections" —
that's this commit. Fox's 2026-05-01 critique landed it: a 7-entry
list of arbitrary frozensets won't scale to a 3.47M-doc corpus, &
adding domains shouldn't require a Python edit + commit + redeploy.
Architecture:
(1) Per-shard concept_relations SQLite table. Append-only, with
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
so re-derivation is idempotent. Lives next to documents in each
shard so mesh sync moves relations alongside the docs that
derived them. A SECONDARY index — writes here NEVER affect
document_root / chunk_root / cache_key, so backfilling is safe
across the entire corpus.
(2) aborist/concepts/ package:
- store.py: add_concept_relation, concept_relations_for_token,
purge_by_evidence_kind, list_evidence_kinds
- query.py: cross-shard synonym_expand, rivalry_excluded;
union-find collapse on synonym edges so partial
pairs build full equivalence classes; mtime-keyed
per-process LRU so retrieval doesn't re-walk
shards on hot loops
- seed.py: one-shot migration of legacy frozensets (8 groups
incl. brain-tech) to evidence_kind='manual_legacy'
rows under source_root='__legacy__concepts__'
- extract.py: pluggable extractor registry. Built-in:
link_reciprocity_synonym — for any reciprocal
edge pair (A→B AND B→A) in the existing edges
table, emit synonym edges between the docs'
title-tokens. Works for Wikipedia (See-also
bidirectional), HTML site internal-link clusters
(russell.ballestrini.net pattern), or any link
graph the corpus already encodes — no new
crawler needed; the html_page parser already
populates `edges` rows on ingest.
(3) aborist/qa/concepts.py rewritten as a backwards-compat shim —
same public API (synonym_expand, rivalry_excluded,
has_compare_phrasing) so query.py call sites unchanged.
shards_dir threaded through _filter_by_title_relevance &
_search_corpus' synonym_expand calls. Without shards_dir
(legacy 2-arg call shape), helpers degenerate to no-op —
matches the behavior the frozenset code had when no group hit.
(4) Cross-shard UNION view: concept_relations added to
_SHARDABLE_TABLES in store.py so connect_query() exposes a
unified view across all shards (same pattern as documents,
chunks, providence_cache, etc.).
Live-verified on the 4-shard 3.47M-doc corpus + the
crawl_russell_ballestrini_net.db shard:
Q: what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without speaking
or sign language?
Result: same as the bde1bd6 in-memory frozenset version —
Telepathy E4 cited alongside Videoconferencing E1/E2,
POINTER-LINKED-PARTIAL 6/10. The DB-backed lookup reproduces
the frozenset behavior byte-for-byte.
Tests: 633 passed (was 624). 9 new concept tests covering
DB-backed synonym/rivalry lookup, shards_dir=None degenerate
behavior, brain-tech group seed, cache invalidation. Old tests
that called helpers directly without shards_dir kept as no-op
assertions (synonym_expand({"athlon"}) without shards_dir returns
{"athlon"} unchanged).
Backfill mechanics: existing live shards needed a one-shot
`connect()` to auto-create the new concept_relations table
(SCHEMA_SQL has CREATE TABLE IF NOT EXISTS). Re-derivation never
mutates the Merkle tree — concept_relations is fully orthogonal
to document_root / chunk_root. Cache_key dimensions are
unaffected.
Deferred (follow-on):
- CLI commands: aborist concepts {seed,list,add,derive,purge}
(currently fox runs the helpers via python -c)
- Wikipedia See-also extractor (requires parsing wikitext sections
beyond what's already in edges)
- Wikipedia category extractor (requires reading Category: links
from chunked wikitext)
Promotes the title-relevance sidecar (diagnose_title_relevance,
qa/inspect.py) to a hard check inside both verify_claim_lattice
and verify_claim_lattice_json. New violation kind TITLE_MISMATCH;
demote-to-HYBRID semantics matching the existing WARRANT_MISSING
pattern.
Catches the 2026-05-02 fox-surfaced retrieval-driven hallucination
class:
Q: "explain spin glass modeling & tensors?"
Pre-Rule-8: EVIDENCE-WARRANTED 1/1, claim cited to Quantum
chromodynamics chunk (single-line "See Also: spin
glass" reference). Token-coverage check passed
accidentally on shared physics vocabulary.
Post-Rule-8: POINTER-LINKED-PARTIAL · title mismatch 1/1.
Cited source title 'Quantum chromodynamics' shares
zero stemmed tokens with claim's {spin, glass,
modeling, tensor, ...} → demote.
Implementation:
_claim_title_overlap(claim_text, source_title) returns True iff
the source title shares ≥1 stemmed content token with the claim.
Uses qa.evidence._content_tokens (≥4-char, post-stopword) and
inline minimal stem (s-strip on tokens >4 chars, skip ss-enders)
to avoid an import cycle.
Per-claim loop in both verifiers checks every cited source's
title; ANY-match suffices (only TITLE_MISMATCH when ALL cited
titles miss). Vacuous-pass when claim or title has no
extractable tokens.
Renderer (_ladder_rung_for_lattice) treats TITLE_MISMATCH
alongside WARRANT_MISSING as the POINTER-LINKED-triggering
signal — both indicate citation/claim structural misalignment.
_render_warrant_tail surfaces "· title mismatch" alongside
"· warrant missing" so an operator sees the specific failure
mode at the audit-line.
Test fixtures updated where the synthetic claims were too minimal
(e.g. "Velociraptors are shown attacking workers" cited to
"Jurassic Park (film)" — claim had no topic anchor). Real model
output naturally references the topic (the model sees the title
in the evidence map and uses it); the fixture revisions reflect
that. 4 new tests in test_verify_json.py covering the helper +
end-to-end TITLE_MISMATCH demote.
Live verification:
Spin-glass query: POINTER-LINKED-PARTIAL · title mismatch ✓
Homer/Mr. Burns: EVIDENCE-WARRANTED ✓ (no regression)
CLAUDE.md updated with the Rule 8 convention; existing
diagnose_title_relevance sidecar marked legacy / dict-form for
per-cache-key inspect use.
Full suite: 738 passed (was 734, +4).
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
Each directive gets a structural pin in tests/test_directives.py.
A future PR that silently weakens a directive fails by name.
D1 (no LLM in hard verifier path):
- verify_quotes / verify_claim_lattice / verify_claim_lattice_json
signatures must NOT carry chat_client / llm / judge / model
parameters.
- SCHEMA_SQL verifier_method CHECK constraint enum excludes
'llm', 'model', 'judge', 'nli', 'hermes', 'ai'.
D2 (Hermes emits pointer clauses):
- ANSWER_MODES contains both lattice variants.
- DEFAULT_ANSWER_MODE in ANSWER_MODES.
- parse_pointer_claims returns structured (claim_text, pointer_ids)
nodes from pointer-line input.
D3 (build CTI internally):
- Runtime parses pointer-line prose into typed claim nodes
(text + pointer_ids list); model never produces structured
output directly in pointer mode.
- Evidence map built by runtime from retrieved chunks; pointer
ids minted sequentially; evidence ids content-addressed.
D4 (bind retrieval AND evidence map):
- build_run_dag accepts evidence_map_root parameter (✓).
- Marker test: build_run_dag does NOT yet accept retrieval_plan_hash
parameter; flip when ticket #000001 lands.
D5 (deterministic pointer verify):
- verify_claim_lattice is deterministic: same inputs → same
verdict byte-for-byte.
- _claim_textually_overlaps_evidence (the per-pair coverage
check) is a pure function with no LLM-side parameters.
D6 (anchor-class warrant before NLI):
- warrant_check signature is lexical-only (no chat_client / llm
/ judge / nli params).
- Date-anchor failure case fires (year missing from cited spans).
- Relation-shape failure case fires when ALL proper-noun anchors
are missing from cited spans.
- Marker test: per-shape detectors for entity-list / count /
why-cause shapes are absent today; flip when ticket #000003 lands.
D7 (rename labels honestly):
- Renderer maps STRICT → EVIDENCE-LINKED for claim_lattice* methods.
- Renderer keeps STRICT for quote / span / entity / paraphrase
methods (those verify against pinned spans, not synthesis).
- SCHEMA_SQL audit_mode CHECK constraint enum stays
{STRICT, HYBRID, UNGROUNDED} regardless of renderer relabel.
D8 (test-pinning before automation):
- docs/seven-point-program.md exists.
- bench/qa_sweep.py exposes _directive_compliance helper.
- docs/TICKETS.md indexes the partial directives (D3, D4, D6) so
the design log doesn't drift from the program doc.
23 new tests; full suite 692 passed (was 669, +23).
The "marker" tests for D4 and D6 document open work: today they
assert that a parameter / detector is ABSENT; when the corresponding
ticket lands, the assertion flips to PRESENT. The test failing
during ticket implementation is the expected signal, not a
regression — the test message names the ticket so the work-in-
progress is legible.
Wires the seven-point program (docs/seven-point-program.md) into
the bench harness as a per-mode coverage table.
Per-row computation in _directive_compliance(answer_mode, result, err):
D2_pointer_clauses — answer_mode is lattice variant
D3_cti_substrate_ready — lattice mode + run_dag_root populated
(full coverage pending ticket #000002)
D4_evidence_map_bound — run_dag_root populated
(retrieval-plan binding pending #000001)
D6_warrant_fired — warrant ran (lattice mode today;
per-shape gating pending #000003)
D7_honest_label — audit_mode in canonical enum
(renderer transformation pinned in
test_cli_render.py)
D1 (no LLM in verifier), D5 (verifier_method enum), D8 (test-pinning
discipline) are global properties of the substrate and don't appear
per-row — they get tracked once in CLAUDE.md / the program doc.
_summarize aggregates per-row booleans into per-mode pass counts.
_render_markdown emits a 'directive coverage (seven-point program)'
section with per-mode per-directive coverage as 'count/N (pct%)'.
Discipline rationale per CLAUDE.md's five-step algorithm step 5:
the bench harness IS the automation substrate, so it must enforce
the directives before any feature gets layered on top. Bench scores
that climb without directive coverage climbing are graveyard-digging
(step 4: don't dig the grave faster).
5 new bench tests cover: per-mode pass aggregation, the markdown
section presence and content, the helper's behavior on quote mode
(D2 fail by design), lattice mode (all directives pass), and error
rows (empty dict — no signal). Full suite 669 passed.
Generalizes warrant-lite from relation-only to anchor-class warrant.
Two anchor classes now compose:
(1) Proper-noun anchors — gated on relation-question shape; any-match
semantics (existing behavior).
(2) Date anchors — always-on, gated on the CLAIM containing a year or
month name. ALL components (year + month name if present) must
appear in some cited span.
Closes the back-to-the-future failure fox surfaced 2026-05-01:
Q: what date did back to the future come out?
Claim: "Back to the Future was released in theaters on July 3, 1985."
Cited span: trilogy/SNES/pinball prose with '...real 1985...' but
no 'July' anywhere
Pre-fix: EVIDENCE-LINKED · via claim_lattice 1/1
Post-fix: EVIDENCE-LINKED-PARTIAL · via claim_lattice · warrant missing 1/1
Year extraction (4-digit, 1500-2199 range — avoids matching ZIP codes
/ elevations / model IDs) and month extraction (full English month
names — three-letter abbrevs deferred until bench shows the gap).
Both anchors required when both are in the claim. Year-only claim
against year-only span continues to pass; month-only against
month-only continues to pass; month-and-year claim against year-only
span (the BTTF case) now correctly fails.
Architectural note: this generalizes warrant-lite without building a
typed-contract framework (per-question-type rule libraries like
"release_date_lookup with required predicate cues"). The date anchor
primitive catches the date lazy-anchor failure class deterministically
without a per-type rule book; typed contracts earn their slot only if
bench evidence shows general anchors miss specific cases. Dropped the
caller-side is_relation_question gate from verify.py — warrant_check
now decides internally what to enforce based on question shape AND
claim content.
Tests: 29 unit tests in tests/test_warrant.py (was 16). New tests
cover extract_date_anchors (year, month, dedup, non-year-skip,
empty), warrant_check date semantics (BTTF case, year-only,
year-in-unrelated-context, month+year composition), proper-noun
warrant skipped without relation question. Full suite: 554 passed,
21 skipped.
Limitation accepted: ISO date format ("1985-07-03") wouldn't satisfy
month-name requirement. Real-world Wikipedia prose uses month names
("released on July 3, 1985"), so the trade-off favors catching the
common case. If bench shows ISO false-rejections, layer in a
month-number alternative to the month-name regex.
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Two UX fixes addressing the 2026-05-01 Orwell run feedback:
(1) Evidence pointer format. Old form `[E5: "<excerpt>"]` looked
visually like a 1-indexed source rank — paired with a "sources (8)"
list whose `[5]` slot was a different document, operators easily
mis-attributed citations. New form:
[E5 | Nineteen Eighty-Four | 682f0a11: "<excerpt>"]
Title comes from EvidenceObject.title (URI-tail fallback when None);
chunk_root prefix is the first 8 hex chars — enough to disambiguate
chunks from the same source while staying compact. Renderer change
in render_claim_lattice; pre-existing claim-lattice tests updated;
3 new unit tests pin the format (title-inline, URI-tail fallback,
no `[E1:` collision-shape).
(2) Audit-label discipline. _render_audit_label already maps STRICT
→ EVIDENCE-LINKED for claim_lattice* modes (verifier output stays
binary, schema CHECK constraint stays); 4 new tests in
test_cli_render.py pin the relabel for STRICT / HYBRID / UNGROUNDED
in claim_lattice modes and the no-relabel for quote-mode where the
verifier verifies pinned spans not synthesis.
Net: 8 new tests, full suite 649 passed.
Closes the lazy-anchor failure class fox surfaced in feedback-3
(2026-05-01) on "who is homer simpson's boss?":
Claim: "Homer Simpson's boss is Mr. Burns."
Cited span: voice-actor / Castellaneta bio prose
Pre-warrant: pointer-linked, source-role allowed, coverage passes
Post-warrant: WARRANT_MISSING violation, audit_mode capped at HYBRID
Pointer verification != warrant verification. A pointer resolves to
a span; a warrant requires the span to actually contain the claim's
named answer entity.
Adds aborist/qa/warrant.py:
- is_relation_question(q) — regex shape detector for "who is X's Y?",
"who is supermans girlfriend?", "who founded Z?", "boss of X" etc.
- extract_answer_anchors(claim) — multi-word proper nouns ("Mr.
Burns", "Lois Lane") with possessive 's stripping; solo-cap
fallback that skips sentence-starter false positives
- warrant_check(claim, spans) — at-least-one-anchor lexical match;
vacuous-pass when claim has no proper-noun anchor
Wired into verify_claim_lattice via new params (question,
warrant_check_enabled). Per-claim WARRANT_MISSING violations cap
audit_mode at HYBRID using the same demote pattern as
lazy_anchor_demoted. CLI renders " · warrant missing" tail when
violations present so the failure mode is legible without scanning
the violations array.
Policy field claim_lattice_warrant_check_enabled added to
DEFAULT_POLICY / DEFAULT_QUERY_POLICY and the verifier-policy hash
field set so changes bump verifier_policy_hash.
Hard or soft? Per CLAUDE.md "Soft hash vs hard hash":
substring/lexical = hard. Output is binary (warrant present or not),
violations enter the proof path through violations[] alongside
SOURCE_ROLE_BLOCKED and CITATION_MISMATCH.
Deferred (follow-on commits):
- full RELATION_PATTERNS table with relation-noun aliases
- two-hop bridge detection ("X founded Y" + "Y is Z" -> "X is Z")
- retrieval expansion for relation questions
Tests: 16 new unit tests in tests/test_warrant.py covering
is_relation_question, extract_answer_anchors, warrant_check
(happy path, lazy-anchor case, vacuous-pass, case-insensitive,
empty inputs). Full suite: 523 passed, 21 skipped.
Implements 6 of fox's feedback-2/3 items (2026-05-01) targeting the
red-fish-blue-fish output. Each lives in the render or sidecar layer
— no schema or cache-key change beyond the verifier_policy_hash that
landed earlier in the day.
(1) Honest audit-label rendering. Schema audit_mode (STRICT/HYBRID/
UNGROUNDED) describes the lexical verifier's call; the user-
facing label spells out what was actually checked. Mapping for
claim_lattice modes:
STRICT → EVIDENCE-LINKED · via claim_lattice
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice
UNGROUNDED → UNGROUNDED · via claim_lattice
Quote / span / entity / paraphrase modes keep the audit_mode
token plus a `· via {method}` tail. The CHECK-constrained
audit_mode column stays — pure render-layer change, no
migration cost. Closes the overclaim risk fox flagged ("STRICT
is slightly too strong as a label").
(2) Sentence-bounded evidence spans. _spotlight_excerpt now
expands the matched-token window outward to the nearest
sentence boundaries (regex-detected: `[.!?][\"')\\]]?\\s+(?=[A-Z])`)
instead of cutting at fixed byte offsets. Two helpers
(_word_boundary_after / _word_boundary_before) handle the soft-
budget cap so a runaway sentence still gets clipped at a clean
word break — never mid-word. Pre-fix the renderer produced
excerpts like `"...freewheeling plot... friends and p..."` (the
`p...` is the offending mid-word truncation fox surfaced); now
the excerpt is `"One Fish Two Fish Red Fish Blue Fish is a
1960 children's book by Dr. Seuss. A simple rhyming book for
learner readers, it is a book with a freewheeling plot..."`.
(3) Source-role display + used/unused annotation. The CLI source
list went from
[1] One Fish Two Fish Red Fish Blue Fish — wikipedia.org/...
[2] Red Dwarf — wikipedia.org/...
to
[1] One Fish Two Fish Red Fish Blue Fish — primary_answer_source — used (E1) — ...
[2] Red Dwarf — background_source — unused — ...
Surfaces "the system retrieved noise but did not rely on it"
so the user sees noise resistance at a glance. Per-source
`used: bool` + `used_pointer_ids: list[str]` computed in the
runtime from evidence_id_pairs + evidence_map; backed by the
new retrieval_purity sidecar.
(4) Retrieval-purity sidecar. Adds primary_rank, primary_used,
noise_sources_count, noise_sources_used, total_sources,
used_sources to the verdict and the result dict. Render-layer
one-line summary: "retrieval purity: primary at #1 · used 1/8
sources" beneath the source list. Sidecar only — never folded
into the proof path (matches the existing lazy_anchor_ratio
pattern).
(5) Ordered-token title scoring. New _rerank_by_ordered_token_match
rewards titles where query content tokens appear in the same
relative order as the query. "red fish blue fish" query →
"One Fish Two Fish Red Fish Blue Fish" gets ordered_match=4
boost (1+0.5*3 = 2.5×); "Red Dwarf" stays at ordered_match=1
(no boost). LCS-based, stem-aware, deterministic. Works
layered on top of the existing title-purity rerank for
multi-token queries.
(6) Noise-resistance fixture (tests/test_qa_quality_live.py).
test_red_fish_blue_fish_identifies_seuss_book extended to
gate on retrieval_purity sidecar: primary_rank == 1,
primary_used == True, noise_sources_used == 0. Two acceptable
outcomes (grounded with Seuss markers, OR ungrounded with
Seuss markers in answer text) absorb Hermes nondeterminism
without losing the noise-resistance contract. Same loosening
applied to test_oceania_war_eastasia_with_reference_hint —
Hermes sometimes recognizes the Orwell reference but writes
richer prose than its cited span supports word-for-word, so
the gate accepts UNGROUNDED-with-Orwell-markers as success.
Live test_qa_quality_live.py: 20/20 fixtures pass. 507 unit tests
green. Working tree leaves bench/qa_questions.txt for fox's
parallel WIP.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Three layers, 14 new tests, full suite 611 passed (was 597):
UNIT — tests/test_query.py
test_query_returns_prompt_chars_breakdown
Asserts the result dict's prompt_chars carries exactly the five
expected keys and messages_total equals sum of message contents
the StubClient saw.
test_query_answer_chars_matches_answer_text
answer_chars == len(answer_text) — drift check.
test_query_cache_hit_also_returns_capacity_metrics
Cache-hit path populates prompt_chars + answer_chars (operators
inspecting cached records still want the breakdown).
test_query_evidence_chars_grows_with_topk
Sanity: more sources / larger budget → more evidence chars
(the metric tracks actual context build, not a stale constant).
INTEGRATION — tests/test_query.py (retrieval_keywords)
test_retrieval_keywords_does_not_alter_question_to_llm
Keywords don't appear in the LLM-facing question segment;
system prompt unchanged across runs. Pins the substrate
contract: keywords are FTS5/title-filter-only.
test_retrieval_keywords_changes_retrieved_sources
Different keyword sets surface different docs (the actual
user-visible behavior).
UNIT — tests/test_cli_render.py
test_render_shows_capacity_line_when_prompt_chars_present
Capacity line appears with messages_total + breakdown when
prompt_chars is in the result dict.
test_render_omits_capacity_line_on_legacy_results_without_prompt_chars
Backwards-compat: legacy results render cleanly without the
capacity line — no KeyError, no '0 chars' noise.
test_render_capacity_thousand_separators
61,550 not 61550 — operator legibility on daily renders.
UNIT/INTEGRATION — tests/test_bench_qa_sweep.py (NEW FILE)
Imports bench/qa_sweep.py via importlib.util so the module's
not in the Python path doesn't matter. Five tests:
- _summarize counts verdicts by mode
- deflections counted only on STRICT/HYBRID rows (not UNGROUNDED)
- rendered markdown has the headline summary + size buckets
- size buckets correctly stratify strict-rate by prompt_chars_total
- empty buckets are skipped (no '0 runs' noise)
FUNCTIONAL — live verification (no automated test, manual)
`make query Q="what is the capital of france?" BURN=1` confirmed
in commit f927298 to render the capacity one-liner under the
source list. Documented in that commit's body.
Also corrected the docstring on query()'s `retrieval_keywords` to
reflect that keywords don't enter cache_key DIRECTLY but do change
context_root + conversation_hash via source selection — so the same
question with different keywords lands under different cache_keys
(legitimately, since the LLM saw different contexts).
Empirical finding 2026-05-01 on 'explain spin glass modeling, tensors?':
returned STRICT 1/1 with claim 'Spin glass modeling involves...
mathematical tools such as tensors' cited to a chunk from the
*Quantum chromodynamics* article. Token-coverage check inside the
chunk passed accidentally on shared physics vocabulary
(spin/model/etc.), but the article isn't about spin glasses — the
claim is a retrieval-driven hallucination.
Root retrieval cause: AND-mode FTS5 returned zero hits (Spin_glass
article has 0 occurrences of 'modeling' or 'tensors'); OR-mode
fallback ranked by raw BM25, so 'Glass' (glass*274) outranked
'Spin glass' (spin*29 + glass*15) despite Spin_glass matching 2
distinct query tokens vs Glass's 1. Verifier then accepted the
hallucinated claim because the cited QCD chunk has incidental
physics vocabulary overlap.
New ``diagnose_title_relevance(claim_text, cited_source_titles)``
sidecar: claim's content tokens (stemmed) must overlap the union
of cited source titles. Zero overlap → ``title_mismatch``. Catches
the spin-tensors case (claim about spin glass, cited title 'Quantum
chromodynamics' → zero stem overlap) without false-positives on
legitimate cross-document grounding (Lois_Lane article cited for a
Lois-Lane claim still has lois+lane overlap).
Sidecar-only for now — staying out of the binary verifier output
until bench evidence justifies hard-promotion. Not yet wired into
inspect_cache_key (per-claim citation source mapping isn't in
providence_cache yet) or the bench. Available as a utility for
operators auditing specific (claim, source_title) pairs.
5 new unit tests covering match/mismatch/no-titles/no-claim-tokens
and underscore-normalization for Wikipedia titles.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
Bench dig (2026-05-01, 65-question sweep): 4 of 7 deflecting
questions were false positives from verb-shape subject anchors.
what year did the berlin wall fall? → answer "1989" (anchor "fall")
when did the soviet union dissolve? → answer "1991" (anchor "dissolve")
how many wives did henry the eighth? → answer "six" (anchor "eighth")
what year does our cold fusion ...? → answer "1989" (anchor "happen")
why did the titanic sink? → "...sank in 1912" (anchor "sink" vs "sank")
why did the dinosaurs go extinct? → "asteroid impact" (anchor "extinct" vs "died")
These shapes ask for a year/count/cause; the answer is structurally
not expected to echo the question's subject. Pre-suppression, all
three modes landed deflection on these — the apparent
"pointer-only" deflection rate (10.3% vs 6.2%) was almost entirely
heuristic noise, not a real substrate failure.
New ``shape_suppressed`` flag returned by diagnose_deflection. When
the question starts with one of:
when, what year, what date, what time,
how many, how much, how long, how old, why
the subject-anchor check is skipped — overlap-ratio takes over.
Mars-BDFL (who-shape) still fires correctly.
Field exposed so the operator can filter the signal: a
shape_suppressed=True row that still classifies as deflection on
overlap-ratio means "answer was a bare year/count/cause with no
topic words" — known noisy, not real topic shift.
5 new unit tests; full suite 582 passed (was 578).
Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.
Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.
Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.
Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.
