Cross-references the 2026-05-03T23-30-12Z full 75-question
regression bench (Addendum 3 in qa-modes-bench.md) into:
- #000010 §13.2 — full-bench validation alongside the §13.1
broad-subset validation. Same verdict: defaults stay on.
Substrate-level wins beyond noise-bounded SR deltas:
pointer FORMAT_COLLAPSED: 0/225 (eliminated globally)
pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only)
JSON mean ratio: +3pp
- seven-point-program.md addendum — bench-validation paragraph
after the test-count line. Names the SR deltas + the
substrate-level wins; concrete evidence the post-landing
directive coverage claim survives full-corpus measurement.
No code changes; 993 tests still passing.
Addendum 3 to qa-modes-bench.md. Validates that flipping
quantifier_reminder_enabled=True for lattice modes (per #000010
§12.10 / §13.1) doesn't regress narrow-question performance.
Prior validation (Addendum 2) covered the 9-question broad subset
only. This run sweeps the full 75-question bench/qa_questions.txt
(~10% broad, ~89% narrow), 225 runs per mode, comparing against
the frozen 2026-05-02T15-07Z baseline.
Findings:
Mode | Pre-flip SR | Post-flip SR | Δ
----------+-------------+--------------+--------
quote | 0.54 | 0.52 | -2pp (within 5pp floor)
pointer | 0.20 | 0.21 | +1pp
JSON | 0.42 | 0.44 | +2pp
Mean ratio: -1/+2/+3pp — all within noise band.
Substrate-level wins beyond the headline metrics:
- pointer FORMAT_COLLAPSED: 0/225 across the full sweep.
Reminder eliminates collapse mode globally, not just on broad.
- pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when
reminder was off). Citation discipline propagates beyond the
rows where the reminder text actually fires — the model's
session attention reinforces.
- JSON mean ratio +3pp consistent with broad-subset finding.
Quote-mode is essentially unchanged because it's mode-gated off
the guard by default.
Latency 33-35s/call this evening (vs 17-19s in prior runs) is
endpoint contention, not a substrate regression — preflight adds
zero LLM calls.
Verdict: no regression. The default flip ships clean across the
full corpus. Defaults stay on. The substrate is strictly more
honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without
sacrificing throughput on non-broad questions.
Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}.
Post-landing validation cell. Same 9-question broad subset as
§12.6 reminder-only baseline; this run flips both
metacognition_enabled and quantifier_reminder_enabled to False so
we can isolate the preflight contribution.
Metric | OFF | ON | Δ
--------------------------+-----------+----------+----------
pointer mean ratio | 0.483 | 0.643 | +16pp
JSON mean ratio | 0.570 | 0.735 | +17pp
JSON UNGROUNDED rate | 7/27 | 1/27 | -22pp
pointer FORMAT_COLLAPSED | 2/27 | 0/27 | -100%
any-mode STRICT-rate | within ±11pp noise (27-sample n=3 floor)
Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all clear the
5pp signal floor on lattice modes. STRICT-rate moves are within
Hermes nondeterminism. The #000010 default flip is doing what
was claimed.
On this subset none of the metacognition detectors fire (no
temporal / contradiction / false-premise / out-of-corpus shapes),
so the delta effectively isolates the reminder contribution from
#000008. A metacog-trigger subset bench is deferred.
Documentation:
- docs/qa-modes-bench.md Addendum 2 captures the comparison
table + interpretation + verdict.
- docs/tickets/ticket-000010-... §13.1 cross-references with
bench artifact stamp.
Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}.
Three Explore agents fanned out in parallel for a docs/ + diagrams/
+ code-comment audit against the shipped state of the three
preflight tickets. This commit lands all the alignment fixes.
Core docs updates:
CLAUDE.md
- dag.py module description: stage counts now read
"7/8 quote · 9/10 CTI · 3 reject" reflecting #000009 preflight
stage + reject-broad early-return shape.
docs/cti-architecture.md §2.2 + §2.3
- §2.3 Merkle-AGI-DAG section rewritten: documents all five DAG
shapes (legacy 7/9, post-#000009 8/10, reject-broad 3),
describes the preflight stage's 5 nested CTI clauses
(classifier / answer_contract / prompt_contract /
evidence_contract / policy_refs), pins
PREFLIGHT_NODE_VERSION = "preflight-node-v1", states the
audit-replay payoff.
- §2.2 CTI section: adds the four new modules
(quantifier, model_profiles, quantifier_reminder,
metacognition) as code anchors. Notes that pre-answer
preflight contract extends CTI upstream of retrieval.
docs/seven-point-program.md
- D3 status ½ → ¾ — pre-answer preflight contract landed via
#000008 + #000010. Code anchors + pinning tests updated.
- D4 status ½ → ¾ — preflight stage adds upstream control
commitment to the run-DAG. Code anchors include
build_reject_run_dag + preflight_node_hash.
- Status snapshot table: tickets column now references
#000008/#000009/#000010 against D1/D3/D4 directives.
- "Post-landing addendum (2026-05-03 / 2026-05-04)" subsection
summarises all three tickets + their commit shas + final
test count (993 passing, up from 734).
docs/modules.md
- Q&A pipeline table: added 4 new modules (quantifier.py,
model_profiles.py, quantifier_reminder.py, metacognition.py).
dag.py row updated to "7/8 quote · 9/10 CTI · 3 reject".
- dag.py subsection rewritten: documents all 5 DAG shapes,
describes the preflight payload's 5 clauses + question_state.
- 4 new module subsections (quantifier / model_profiles /
quantifier_reminder / metacognition) explaining each
module's purpose, signature, and how it feeds the run-DAG
preflight clause.
Diagram updates:
docs/diagrams/query-pipeline.dot + .svg
- New "PREFLIGHT (#000008 + #000010)" node inserted between
cache_check and concepts_lookup.
- New "REJECT-BROAD" node showing the 3-stage minimal DAG
escape path.
- render node label extended with the audit-line tail token
catalog.
docs/diagrams/aborist-modules.dot + .svg
- 4 new qa_* nodes in the retrieval & verifier cluster.
- 8 new edges: qa_query/qa_runner each call into all 4
preflight modules; qa_dag has dotted edges to qa_quantifier
+ qa_metacognition (preflight clause sources).
- qa_dag label updated to mention preflight_node_hash + 5 clauses.
docs/diagrams/verifier-ladder.dot + .svg
- Soft-demote violations list extended: BROAD_QUANTIFIER_RUNAWAY
/ CAP_APPLIED / SCOPE_UNBOUND, FORMAT_COLLAPSED, BARE_NAME_CLAIM.
- New "AUDIT-LINE TAILS" annotation node listing all 11 tail
tokens (#000008 broad-* + #000010 metacog + classic verifier).
- Dashed edges from each rung to tails note showing tails
compose onto labels.
Code-side stale-comment fixes (caught by 3rd Explore agent):
aborist/qa/keys.py:218
- "The four fields" → "The seven fields"; mention #000010 adds
six more for metacognition.
aborist/qa/query.py:2644
- 7-stage / 9-stage comment expanded to enumerate all four
base+preflight shapes plus the 3-stage reject path.
aborist/qa/runner.py:835
- same expansion as query.py for runner.ask() callsite.
mesh-*.dot, ingest-pipeline.dot, qa-modes-bench.md, bench-maxing.md,
bench-emergent-design.md, verifier-semantic-gap-design.md,
self-reference-design.md, concept-relations-design.md confirmed
orthogonal — no edits needed.
993 tests still passing (no behavior change). 7 files modified
across docs/ + 3 dot diagrams + 3 SVGs + 4 code-comment fixes.
Architectural feedback at ~/Downloads/RESPONSE-ticket-000009-... .txt
(2026-05-04) flagged five gaps in the c36e85c landing. Most
critical: reject-broad early-return path emitted no run_dag_blob,
so audit replay couldn't see that a rejection happened (let alone
under what policy state).
A — reject-path DAG (the critical gap):
aborist/qa/dag.py: build_reject_run_dag() — 3-stage minimal DAG
question → preflight → final_label. final_label payload carries
rejection_reason + answer_text_hash so two rejections under
different policy state produce different roots.
query.py reject path now wires it: returns run_dag_root +
run_dag_blob on the rejection result dict. Live-verified end-
to-end on `make query Q="winners of all major sports?"
REJECT_BROAD=1 BURN=1`.
Audit replay rule: 3 stages always means reject path. Operators
can read the stage list and tell instantly without parsing the
payload.
B — nested CTI clauses:
preflight_node_hash() payload restructured from flat 3-key to
nested 5-clause:
classifier — quantifier classifier output (#000008)
answer_contract — guard / cap / reject / metacog state (per-run)
prompt_contract — reminder enabled / injected / template_id
evidence_contract — exposure budget, line discipline
policy_refs — governance_policy_hash, model_profile_hash,
answer_mode (reference, not raw policy)
Plus question_state (metacog) as its own clause and top-level
stage + node_version. Single DAG stage; nested clauses inside
for diff legibility (feedback §3).
C — node_version field:
PREFLIGHT_NODE_VERSION = "preflight-node-v1" pinned in the
payload so legacy runs without the node can be unambiguously
labeled `unavailable_legacy_run` by audit tools (feedback §9).
D — reference hashes only:
policy_refs uses governance_policy_hash + model_profile_hash
rather than bundling raw policy booleans. Avoids
double-committing already-hashed state (feedback §4).
E — reminder_template_id:
prompt_contract.reminder_template_id = "broad-quantifier-bounded-v1"
or "broad-quantifier-unbounded-v1" depending on scope_bound_hint,
populated only when reminder actually fires.
F — stage name kept as `preflight` (not `quantifier_preflight`):
Node carries both #000008 quantifier AND #000010 metacognition
payloads. node_version disambiguates schema for audit tools.
G — docs/cti-architecture.md update deferred to a small follow-up.
Bug fixes:
- free-variable shadowing on verifier_policy_hash /
model_profile_hash / question_hash — local re-imports inside
the reject branch shadowed module-top imports used elsewhere
in query() / runner(); now use the module-top names.
- reject path question_hash signature: takes `mode=` not
`dedup_mode=` — fixed in the reject DAG builder caller.
Hash compatibility:
Rows written between c36e85c and this commit have hash payloads
matching the OLD flat 3-key shape. The persisted run_dag_blob
captures the actual payload that was hashed, so those rows
still verify via verify_run_dag(). New rows use the nested
5-clause shape.
7 new tests in tests/test_dag.py:
- hash sensitivity to answer_contract / prompt_contract /
policy_refs flips (audit-replay payoff demonstrations)
- PREFLIGHT_NODE_VERSION pinning
- reject DAG: 3-stage shape, root changes with preflight hash,
round-trips through verify_run_dag
993 tests passing (6 net new); 36 skipped.
Live verification:
make query Q="winners of all major sports?" REJECT_BROAD=1 BURN=1
→ status=broad_quantifier_rejected, run_dag_root populated,
blob carries 3-stage shape.
make query Q="winners of all major sports?" BURN=1
→ 10-stage shape preserved (question → preflight → retrieval
→ ... → final_label).
Ticket #000009 status: closed · re-landed 2026-05-04 with §8
corrections.
Closes ticket #000009 zero-shot. Scope expanded to cover BOTH
ticket #000008 (broad-quantifier) AND ticket #000010
(meta-cognition) preflight contracts in a single combined node —
both share the same audit-replay gap and inserting two separate
nodes between question and retrieval was operationally awkward.
aborist/qa/dag.py:
+ preflight_node_hash() — combines QuestionState +
quantifier classifier output + behavioral policy_state into
one canonical SHA-256 hex.
+ build_run_dag() gains optional preflight_hash parameter.
When supplied, inserts {"stage": "preflight", "hash": ...}
at position 1 (between question and retrieval).
Backward-compat: None → original 7/9-stage shapes preserved
for legacy run_dag_root re-validation.
Quote-mode: 7 → 8 stages with preflight.
Pointer-mode CTI: 9 → 10 stages with preflight.
aborist/qa/query.py + runner.py:
Both build the preflight payload from question_state +
quantifier dict + 10-field policy_state (guard_enabled,
guard_apply_caps, guard_apply_caps_mode_gated,
claim_cap_resolved, claim_cap_actually_applied,
reminder_enabled, reminder_eligible, reject_broad_active,
metacognition_enabled, block_on_contradiction).
This means two cache rows that share the same question + same
model output + same verifier verdict but DIFFERENT preflight
policy state now produce different run_dag_root values. Audit
replay can pin the policy decision per row.
9 new tests in tests/test_dag.py:
- preflight_node_hash determinism
- hash bumps on question_state change
- hash bumps on policy_state change (the audit-replay payoff)
- all-None defensive shape
- 7→8 stage transition (quote mode)
- 9→10 stage transition (pointer mode), preflight at index 1
- run_dag_root bumps when preflight_hash bumps
- verify_run_dag round-trips through preflight stage
Live verification: latest providence_cache row carries
['question', 'preflight', 'retrieval', 'evidence_map', 'prompt',
'raw_answer', 'parsed_claim_lattice', 'verify', 'render',
'final_label'] — preflight stage living in the persisted DAG.
987 tests passing (9 new); 36 skipped.
Tickets:
#000009 status: closed · landed 2026-05-03 (zero-shot)
#000010 cross-ref updated: "DAG binding shipped via #000009"
What's NOT in this ticket (logged in §7.2):
- CLI flag for inspecting preflight node from cache_key
- Bench harness preflight_hash field for cross-row comparison
- SOFT_PREFLIGHT_HINT (model-assisted preflight sidecar)
Closes ticket #000010 (Meta-Cognition Preflight Guard). Mechanism
complete; defaults preserve the dry-run discipline pattern from
#000008.
Phase 2 — wire preflight into query() and runner.ask():
- preflight_question() runs after policy resolution + quantifier
classification, before retrieval.
- QuestionState surfaces on miss path, cache-hit path, AND
reject-broad early-return path of query() — schema column-
aligned across all four returns.
- runner.ask() carries the same fields for `aborist ask` parity.
Phase 3 — policy fields + governance hash + CLI flags:
- 6 new policy fields, all default-on except
metacognition_block_on_contradiction (default False — label-
only by default; opt-in via --block-on-contradiction).
- All 6 folded into _VERIFIER_POLICY_FIELDS so flipping any
invalidates prior cache records on lookup.
- 2 new CLI flags on `aborist query`:
--no-preflight Level 2 master kill
--block-on-contradiction strict mode (hard-block on
lexical contradictions)
Phase 4 — audit-line labels + bench fields + tests:
- _render_warrant_tail extended with 5 metacog tail tokens:
· false premise
· contradictory
· stale risk
· out of corpus
· frame ambiguous
- Bench rows in qa_sweep.py gain 7 new bounded-size projection
fields (logical_statuses, question_shape, preflight_result,
temporal_sensitivity, has_false_premise, has_contradiction,
corpus_requirement). Full QuestionState stays on result dict
for CLI render only.
- tests/test_metacognition.py grew from 42 → 68 tests
(16 new: 6 governance + 6 audit-line tail + 4 default-policy
pinning).
Live verified end-to-end:
$ make query-dry Q="Who is the current CEO of OpenAI?" BURN=1
UNGROUNDED · via claim_lattice · stale risk
$ make query-dry Q="When did Mr. Burns become Homer's biological
father?" BURN=1
UNGROUNDED · via claim_lattice · false premise
978 tests passing; 36 skipped.
What's NOT shipped (deferred):
- Run-DAG node binding for metacognition_preflight stage —
joins ticket #000009 Phase 5 (same audit-replay gap; both
nodes can land together).
- Reference-frame plumbing — frame_detection runs post-retrieval,
preflight here is pre-retrieval; deferred until two-pass
or post-classification update lands.
- SOFT_PREFLIGHT_HINT (model-assisted sidecar) — source doc §18
reserves this label; hard rule preserved (no LLM in preflight
hard path).
- Bench A/B measuring preflight on vs off — quick to run once
stack settles.
Ticket #000010 status: closed · landed 2026-05-03.
Implements the Meta-Cognition Preflight Guard (M0 / MCTL) per
fox's directive at ~/Downloads/meta-cognition_for_hermes(1).txt
(2026-05-03).
aborist/qa/metacognition.py:
- QuestionState dataclass (frozen, JSON-serializable via to_dict)
- preflight_question() pure function: classifies a question
deterministically into a QuestionState before generation
- 4 new detectors:
detect_temporal_sensitivity() — current/latest/today/CEO/etc.
detect_contradiction() — lexical pairs (unmarried+spouse,
always+never, alive+dead, etc.)
detect_false_premise() — presupposition patterns:
when did X stop/become Y,
why did X cause Y,
how did X become Y
detect_out_of_corpus() — my-uploaded-X / file-I-sent shapes
- Reuses #000008 quantifier classifier (no duplication)
- Composes 8 LogicalStatus values:
well_formed, under_specified, false_premise_suspected,
contradictory_question, out_of_corpus_risk, stale_risk,
reference_frame_ambiguous, broad_quantifier_unbounded
- Three preflight results: PREFLIGHT_OK / _PARTIAL / _BLOCKED
- Per-detector enable switches in policy:
metacognition_enabled (master kill)
metacognition_temporal_check
metacognition_contradiction_check
metacognition_false_premise_check
metacognition_out_of_corpus_check
metacognition_block_on_contradiction (default False — label
only by default; opt-in
to hard-block)
- preflight_policy_hash for governance binding (Phase 3)
- PREFLIGHT_VERSION = "metacognition-v0.1"
Hard rule (D1): no LLM in this hard path. Pure regex + lexical
matching. Model-assisted preflight, if added later, labels itself
SOFT_PREFLIGHT_HINT (not implemented in this phase).
42 new tests cover the seven test cases from source doc §14
(false-premise, contradictory, broad-quantifier, reference-frame,
time-sensitive, out-of-corpus, model-cutoff) plus per-detector
unit tests, gating (master kill, per-detector disable,
block-on-contradiction opt-in), determinism (question_hash
stable, policy_hash bumps on flip), and serialization.
Ticket #000010 opened with status `open · in progress
(zero-shot 2026-05-03)`. TICKETS.md index updated; Next ID bumped
to 000011.
Phases 2-4 still queued (wire into query/runner, policy fields +
governance, audit-line labels + bench fields).
962 tests passing (42 new); 36 skipped.
Closes#000008 with status `closed · landed in 4f2b5a6` per the
docs/TICKETS.md convention. The preflight guard mechanism + bench
cycle + default flip all shipped 2026-05-03; the design log stays
in place.
Opens #000009 — Quantifier preflight run-DAG node binding. Splits
the Phase 5 follow-up out of #000008 §11.11 into its own ticket.
Scope: bind the classifier output + policy decision into
`run_dag_root` so audit replay can distinguish guard-on vs guard-off,
cap-applied vs not, reminder-injected vs skipped. Currently those
appear on the result dict but are NOT in the run-DAG hash. Required
to close the audit-replay gap that blocks the §9.5 Merkle-AGI-DAG
framing from fully holding. Estimated 3-4h. Awaiting go/no-go.
Makefile shortcuts for the #000008 CLI flags (operator ergonomics):
BROAD=1 → --apply-quantifier-caps (flip cap apply-gate)
REJECT_BROAD=1 → --reject-broad (preflight rejection)
ALLOW_BROAD=1 → --allow-broad (emergent search)
Available on both `make query` and `make query-dry`. Default
behavior unchanged: ANSWER_MODE=claim_lattice (JSON), reminder ON
for lattice modes, cap operator-opt-in.
Smoke-tested:
$ make query-dry Q="winners of all major sports?" BROAD=1
→ cap applies on JSON; classifier reports ALL/unbounded
$ make query-dry Q="winners of all major sports?" REJECT_BROAD=1
→ preflight rejection, exit-1 (consistent with UNGROUNDED)
TICKETS.md index:
#000008 closed · landed in `4f2b5a6`
#000009 open · awaiting go/no-go (D3, D4)
Next ID bumped 000009 → 000010
n=5 verification of cap+reminder cell (135 runs):
Metric | n=3 | n=5
--------------------+-----------+------------
JSON SR | 0.30 | 0.33 ← matches cap-only
JSON UNGROUNDED rate| 1/27 (4%) | 2/45 (4%) ← matches reminder-only
pointer SR | 0/27 | 0/45 ← unchanged across all cells
The §12.8 0.30 was Hermes nondeterminism. n=5 confirms cap+reminder
delivers cap-only's STRICT-rate AND reminder-only's UNGROUNDED-rescue.
§10.8 strict gate met at n=5:
vs reminder-only on JSON SR: +11pp (clears floor)
vs cap-only on JSON UNGROUNDED: -18pp (clears floor)
vs cap-only on ptr mean ratio: +12pp (clears floor)
Defaults flipped — Option A landing (per-mode tailored):
quantifier_reminder_enabled False → True
(load-bearing on both lattice modes)
NEW field: quantifier_apply_caps_modes = ["claim_lattice"]
(allowlist for which modes apply caps
when apply_caps=True; JSON-only since
cap-on-pointer is wasted noise per
§12.10 0/45 STRICT data)
quantifier_guard_apply_caps False → False (UNCHANGED)
(operator opts in via
--apply-quantifier-caps; preserves
§10.11.3 dry-run discipline)
Cap-application gate now reads:
if apply_caps AND mode in apply_caps_modes AND cap is not None:
effective_max_claims = looked_up_cap
quantifier_apply_caps_modes folded into _VERIFIER_POLICY_FIELDS so
flipping the allowlist invalidates prior cache records.
5 new tests pin: reminder default ON for both runner.DEFAULT_POLICY
and query.DEFAULT_QUERY_POLICY; apply_caps_modes default
["claim_lattice"]; governance-hash invalidation on allowlist flip;
apply_caps default still False (dry-run preserved).
920 tests passing (5 new); 36 skipped.
Operator behavior:
$ aborist query "winners of all major sports?"
→ reminder ON, cap OFF (default after this commit)
$ aborist query --apply-quantifier-caps "..."
→ cap applies on claim_lattice (JSON) only
$ aborist query --apply-quantifier-caps \
--policy quantifier_apply_caps_modes='["claim_lattice","claim_lattice_pointer"]' "..."
→ Option D for one call
Phase 5 (run-DAG node binding for quantifier_preflight) and
cross-model Qwen/GPT-4 verification remain as follow-ups per §11.11.
#000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs):
Metric | Base | Rem | Cap | Cap+Rem
--------------------+------+------+------+--------
JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR
JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue
pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best
pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven
§10.8 strict gate "compound beats either alone by ≥5pp on every metric"
NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR
(clears floor) and beats cap-only by +17pp on pointer mean ratio
(clears floor), but is -3pp vs cap-only on JSON SR (regression,
within noise).
#000008 §12.9 — Final verdict + recommendation:
Mechanism asymmetry (clean signal):
Reminder rescues UNGROUNDED → HYBRID
Cap rescues HYBRID → STRICT
Recommendation: Option A — single-knob defaults, per-mode tailored:
claim_lattice (JSON): apply_caps=True + reminder=True
claim_lattice_pointer: apply_caps=False + reminder=True
(cap can't rescue pointer-tag discipline
upstream of cap; cap fires 20× without
verdict gain)
quote: guard mode-gated off (already default)
Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5
verification on cap+reminder before flipping defaults.
Cross-doc updates:
- CLAUDE.md: architecture diagram now lists the three new Phase 1-3
modules (quantifier.py, model_profiles.py, quantifier_reminder.py).
New "Broad-quantifier preflight guard" conventions entry covers
the 7 policy fields, six-level disable hierarchy, dry-run
defaults, CLI flags, and §12 bench summary.
- docs/qa-modes-bench.md: addendum at end pointing at #000008 §12
four-cell A/B + 4 bench artifact stamps. Original 2026-05-02
journal frozen; 2026-05-03 broad-subset findings flagged as
question-mix-dependent narrative on top of the global
per-mode recommendation.
§12.7 captures the 2026-05-03T12-47-23Z cap-only A/B (apply_caps=
True, reminder=False) on the 9-question broad subset.
Mode | Baseline | Reminder | Cap-only
----------+----------+----------+----------
JSON SR | 0.19 | 0.22 | 0.33 ← +14pp
JSON U | 7 | 1 | 6 ← reminder dominates
JSON S | 5 | 6 | 9 ← cap dominates
ptr ratio | 0.473 | 0.643 | 0.516
JSON ratio| 0.524 | 0.735 | 0.643
§10.8 gate MET on JSON mode (+14pp STRICT-rate).
Headline insight: cap and reminder help in DIFFERENT ways.
- Reminder rescues UNGROUNDED → HYBRID (restates citation rule).
- Cap rescues HYBRID → STRICT (forces fewer-but-better claims).
The two are complementary, not redundant. If §12.8 (cap+reminder)
confirms the compound effect, that's the §10.8 trigger to land
Option D as default. Predicted JSON SR ~0.40 if effects compound.
Pointer mode still 0/27 STRICT under cap-only — TOO_MANY_CLAIMS
fires 20× (vs 7× baseline) but pointer-tag failures upstream of
the cap still gate the verdict.
Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):
intensity pre-fix post-fix
SINGULAR 61 (84%) 65 (89%)
MANY 4 ( 5%) 0 ( 0%) ← all 4 were `how many X?`
ALL 1 ( 1%) 1 ( 1%)
COMPREHENSIVE 1 ( 1%) 1 ( 1%)
OPEN_REQUEST 5 ( 7%) 5 ( 7%)
SMALL_NUM 1 ( 1%) 1 ( 1%)
Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).
Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.
9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).
Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:
name all members of the beatles
list all planets in the solar system
Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.
915 tests passing (9 new); 36 skipped.
Single-source-of-truth section for what was actually built in the
2026-05-03 implementation pass. Complements §8 (commit table) and
§9.6 (per-phase notes) with a full inventory:
- §11.1 New modules: aborist/qa/{quantifier,model_profiles,quantifier_reminder}.py
- §11.2 Modified modules: query.py, runner.py, keys.py, cli.py, qa_sweep.py
- §11.3 New test files: 4 new + 1 extended; 120 new tests total
- §11.4 Seven new policy fields in _VERIFIER_POLICY_FIELDS
- §11.5 Four new CLI flags on `aborist query`
- §11.6 Four new violation kinds (3 soft + 1 hard) with audit tails
- §11.7 New result-dict fields on miss + cache-hit paths
- §11.8 Eight new bench-row fields
- §11.9 Implementation-time decisions not in §9/§10:
RUNG_PRIORITY ordering, bounded-domain anchors, reminder templates,
reject answer_text format, render branch, EXPLICIT_COUNT fallback,
quote-mode opt-out
- §11.10 Live verification artifacts (Winners-of-all-major-sports
rejected; Beatles-bounded NOT rejected)
- §11.11 What was NOT shipped: quantifier_preflight run-DAG node,
three-clause CTI contract DAG binding, A/B/D bench measurements,
cross-model verification — all queued as Phase 5 / bench follow-up
Implementation-time decisions section is the most operationally
useful — captures judgment calls made during coding that aren't in
the design docs but are now binding via tests.
Updates §8 Status, §9.6 Phase details, §9.8 Test surface, §9.11
commit sequence to reflect actual implementation state:
§8 Status — Phase 0 through Phase 4 all landed 2026-05-03 across
six commits (2ffed00 → 5a60e85). 906 tests passing (120 new).
Defaults preserve §10.11.3 dry-run discipline:
quantifier_guard_apply_caps=False, reminder=False, reject=False.
Six-level disable hierarchy fully wired. Live verification recorded
for both reject-broad-fires (Winners of all major sports?) and
reject-broad-skips (name all members of the Beatles → bounded
universal, NOT rejected).
§9.6 Phase details — each phase tagged LANDED <sha>; description
trimmed to what actually shipped vs the original proposal.
§9.8 Test surface — every check box flipped to [x] with the
matching test file path and test count. The two reject-broad
integration tests remain skipped (exercised by live bench).
§9.11 Implementation commit sequence — actual SHAs replace planned
commit numbers. Notes that DAG/audit binding for quantifier_preflight
node ("commit 7" in plan) is NOT shipped — tracked as Phase 5
follow-up. Optional now; required for §9.5 Merkle-AGI-DAG framing
to fully hold.
Next steps are bench measurement, not code: full bench under dry-run,
classifier review, then flip apply_caps and measure §10.8 deltas.
CLI flags on `aborist query`:
--no-quantifier-guard Level 2 disable: kills the guard for
one call. Telemetry → None.
--allow-broad Emergent-search: classifier on, caps
off. For exploratory enumeration.
--reject-broad Strict reject: ALL/COMPREHENSIVE/
OPEN_REQUEST + scope_bound_hint==
"unbounded" returns UNGROUNDED before
the LLM call (saves ~10-15s). Bounded
universals (Beatles, year-anchored)
are NOT rejected per §10.1.
--apply-quantifier-caps Flip Phase 2 dry-run gate per-call.
Path from dry-run to live cap.
Three new soft-demote violation kinds (§10.3) — no new audit_mode
token; tails on the existing audit-line:
BROAD_QUANTIFIER_RUNAWAY "broad runaway"
BROAD_QUANTIFIER_CAP_APPLIED "broad cap N" (cap value rendered)
BROAD_QUANTIFIER_SCOPE_UNBOUND "broad unbounded"
All three cap the ladder at ANCHOR-WARRANTED. Plus one HARD demote
(early-return UNGROUNDED):
BROAD_QUANTIFIER_REJECTED "broad rejected" (preflight rejection)
The reject-broad path early-returns from query() before the LLM
call when policy enables quantifier_reject_broad AND the question
is broad-unbounded. Result schema mirrors a normal UNGROUNDED row
(answer_text carries the rejection rationale + actionable narrowing
hints). _render_query_human gets a dedicated branch for the new
status so operators see the rejection without --json.
Live verification (post-commit):
$ aborist query --reject-broad "Winners of all major sports?"
UNGROUNDED · via BROAD_QUANTIFIER_REJECTED · ALL ("all") · cap was 8
0/0 0.0s (preflight)
BROAD-QUANTIFIER PREFLIGHT REJECTED · scope unbounded
Question matched ALL intensity ("all") with an under-specified
universe. Narrow ... or run with --allow-broad for exploratory
enumeration.
$ aborist query --reject-broad "name all members of the Beatles"
UNGROUNDED · via claim_lattice · title mismatch 4/4 20.9s
[Beatles enumerated, scope_bound_hint=bounded → not rejected]
`quantifier_reject_broad` folded into _VERIFIER_POLICY_FIELDS so
flipping reject default invalidates prior cache records.
16 new tests cover: soft-demote registration, hard-demote NOT in
soft-demote set, ladder rung mapping for each kind, tail rendering
(including cap value interpolation), tail combination with
existing kinds, end-to-end render through _render_query_human,
governance-hash binding. Two skipped placeholders mark the
integration paths exercised by live bench.
Lands aborist/qa/quantifier_reminder.py with broad_quantifier_
reminder(): one-line user-turn message restating the cap and the
[E\d+] citation rule for broad-intensity questions. Two templates:
bounded universe:
"This is a broad-quantifier query with a bounded universe.
Return at most N pointer-linked claim lines. Each claim must
cite an evidence id like [E5]; do not write claim lines
without bracket citations."
unbounded universe:
"This is a broad-quantifier query with an under-specified scope.
Return at most N pointer-linked claim lines. If you cannot
ground N claims with evidence IDs, return fewer grounded
claims. Do not enumerate from training prior. Each claim must
cite an evidence id like [E5]; do not write claim lines
without bracket citations."
The bounded template skips the "do not enumerate from training
prior" clause — the corpus has the answer set. Unknown scope falls
through to the stricter unbounded template (over-warn rather than
under-warn).
Wired into both query() and runner.ask() at the same insertion
point as the existing grounding_reminder — between
grounding_reminder and the evidence/question payload, where
Hermes-3-8B's most-recent-token attention catches it.
Default OFF (`quantifier_reminder_enabled: false`). Empirical
justification: ticket §3 Option B con notes Hermes already ignores
parts of the existing reminder under enumeration pressure. The
mechanism lands so an operator can A/B test cap-only vs cap+reminder
without code changes; default flips on after bench shows ≥5pp delta
on FORMAT_COLLAPSED or pointer-loss rate per §10.8.
`quantifier_reminder_enabled` folded into _VERIFIER_POLICY_FIELDS
so flipping the switch invalidates prior cache records.
19 new tests cover: gating (non-broad → None, missing cap → None,
None intensity → None), bounded-vs-unbounded template selection,
unknown scope falls back to unbounded, cap interpolation,
[E\d+] citation rule restatement, governance-hash invalidation.
Lands aborist/qa/model_profiles.py with two profiles:
- adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic
tight caps for broad intensities (ALL=8, COMPREHENSIVE=5,
OPEN_REQUEST=5) reflecting the 2026-05-02 runaway case.
- default
large-reasoner-class fallback (ALL=12, COMPREHENSIVE=15,
OPEN_REQUEST=12). Used when model_profile_id has no entry.
cap_for_intensity() resolves caps via three-source chain:
1. policy_overrides (per-call dict, highest priority)
2. per-model profile from PROFILES
3. "default" profile fallback
EXPLICIT_COUNT sentinel handles SMALL_NUM_EXPLICIT and
COMPARATIVE_BOUND — cap is the question's explicit count, not a
profile-set value. Defensive fallback to MANY cap if classifier
fired the rung without extracting a count.
Four new policy fields, all folded into governance_policy_hash via
_VERIFIER_POLICY_FIELDS:
- quantifier_guard_enabled master kill (default True)
- quantifier_guard_apply_caps dry-run gate (default False per
§10.11.3 — cap LOOKED UP and reported
on result, but NOT applied to the
verifier until operator flips True)
- quantifier_caps_by_intensity per-call override dict
- quantifier_guard_modes per-mode opt-in list (default
["claim_lattice_pointer",
"claim_lattice"]; quote opts out)
Six-level disable hierarchy (§10.11.2) implemented:
- Per-test: policy={"quantifier_guard_enabled": False}
- Per-call: --no-quantifier-guard (Phase 4)
- Per-phase: each policy switch is independent
- Per-mode: quantifier_guard_modes filter
- Per-model: model_profiles.py lookup
- Master: governance_policy_hash invalidation on flip
Wired through both query() and runner.ask() — both compute
effective_max_claims from the (classifier_intensity, model_profile,
policy_overrides) triple and pass it as max_claims_per_answer to
the verifier. Dry-run mode keeps effective_max_claims at the policy
default (12) until apply_caps flips True.
Result dict surfaces claim_cap_applied (the LOOKED-UP cap, even in
dry-run) plus all Phase-1 quantifier fields on miss-path AND
cache-hit path so bench rows stay column-aligned.
19 new tests pin: per-model selection, EXPLICIT_COUNT sentinel,
override precedence, governance-hash invalidation on every cap
field, profile shape (all ten rungs covered), default profile
presence.
Lands aborist/qa/quantifier.py with classify_question_quantifier(),
a pure function mapping a question string onto the ten-rung
intensity ladder (ticket #000008 §2):
ABSENT < SINGULAR < PROPORTIONAL < SMALL_NUM_EXPLICIT
< COMPARATIVE_BOUND < FEW < MANY < ABSENT < ALL
< OPEN_REQUEST < COMPREHENSIVE
Returns intensity, matched_token, explicit_count, is_broad,
operational_shape, scope_bound_hint, classifier_version. Pure: no
I/O, no model call, no retrieval call.
Highest-intensity-wins arbitration: COMPREHENSIVE strictly stronger
than OPEN_REQUEST (both > ALL). Catches "tell me everything about
all wars" → COMPREHENSIVE rather than dropping to one of the softer
shape detectors.
Scope_bound_hint heuristic (§10.1): bounded vs unbounded universals.
"All members of the Beatles" → bounded (corpus-known finite set).
"Winners of all major sports" → unbounded (scope undefined). Year-
anchored questions ("…in 2024") bound the universe to one event.
Heuristic only — corpus-arity check left for future refinement.
Wired into query() right after policy resolution. Both miss and
cache-hit paths surface quantifier_intensity, quantifier_matched
_token, scope_bound_hint, quantifier_explicit_count on the result
dict. claim_cap_applied is None until Phase 2 lands the cap-
application gate (default-off per §10.11.3 dry-run discipline).
61 new tests cover every rung, scope-bound detection (bounded /
unbounded / unknown), highest-wins arbitration, and regression
fixtures (factoid/wh-questions don't over-classify as broad).
Adds the §10.6 + §9.6 Phase-0.x bench-row fields. Pure additive —
no row-schema renames, no policy effects yet. Phase 1 classifier
will fill the quantifier_* slots; Phase 2 cap-table will fill
claim_cap_applied. Keeping the keys present here makes the JSONL
schema stable across the rollout so post-Phase-1 markdown can
re-render against pre-Phase-1 rows without column-misalignment.
New per-row fields:
- answer_pointer_count distinct E\d+ ids in raw_answer
- answer_chars_with_brackets chars inside [E\d+,...] regions
- raw_meaningful_line_count >20-char lines in raw_answer
(matches verifier FORMAT_COLLAPSED
denominator)
- quantifier_intensity slot for Phase 1 classifier
- quantifier_matched_token slot for Phase 1 classifier
- scope_bound_hint slot for Phase 1 classifier
- claim_cap_applied slot for Phase 2 cap-table
- model_profile_id configured model id verbatim
New helper _bracket_diagnostics() bundles the bracket/pointer/line
extraction in one place; module-level regexes (_BRACKET_RE,
_BRACKET_REGION_RE, _POINTER_ID_RE) avoid per-row recompilation.
5 new tests cover the helper: empty input, single pointer, multi-
pointer-in-one-bracket, separate brackets with shared id, format-
collapsed shape (5+ meaningful lines, 0 brackets — the
2026-05-02 winners-of-all-major-sports case).
Closes the bench-side gap surfaced in §5.2: JSONL was carrying summary
numbers only, blinding the harness to FORMAT_COLLAPSED rate and per-
violation-kind distributions. Without these, A/B/D bench measurements
on the broad-quantifier preflight guard would be guesses.
- query() result dict surfaces format_collapsed + raw_answer (lattice
modes only) so the bench can read them directly instead of re-deriving
from cache rows that --burn overwrites.
- Each bench row gains format_collapsed, violation_kinds (sorted unique
list — full payloads stay off the row to keep size bounded), and
answer_brackets (count of [E\d+] in raw_answer for lattice modes).
- _summarize aggregates per-mode FC count (only explicit True; None
means check didn't apply), per-kind tallies (each kind once per row),
and lattice-only bracket sum/n.
- Markdown renderer adds a `## format-collapse + violation kinds`
section with per-mode FC rate, mean raw brackets, and one column per
observed violation kind. Degrades gracefully when the sweep produces
no violations.
- 5 new bench-harness tests pin the aggregation rules.
Re-baseline (2026-05-02T20-58-57Z) sharpens §5.1 analysis dramatically:
NO_EVIDENCE_POINTER fires 3/3 in pointer mode and is the dominant gate,
not TITLE_MISMATCH (1/3) as §5.1 inferred from JSONL alone. FORMAT_
COLLAPSED actually fires 1/3 — not the rare corner the first baseline
called it. Implies Option B (prompt reminder) is the load-bearing fix
for the verdict; Option A (cap reduction) only moves secondary kinds.
§5.3 sub-investigation closed on first read — SCHEMA_INVALID:1 in
pointer mode is a legitimate kind emitted by verify_claim_lattice for
empty-claim-text (verify.py:1242) and bare-name-claim (verify.py:1270),
not a JSON-mode leak.
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.
Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.
verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict
Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
into verifier_policy_hash
- threaded through ask() and query() call sites
CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail
Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.
Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).
Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.
Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.
New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.
Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
gain `subject_tokens_absent_threshold` kwarg + per-claim check
block (mirrors TITLE_MISMATCH plumbing, sits right after it
in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
`verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value
Live verification (cache-split cleanly via policy-hash bump):
pre-fix cache_key 08dbd2c1… : STRICT (false positive)
post-fix cache_key 6a519636… : UNGROUNDED
Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT
Full suite: 776 passed, 34 skipped.
aborist now writes one JSONL session per `make query` invocation
and per `bench-emergent` cycle to:
~/.aborist/unfirehose/{project-slug}/{session-uuid}.jsonl
Unfirehose's native-harness auto-discovery picks up any
~/.{name}/unfirehose/ directory (see ingest.ts:discoverNativeHarnesses)
without registration — once a session lands, the unfirehose watcher
debounces, ingests, and exposes it in the dashboard alongside
Claude Code / Fetch / uncloseai sessions.
Schema: unfirehose/1.0 (per ~/git/unfirehose-nextjs-logger/docs/
unfirehose-schema.md). Each session file:
line 1 type=session (header — id, projectId, firstPrompt,
harness="aborist", harnessVersion)
line 2 type=message role=user
line 3 type=message role=assistant
content=[text]
model=hermes-3-llama-3.1-8b-fp8-dynamic
provider=hermes
durationMs=<wall>
aborist_meta={audit_mode, n_verified/n_quotes,
cache_key, cache_status, lookup_path,
violations, sources, timings_ms, answer_mode}
line 4 type=message role=system subtype=session_end durationMs
aborist-specific extras (verifier verdict, sources, timings) ride
under namespaced ``aborist_meta`` so the canonical fields stay clean
for off-the-shelf consumers; per the spec, unknown fields are
ignored downstream.
Bench-emergent cycles emit an additional system init message at
the start of each session noting the 3 random words, marking the
session as a generator-driven cycle vs a normal user query.
Failure-isolation: journal write is wrapped in a broad try/except
at every call site. A journaling bug must NEVER break the query
or bench loop.
Tests: 10 new in tests/test_journal.py (slug encoding, session
header, parent-id chain, session_end on close, aborist_meta
passthrough, usage block, idempotent close). Full suite: 663 passed.
Live verified: `make query Q="what is photosynthesis?"` produced
a 4-line JSONL with STRICT 3/3, all sources + timings populated,
ready for unfirehose ingestion.
Re-aggregated bench/emergent_log.jsonl at 200 cycles. Distribution
holds (79.5% UNGROUNDED, 17.5% HYBRID, 3.0% STRICT). Six STRICTs
total; five defensible (known-truth or non-relationship grounding).
The sixth — `steer/reply/correcter` — is the FIRST CONFIRMED
EVIDENCE-WARRANTED false-positive in 200 cycles. Decompressed all
10 chunks of the cited document (Glossary of language teaching
terms and ideas, 33.5K chars) and confirmed the words `correcter`,
`steer`, and `reply` appear NOWHERE in the source. Yet STRICT 1/1
verified, no violations.
Mechanism: per-claim citation-coverage Rule 5 (≥30% claim tokens
in cited span) passed on GENERIC linguistic vocabulary (language,
communication, grammar, exchange, relationships) which appears in
any linguistics article. The question-distinctive tokens
(correcter, steer, reply) rode along unverified. PREMISE_PARROTING
shape predicted in original ticket §D, exhibit A.
Headline correction: from "zero false-positives" to "one confirmed
false-positive (0.5%)."
Action item: opens design space for #000008 — PREMISE_PARROTING
detector. Lexical signal: question-distinctive tokens absent from
cited chunk while Rule 5 still passes on generic vocabulary →
SUBJECT_TOKENS_ABSENT violation, demote STRICT → HYBRID. Stays
binary, stays lexical, folds into verifier_policy_hash.
The ticket file was at docs/ticket-000006-... at top-level; the
tickets/ subdir convention landed before #6 was opened. Move into
docs/tickets/ to match every other ticket.
Amend with the 62-entry post-ticket delta. bench/emergent_log.jsonl
now holds 134 cycles total. Distribution stable vs original 72:
106 UNGROUNDED (79.1%), 22 HYBRID (16.4%), 6 STRICT (4.5%). Ten
new grounded cases all carry appropriate violation labels — zero
EVIDENCE-WARRANTED false-positives across all 134 cycles. Original
tuning candidates: Tomas-deflection resolved by a1dd330,
xxviii-STRICT kept as designed, metaphor sidecar calibration
deferred (still rare).
Reframe ticket as a rolling research log. Emergent stress-testing
is an ongoing thread, not a one-shot defect-fix; future
bench-emergent re-runs append new ## Amend sections here, and any
code-level tunings open their own tickets linking back.
TICKETS.md row reflects the rolling-research framing. Stale
"## Next ID" footer scrubbed from the ticket body (TICKETS.md is
canonical for the next-id counter).
Three changes that shape the same lever:
(1) The metaphor-cue wordlist now unions /usr/share/dict/words +
/usr/share/dict/american-english + /usr/share/dict/british-english.
The Debian split made the prior 'just symlink to american-english'
miss British spellings (colour, organisation, realise) which
silently became false negatives on British-speaker questions.
Union: 102,485 → 104,305 entries on this machine. ~1,820 added
British-specific entries.
(2) Supplemental dictionary support: operators can layer
domain-specific vocabulary into the morphological substrate.
Two paths:
- Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b
- Programmatic: register_metaphor_dictionary(path)
Each supplemental dict is one word per line. The cue suffix
tests (-ly stem, -ing stem, -est stem) then resolve domain
stems automatically — adding 'aerodynamic' to a custom dict
makes 'aerodynamically' classify as adverbial without code
changes.
Use case: 'a tree with its own vocabulary' — an aviation
forest, a medical corpus, a legal-domain shard each carries
jargon the standard wordlist doesn't cover. Register once,
suffix tests pick up domain stems forever.
(3) README gains a 'Sidecar diagnostics' section with a table of
the three sidecars (deflection, title-relevance, metaphor-
deflection) plus a 'Metaphor-deflection cue dictionary' subsection
explaining the derivation rule, the load order, and the per-
forest vocabulary configurability. Architecturally documents
why the rule is *derived* from the union (Phase-2 lesson) and
not hand-curated.
3 new tests in tests/test_inspect.py:
- register_metaphor_dictionary unions a custom path's words
- ABORIST_METAPHOR_DICTS env var supplements with two paths
- re-registering same path is idempotent
763/34 tests pass.
Pins the Ticket #000007 reproduction case as a permanent bench
fixture. Stresses the FTS5 hyphen-tokenization asymmetry where
query orthography (`bi-polar`) and indexed orthography (`Bipolar
disorder`) use different hyphen conventions for the same concept.
Pre-fix the medical-condition cluster never surfaced; post-fix
EVIDENCE-WARRANTED 2/2 grounded in `Bipolar disorder` + `Bipolar`
disambiguation.
Closes the FTS5 hyphen-tokenization asymmetry: `bi-polar is rare?`
retrieved only the Bi-Polar album/disambiguation cluster while the
medical-condition cluster (Bipolar disorder, Bipolar I/II disorder,
etc.) sat in the same shards untouched. `unicode61` splits hyphens
at index AND query time; `Bi-Polar Blues` indexes as [bi, polar, ...]
while `Bipolar disorder` indexes as [bipolar] — non-overlapping
token sets that never met.
Fix is query-layer only — no canonicalization_version bump, no
re-index, existing cache_keys stay valid:
- _hyphen_fold_variants(s): emit joined-no-hyphen variants for
every hyphenated run.
- _title_query_tokens(s): additively merges variants symmetrically
(queries AND titles when called on either).
- _filter_by_title_relevance: accept-path 5 — title stem-overlap
with hyphen-fold anchors passes the breadth gate. Rescues
`Bipolar disorder` (1-of-N qtoken match) without disrupting
non-hyphen queries (anchors empty → zero side effect).
- DEFAULT_QUERY_POLICY / DEFAULT_POLICY: hyphen_fold_v1: True
marker folds into governance_policy_hash; new records
cache-split cleanly from pre-fold records.
Live verification on /home/fox/.aborist/shards: same query now
retrieves `Bipolar disorder` (#5) and `Bipolar` disambiguation
(#7); model cites both, answer reads "Bi-polar disorder is not
rare; it affects approximately 2.8% of the U.S. population".
EVIDENCE-WARRANTED 2/2, properly grounded.
Tests: 4 new (3 unit, 1 integration with regression-pinned
Bipolar-disorder retrieval). Full suite 760 passed, 34 skipped.
Also: CLAUDE.md gains a close-when-complete hint for tickets — an
open ticket whose code already shipped is a stale map.
72 random-word triangulation cycles ran on 2026-05-02. Aggregate:
- 54 UNGROUNDED (75%)
- 14 HYBRID (19%)
- 4 STRICT (6%)
- 18 grounded total (25%)
Catalogs five failure/success shapes the curated bench-qa fixture
set can't surface, with one-line per category. Key finding: the
verifier-ladder + soft-demote stack is doing its job — every
HYBRID demoted via TITLE_MISMATCH / DEFLECTION_DETECTED /
CITATION_MISMATCH / TOO_MANY_EVIDENCE_IDS appropriately. Zero
EVIDENCE-WARRANTED false-positives in the sample.
Tuning candidates queued for the parallel-agent work:
1. Tomas-pulls-transceivers case escaped DEFLECTION_DETECTED
(subject anchor buried mid-paragraph; consider last 2-3
content tokens vs just last 1)
2. Roman-numeral / archaic-word triplets — accept gracefully
(system already does)
3. metaphor_deflection sidecar (added in parallel work) only
fired 1×/72; calibration pass when more samples land
Ticket is OPEN and awaiting follow-up tunings; re-run
`make bench-emergent EMERGENT_N=50` after parallel work commits
land and amend with the delta.
Status table + Next ID bumped to 000007.
Phase-2 lesson applied. The previous metaphor-deflection sidecar
shipped two hand-curated frozensets (_METAPHOR_LY_STOPWORDS,
_METAPHOR_ING_BLACKLIST) listing English -ly nouns and common -ing
verb forms. Same trap as concept_relations Phase 1: hand-curating
exception lists for English suffix patterns has an infinite tail
(every newly-encountered -ly noun earns one more git commit).
Replace with derivation. The system English wordlist
/usr/share/dict/words is already on the bench-emergent picker's
critical path; reuse it as the morphological substrate.
Cue rule (purely lexical, derived):
- -ly word is adverbial iff stem (or stem+e, or stem-i+y) is in
the wordlist:
gracefully → graceful ✓
truly → true ✓
happily → happy ✓
butterfly → butterf ✗ (noun, filtered automatically)
italy → ital ✗ (proper noun)
family → famil ✗ (noun)
- -ing word is participle iff stem (or stem+e, or de-doubled
consonant stem) is a verb in the wordlist:
fluttering → flutter ✓
running → run ✓ (consonant de-doubled)
making → make ✓
sterling → sterl ✗ (noun, filtered)
during → dur ✗ (preposition, filtered)
- -est word is superlative iff stem (or stem+e, or stem-i+y) is
an adjective in the wordlist:
rockiest → rocky ✓
longest → long ✓
safest → safe ✓
The closed-class prepositional cue set stays hand-listed (~10
words) — these are a finite grammatical class, not a vocabulary
tail. _METAPHOR_LY_STOPWORDS and _METAPHOR_ING_BLACKLIST deleted
entirely.
Graceful degradation: if /usr/share/dict/words doesn't exist
(BSD, some containers, Windows), the wordlist function returns
empty; suffix tests all return False; the sidecar quietly returns
no_signal. The closed prepositional set still works.
Verified on swallowtail canary: 5 cues found (amidst, fluttering,
gracefully, rockiest, upbraiding), threshold met, fires
metaphor_deflection.
Minor residual noise (holly → 'hol' in dict; interest → 'inter'
in dict) accepted as soft-signal cost. The signal is read by
human reviewers; over-flagging at the +1-cue level rarely
crosses the 3-cue threshold for a real question.
756/34 tests pass.
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
Emergent stress-test log surfaced two failure shapes the curated
bench can't reach. Reviewer (claude-opus-4-7 + fox) appended
teacher: blocks to four log entries; two of them named real
defects in the demote chain. This commit ships those fixes plus
the teacher-block annotations.
Cases reviewed (2026-05-02 emergent_log.jsonl):
✅ Menkar/sterilizers/trifle — working as designed
🟡 swallowtail/upbraided/rockiest — metaphorical deflection
🔴 cashback/widescreens/within — warrant gap on inferential claims
🔴 comeliness/fetish/investitures — topic shift earned ANCHOR-WARRANTED unfairly
Two code changes:
1) verify_claim_lattice + verify_claim_lattice_json: when EVERY
resolving claim has TITLE_MISMATCH, demote audit_mode →
UNGROUNDED. Previously HYBRID. Catches the cashback case
('widescreens offer cashback' cited to a generic Coupon
article — n_verified=1 overclaimed when the citation was
meaningless).
2) cli._ladder_rung_for_lattice: DEFLECTION_DETECTED moves from
soft-demote (cap at ANCHOR-WARRANTED) to hard-demote (cap at
POINTER-LINKED) alongside WARRANT_MISSING and TITLE_MISMATCH.
Catches the comeliness case (model fully shifted topic to
'Rock & Chips'; pre-fix the verified Rock & Chips claims
landed at ANCHOR-WARRANTED unfairly).
Test updates:
- tests/test_verify_json.py:test_verify_json_title_mismatch_demotes_to_ungrounded
(renamed; expected outcome flipped HYBRID → UNGROUNDED)
- 4 fixtures in tests/test_claim_lattice.py and 1 in
tests/test_verify_json.py: claim text augmented with 'in the
film' so the Jurassic-Park-titled fixture's title shares a
stem with the claim. These tests were testing other paths
(NO_EVIDENCE_POINTER, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID)
and the title-mismatch was incidental to the fixture choice.
Teacher blocks on the four log entries record the verdict +
failure_class + recommendation + notes, with reviewer + date.
The two ✅/🟡 entries have no recommended code change; the two 🔴
entries reference this commit as the fix.
751/34 tests pass.
12 per-module files + index = 13 files of 50-185 lines each =
1,064 lines of API reference scattered across a directory.
Each per-module file had real meat (API examples, ASCII tree,
conventions) but the cognitive cost of 'which file is this in?'
outweighed the navigation benefit.
Built via concatenation + patch-fix:
- cat index.md + per-module files in topological order
- rewrite ../diagrams/ -> diagrams/ (relative to docs/modules.md)
- rewrite ../../aborist/ -> ../aborist/
- rewrite ../TICKETS.md -> TICKETS.md, ../mesh.md -> mesh.md, etc.
- inter-module links (./<name>.md, <name>.md) -> #<name>-py anchors
- demote per-module H1 -> H2, H2 -> H3, etc., so the wrapper H1
is the only top-level heading
- de-dup the index.md's (now-H2) 'Aborist module reference'
header against the wrapper, replace with 'Diagrams index'
- inject explicit <a id="<name>-py"></a> anchors after each
module's H2 so the TOC links resolve regardless of GitHub's
auto-slug rules
- polish TOC link text: '[merkle.md](#merkle-py)' -> '[↓](#...)'
(the '.md' suffix made no sense once it's an in-doc anchor)
References updated:
- README.md (×2)
Net: 1,124 single-file lines vs 1,064 across 13 files. Slightly
larger because of the patch-fix scaffolding (anchors + section
markers), but one Cmd-F covers everything.
751/34 tests still pass.
Two cleanup operations bundled (separate scopes, single commit
since they share the doc-tree settle):
1. Move ticket-NNNNNN-<slug>.md files into docs/tickets/. The
directory makes browsing the design log easier; the index
stays at docs/TICKETS.md (top-level pointer). Convention text
in TICKETS.md updated to spell the new path.
2. Delete three docs whose load-bearing content has either been
absorbed into the codebase or distilled into closed tickets:
- docs/naming-deferral.md (147 lines) — explained why we
don't rename claim_lattice → CTI/PROMETHEUS-Σ. Decision
stays in place; the rationale is no longer worth a
dedicated doc. Inline citations removed from
cti-architecture.md (4 refs), warrant.py, ticket-000003
(closed-ticket internal ref).
- docs/reference-frame-failure-class.md (169 lines) — Orwell
case journal that motivated the phrase-pattern retrieval
route. The route shipped; the analysis is now duplicate
with the closed Ticket #000002. Inline citation removed
from CLAUDE.md retrieval pipeline section + frame.py.
- docs/test-coverage-audit-2026-05-01.md (46 lines) —
point-in-time audit checking 16/16 of fox's §11 list. Tests
themselves live in tests/; the audit was a one-shot
checkmark exercise.
References updated:
CLAUDE.md, aborist/qa/frame.py, aborist/qa/retrieval_plan.py,
aborist/qa/warrant.py, docs/cti-architecture.md, docs/TICKETS.md,
docs/tickets/ticket-000003 + ticket-000004 (internal links).
Net: -362 lines + tickets/ subdir. 751/34 tests still pass.
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
The journal had two stapled sections — the morning 11:31Z bench
(n=2 cell-grouped) followed by an 'Update' divider and the
afternoon 15:07Z bench (n=3 sample-shuffled). Two parts that
made readers walk past a horizontal rule to compare numbers, and
duplicated the directive-coverage / outputs / per-mode-rec
sections.
Rewrote as one coherent narrative:
- Frontmatter mentions both stamps in one table (when/what/wall)
- 'Hardening' section walks the chronology in two beats:
* pre-11:31Z (Rule 8, warrant gen, ladder, etc.)
* 11:31Z → 15:07Z (Sprint 1b, Sprint 2, DRY, keep-alive,
sample-shuffle, --resume, concurrency-sweep, surrogate v2)
- 'Aggregate' = authoritative 15:07Z table
- 'Δ across the day' = 3-column comparison
(2026-04-30 → 11:31Z → 15:07Z) with net deltas
- 'Per-bucket strict-rate' = 15:07Z bucket data
- 'Recommended context budget' = final
- 'Pointer-mode signal' kept (lazy-anchor analysis still valid)
- 'Wall-clock & throughput evolution' = both benches in one
table + concurrency sweep table
- 'Errors' = surrogate story across both benches
- 'Verdict' = final recommendation
- 'Outputs' = both jsonl files
212 lines → 175 lines (-36). One source of truth, one read.
bench-maxing.md gains a 'Bench harness — the speed playbook'
section capturing the 2026-05-02 speed wins as durable doctrine:
- Sample-level shuffled scheduling vs cell-grouped (+58%
throughput, true i.i.d. variance for n>=3)
- Persistent httpx client (TLS handshake amortization)
- Concurrency tuning (vLLM peak at c=3-4, brutal past c=4)
- Per-mode max_context_chars from bench feedback (the bench
is the substrate's voice; let it drive policy)
- --seed for reproducibility
- --resume for stop/start-able bench
- Smoke fixture for inner-loop iteration
- pytest -n auto (3.6× speedup on test suite)
README:
- Updated whitepaper section refs (§6/7/8/9/13) — old refs
pointed at §13.4.11/13.8/13.9 which no longer exist after
the whitepaper rewrite landed.
- 'Tests' section renamed 'Tests & bench' with make targets
for bench-qa, bench-qa-smoke, test-live. Resume + concurrency
semantics surfaced.
The c=4 sample-shuffled bench at 15:07Z lands the post-Sprint-1b
+ post-Sprint-2 + post-DRY + post-keep-alive + post-shuffle
state.
Headlines:
quote 0.50 → 0.54 (+4pp)
claim_lattice_pointer 0.23 → 0.20 (-3pp)
claim_lattice (JSON) 0.44 → 0.42 (-2pp)
Quote's +4pp is the cleanest lift of the sprint set: the per-mode
24KB cap (Sprint 1b) surfaces tighter retrievals that quote can
ground verbatim, and the bucket data confirms quote peaks at
8-16KB (0.58 strict-rate). JSON's peak migrated to its targeted
32-64KB bucket (0.48 strict-rate, vs 0.38 at 16-32KB) — Sprint
1b's intent confirmed at the per-bucket level even though the
aggregate slipped 2pp.
Pointer's slight drop is consistent with Sprint 2's smoke result
— the chunk-specificity Rule 9 didn't lift Hermes-3-8B's
lazy-anchoring at n=3. The structural fix will need a stronger
intervention than a prompt nudge.
Wall-clock & throughput:
11:31Z: cell-grouped, c=4, n=2, 426 tasks, 51 min, 8.4/min
15:07Z: sample-shuffled, c=4, n=3, 639 tasks, 48 min, 13.3/min
Sample-shuffled scheduling delivers +58% throughput at same
concurrency. n=3 (50% more work) ran in 6% LESS wall-clock.
Per-call mean latency dropped 35-42% across all modes — vLLM's
continuous batcher fills better when fed a diverse request
stream instead of cache_key-correlated cells.
Concurrency sweep: c=3 peak, c=4 within 4% (chosen), c=5 12%
slower, c=6 brutal (45% slower). vLLM saturates at c=3-4 on
this endpoint.
Errors: 6, all on 'tell me about the roman empire' question.
Root cause traced & fixed in 41d1d9b (lone UTF-16 surrogates
in Wikipedia chunk content broke httpx's outbound JSON encode
— different path from the 3b91223 SHA-256 hashers fix which
hardened the OUTPUT side). Next bench: 0 errors.
Six bench runs aborted with:
UnicodeEncodeError: 'utf-8' codec can't encode characters
in position N-M: surrogates not allowed
The error fires inside httpx's json-encode path: when the
request body's JSON contains lone surrogates (from Wikipedia
chunks ingested with invalid-UTF-8 source bytes), httpx's
.encode('utf-8') raises before the request even leaves the
client.
Earlier surrogate fixes (3b91223) hardened the OUTPUT side —
sha256 hashers now use errors='surrogatepass' so the run-DAG
roots survive surrogate-bearing model output. But the INPUT
side (corpus text injected into the prompt) was still
vulnerable: the LLM never sees the surrogate but the HTTP
client tries to send it.
Fix: scrub message content via WTF-8 → UTF-8-with-replace
roundtrip in OpenAICompatibleClient.chat_completion. Lone
surrogates become U+FFFD (REPLACEMENT CHARACTER); the prompt
serializes cleanly. Verified: 'tell me about the roman empire'
under claim_lattice mode now classifies HYBRID 5/7 instead of
erroring out (this question was 6/6 lattice runs failing on
the 2026-05-02 c=4 bench).
The scrub lives in the client because the hot path needs to
guarantee the outbound HTTP body is valid UTF-8, regardless of
what upstream code injected. Defense-in-depth: ingest-time
sanitization would be cleaner but the existing corpus already
has surrogates baked in, and re-ingest would invalidate every
document_root in 6 GB of shards.
Tests: 751/34 still pass clean in 11s with pytest -n auto.
Surfaces where the per-call cost lands. The audit-line `7.5s`
already shows total_ms but it's hidden inline with the verdict
& cache_status. The new `timings:` line (after `capacity:`) breaks
total wall-clock into its components so an operator can see at a
glance whether a slow query was Hermes-bound, search-bound, or
something else.
Format:
timings: cache 0.00s · search 0.30s · context 0.16s ·
llm 6.50s · persist 0.20s · **total 7.16s**
Skipped phases (cache=0 on cache-miss; persist=0 on cache-hit;
context=0 when retrieval pre-loaded) are filtered so the line
stays compact. **total** stays bold to match the audit-line
elapsed value already on screen.
JSON mode (`--json`) already exposes `timings` — this just lifts
the same data into the human render.
Tests pass (641 / -n auto / 11s).
Adds --resume <jsonl-path>: read the existing JSONL, build a set
of (question, mode, sample_idx) tasks already done, skip those
in the shuffled task list, and append fresh rows to the same
file. The markdown rollup uses the union of pre-existing + new
rows. Same --seed required for the order to align across the
resumed run.
Why: a 60-90 min full bench is annoying to re-run from zero when
something interrupts (network blip, kill, kernel panic). The
JSONL has been the durable artifact for ages; the bench just
didn't know how to read it.
Implementation:
- args.resume: Path | None
- if given: parse JSONL line-by-line, populate done_tasks set,
reuse the resume_path's stem as the run stamp, set jsonl_path
+ md_path to the resumed paths
- file_mode = 'a' if resuming else 'w'
- tasks filter: drop entries already in done_tasks
- rows initialized with existing_rows so the markdown summary
sees the union
- done counter starts at len(existing_rows) so the print
countdown reflects total progress
Empirical context: the c=6 bench killed at 142/639 left a
recoverable JSONL. Future kills can resume via:
.venv/bin/python bench/qa_sweep.py \
--resume bench/qa_results/<stamp>.jsonl \
--seed 0 --concurrency 4
(Same seed reproduces the shuffle so the remaining tasks come
out in the original order.)
Concurrency sweep on the smoke (15 tasks at each c):
c=3 102s (peak throughput)
c=4 106s (4% slower; chosen for full bench — more
forgiving on single-call hiccups)
c=5 119s (12% slower than c=3)
c=6 185s (45% slower; vLLM batching ceiling)
Lesson: more concurrent ≠ faster. vLLM continuous batching has
a sweet spot around c=3-4 for this endpoint.
Adds `workflow:rules: - when: never` at the top of .gitlab-ci.yml
so GitLab refuses to create any pipeline on push. The job
definitions stay in place — re-enable is a 4-line revert (delete
the workflow block) when bench infrastructure & CI runners are
decoupled.
Reason: aborist CI runners overlap with the same in-house pool
fox uses for `make bench-qa` against the live Hermes endpoint.
Auto-triggered pipelines on every push contend for the runner
& risk skewing bench latencies (already saw concurrency=4 push
mean per-call latency from 6-8s to 29s — adding background CI
jobs makes the signal noisier).
Local `make test` is unaffected — still 11s with -n auto.
Previously the bench scheduled at the (question, mode) cell level —
each cell ran its n samples sequentially before the next cell
started. With n=3 + concurrency=6, that meant 6 workers each
chained 3 sequential calls, vLLM's continuous batcher saw
correlated requests, and the n=3 variance signal was confounded
with 'what was vLLM doing in this same-cell batch.'
Two coupled defects, one fix:
(1) Statistical: samples-of-the-same-cell are NOT i.i.d. when run
back-to-back. Adjacent samples share vLLM batch composition,
KV-cache locality, and queue load. The 'variance' you see
across n=3 is partly that batch's particular weather, not true
model nondeterminism.
(2) Throughput: vLLM batches diverse requests well; correlated
requests fill the batch with similar work and starve.
Cell-grouped scheduling correlated by cache_key.
Refactor: build sample-level tasks ((q, mode, sample_idx)),
shuffle with deterministic seed (default 0), submit all to
ThreadPoolExecutor. Per-cell Lock dict (defaultdict(Lock))
serializes burn+insert against the shared cache_key — two samples
of the same cell that happen to land in adjacent worker slots
will queue on the cell's lock instead of racing. Lock contention
is rare under shuffle (samples of one cell are spread across
time) so throughput cost is near-zero.
New CLI flag --seed N for reproducible task ordering.
Changes:
- tasks list at sample granularity
- rng = random.Random(seed); rng.shuffle(tasks)
- cell_locks = defaultdict(Lock); held over _run_one + sample_idx assignment
- print line shows '#i/n' so order-of-completion is visible
Smoke at --n 1 --concurrency 6: clean, no errors, ~3 min for 15
tasks (essentially same wall-clock as cell-level scheduling at
n=1, as expected — the real win comes at n=3 where sample-level
concurrency unlocks parallelism within cells).
`.gitlab-ci.yml` runs the unit test suite (`make test`, which uses
`-n auto`) on every push. PYTEST_XDIST_AUTO_NUM_WORKERS=4 caps the
worker count so we don't oversubscribe on shared runners — pytest-
xdist's `-n auto` reads os.cpu_count() and on cgroup-limited
containers that over-reports the host's cores. Local `make test`
stays uncapped (uses full machine).
Wall-clock on the 641-test suite:
serial: 38s
-n auto local: 11s (full machine, 8+ cores typically)
-n auto + cap 4: 22s (CI runners — 1.7× speedup, still safe)
Job structure mirrors sibling repos (unsandbox.com,
unfirehose-nextjs-logger):
- tags: build → in-house runner pool
- cache keyed on pyproject.toml, paths .venv/ + .pip-cache/
- artifacts on failure → .pytest_cache/ for triage
Excluded from CI (need live state or network access):
- make test-live → live Hermes endpoint + populated shards
- make bench-qa{,-quick,-smoke} → live LLM bench
- chain-check-shards / analyze-shards → real shard data
Optional `test-crawler` job: gated on commit-message tag
`[ci-crawler]` or manual trigger; installs `[crawler]` extras and
runs the network-bound crawler suite. allow_failure: true so it
doesn't block merges.
