russell@unturf.com 38cfea1983

qa(verify): FORMAT_COLLAPSED soft-demote + open #000008 (broad-quantifier preflight)

Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same
session. Both demote STRICT → HYBRID but on orthogonal signals:
Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches
protocol abandonment.

Surfaced by fox's "winners of all major sports?" 2026-05-02 case:
Hermes-3-8B melted under an under-specified broad-quantifier
question, dumped 50+ free-form prose claims with zero [E\d+]
pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser
caught two line fragments), but operators couldn't distinguish
"tried & failed to ground" from "abandoned the protocol." This
soft-demote separates the two failure shapes at audit-line glance.

verify_claim_lattice (pointer-mode only — JSON collapse already
shows as SCHEMA_INVALID):
- count meaningful_lines (>20 chars after strip) and [E\d+ regex
  matches in raw answer
- ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED
  violation, soft-demote STRICT → HYBRID
- format_collapsed: bool added to verdict dict

Plumbing:
- claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY
  and DEFAULT_QUERY_POLICY
- _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds
  into verifier_policy_hash
- threaded through ask() and query() call sites

CLI:
- _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the
  audit-line ladder rendering treats it as a soft demote
- _render_warrant_tail appends "· format collapsed" tail

Bench fixture: new "under-specified 'all'" section in
qa_questions.txt with `winners of all major sports?` and rationale
about cross-model resilience signal.

Tests:
- test_format_collapsed_fires_on_bracketless_multi_line_prose
- test_format_collapsed_does_not_fire_when_pointer_tags_present
- CLI render coverage
Full suite: 781 passed (up from 776).

Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner
upstream fix: detect quantifier-intensity at query layer and
apply a per-model claim ceiling BEFORE the 13-second LLM call.
FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes
the upstream prevention. TICKETS.md index + Next ID 000008→000009.

2026-05-02 16:43:41 -04:00

3.4 KiB

Raw Blame History

Aborist tickets — index

Aborist uses a flat ticket-NNNNNN-<slug>.md convention for actionable proposals: defects with a known root cause, design proposals awaiting go/no-go, or scoped enhancements queued for future implementation.

Convention

Filename: docs/tickets/ticket-NNNNNN-<slug>.md. The ID is a six-digit zero-padded integer; the slug is hyphen-lowercase and short enough to stay readable on a 100-column terminal.
Numbering: monotonic, never reused. Pick the next free integer greater than every existing ticket. The current next ID is on the Next ID line below; bump it when you open a ticket and commit the bump in the same commit as the new ticket file.

Header: every ticket starts with

# Ticket #NNNNNN — <short title>

**Status:** open · awaiting go/no-go      (or "in progress", "closed")
**Opened:** YYYY-MM-DD
**Scope:** <one-liner — what this ticket is for>
**Audience:** <who reads this>
**Hard constraint:** <invariants this ticket commits to NOT break>

Body: problem statement, design choices with trade-offs, recommendation, implementation sketch, scope boundaries, status. See ticket-000001-retrieval-keywords-audit-gap.md for a worked template.
Closure: when a ticket lands, flip Status to closed · landed in commit <sha> and leave the file in place. Do not delete tickets; they are the design log.

Distinction from other docs

docs/ also contains architecture references and bench journey notes that are NOT tickets — they describe state of the world rather than proposing change. Examples that stay un-numbered:

cti-architecture.md (architecture reference)
mesh.md, mesh-deploy.md (architecture + ops reference)
qa-modes-bench-2026-04-30.md (point-in-time bench journal)
verifier-semantic-gap-design.md, self-reference-design.md (older design docs that pre-date the ticket convention; leave as-is unless retroactively promoting one to a ticket adds value)
concept-relations-design.md (architecture reference for the per-shard concept_relations synonym/rivalry layer + the 1.6% storage-tax rationale; landed 2026-05-01)

If a doc proposes change AND awaits a decision AND has scoped implementation cost, it's a ticket. Otherwise it's reference.

Index

Newest first. Update on every open/close.

ID	Title	Status	Opened	Directive
#000008	Broad-quantifier preflight guard	open · awaiting go/no-go	2026-05-02	—
#000007	Query-layer hyphen folding	closed · 2026-05-02	2026-05-02	—
#000006	Bench-emergent findings (rolling research log)	open · rolling	2026-05-02	—
#000005	Label ladder migration (POINTER-LINKED → …)	closed · 2026-05-02	2026-05-01	D7
#000004	Directive coverage in bench summary	closed · `acd1f9c`	2026-05-01	D8
#000003	Anchor-class warrant generalization (Module H+)	closed · 2026-05-02	2026-05-01	D6
#000002	Reference-Frame Polarity Contract (Module L)	closed · 2026-05-02	2026-05-01	D3
#000001	Retrieval-keywords audit gap	closed · 2026-05-02	2026-05-01	D4

Next ID

000009

3.4 KiB Raw Blame History