russell@unturf.com
38cfea1983
Sister rule to Rule 9 (SUBJECT_TOKENS_ABSENT) landed in the same session. Both demote STRICT → HYBRID but on orthogonal signals: Rule 9 catches premise-parroting; FORMAT_COLLAPSED catches protocol abandonment. Surfaced by fox's "winners of all major sports?" 2026-05-02 case: Hermes-3-8B melted under an under-specified broad-quantifier question, dumped 50+ free-form prose claims with zero [E\d+] pointer tags. Verifier honestly returned UNGROUNDED 0/2 (parser caught two line fragments), but operators couldn't distinguish "tried & failed to ground" from "abandoned the protocol." This soft-demote separates the two failure shapes at audit-line glance. verify_claim_lattice (pointer-mode only — JSON collapse already shows as SCHEMA_INVALID): - count meaningful_lines (>20 chars after strip) and [E\d+ regex matches in raw answer - ≥5 meaningful lines AND 0 bracket tags → FORMAT_COLLAPSED violation, soft-demote STRICT → HYBRID - format_collapsed: bool added to verdict dict Plumbing: - claim_lattice_format_collapse_check_enabled: True in DEFAULT_POLICY and DEFAULT_QUERY_POLICY - _VERIFIER_POLICY_FIELDS in keys.py adds the field so it folds into verifier_policy_hash - threaded through ask() and query() call sites CLI: - _SOFT_DEMOTE_VIOLATION_KINDS includes FORMAT_COLLAPSED so the audit-line ladder rendering treats it as a soft demote - _render_warrant_tail appends "· format collapsed" tail Bench fixture: new "under-specified 'all'" section in qa_questions.txt with `winners of all major sports?` and rationale about cross-model resilience signal. Tests: - test_format_collapsed_fires_on_bracketless_multi_line_prose - test_format_collapsed_does_not_fire_when_pointer_tags_present - CLI render coverage Full suite: 781 passed (up from 776). Open Ticket #000008 — Broad-quantifier preflight guard. Cleaner upstream fix: detect quantifier-intensity at query layer and apply a per-model claim ceiling BEFORE the 13-second LLM call. FORMAT_COLLAPSED stays as the downstream catch; #000008 proposes the upstream prevention. TICKETS.md index + Next ID 000008→000009. |
||
|---|---|---|
| .. | ||
| emergent_log.jsonl | ||
| qa_questions.txt | ||
| qa_questions_smoke.txt | ||
| qa_sweep.py | ||
| run.sh | ||