russell@unturf.com
a94d6a3244
Big batch — closes 4 of the 5 deferred items from the prior status report plus opens & implements a previously-deferred design ticket (#000011) zero-shot. #000025 — Metacog test fixture expansion: bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions covering edge cases per detector kind: temporal (4 cases), contradiction (4), false-premise (5), out-of-corpus (3), multi- trigger (2), well-formed controls (5). Documents two known detector ceilings: Q11 over-fires on past-tense factoid ("who was the first president"); Q16/Q17/Q19 (Edison/Australia/ NASA-fake) miss false premises that lack a presupposition pattern match. Fixture now serves as long-term regression suite. #000026 — --show-preflight full clause render: build_run_dag() and build_reject_run_dag() gain optional preflight_payload kwarg. When supplied, the canonical 5-clause CTI payload (classifier / answer_contract / prompt_contract / evidence_contract / policy_refs + question_state + node_version) persists alongside the leaf hash in run_dag_blob. aborist providence --show-preflight CACHE_KEY now renders the full payload + verifies the persisted hash matches the recomputed canonical hash (audit-replay tamper detection). Legacy rows fall through cleanly: payload_hash_check reports "unavailable: legacy row predates preflight_payload persistence". #000027 — Latency profile: Microbenched preflight: 0.46ms/question (negligible). Single fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the 33-35s in Addendum 3 was vLLM concurrency contention at c=4 (per qa-modes-bench.md saturation note), not substrate overhead. Added preflight_ms + soft_preflight_ms to timings dict for explicit confirmation in future cycles. #000028 — Auto-quality-check sweep revival: scripts/bench_emergent.py running with EMERGENT_N=100 in background (PID 125680). Will accumulate cycles into bench/emergent_log.jsonl for #000006 rolling log re-aggregation. Async — not blocking on completion. #000029 — #000011 SOFT_PREFLIGHT_HINT implementation: aborist/qa/soft_preflight.py — new module. SoftPreflightHint dataclass + soft_preflight_question() pure function. 9 canonical labels mapping to soft analogues of #000010 hard detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL). Constrained-generation prompt (max_tokens=128, temp=0.0) asks the model to pick ONE label + one-line rationale. Fail-closed across every parse path: - chat_client raises → SOFT_PARSE_FAIL - response unparseable → SOFT_PARSE_FAIL - label outside enum → SOFT_PARSE_FAIL Sidecar enforces SOFT_ prefix at the normalize step so a model that drops the prefix still gets caught. Wired into query() between preflight & retrieval. Default OFF (`soft_preflight_enabled: False`). NOT folded into _VERIFIER_POLICY_FIELDS — soft hints don't gate cache identity (#000011 §4). Audit-line tail renders as "· soft: <label>" (e.g. "· soft: time sensitive") so the signal is visually distinct from hard tails. --soft-preflight CLI flag opts in per-call. End-to-end live-verified on "When did Mr. Burns become Homer's biological father?" — produces: EVIDENCE-WARRANTED · via claim_lattice · false premise · soft: time sensitive 1/1 16.4s Hard `· false premise` (from #000010 deterministic detector) composed with soft `· soft: time sensitive` (from #000011 sidecar). The model classified a different shape than the hard detector — by design; soft hints are independent advisory signals, not redundant with the hard layer. 25 new tests pin: default-OFF behavior, parse-failure modes, label normalization (SOFT_ prefix enforced), all 8 actionable labels round-trip, fail-closed on client exceptions, dataclass JSON round-trip, rationale-length cap. Other: - #000010 §13.3 documents 2/5 metacog-trigger questions return STRICT despite hard-detector warning — direct empirical motivation for #000011 design. - tests/test_dag.py extends with 3 _extract_preflight_hash_* helper tests (cleaning #000009 §7.2 unfinished state). - bench/emergent_log.jsonl adds new cycles from background run. #000011 status: closed. Hard rule (D1) preserved across all 1021 tests (up from 996, +25 new). Soft preflight is purely advisory; the verifier proof path is unchanged.
112 lines
4 KiB
Text
112 lines
4 KiB
Text
# Meta-cognition trigger subset — Ticket #000010 detector validation.
|
|
#
|
|
# Six questions, one per metacog detector kind plus a mixed multi-trigger.
|
|
# Without these fixtures, the metacog detectors shipped in #000010 have
|
|
# zero empirical bench data — unit tests pin synthetic-string behavior
|
|
# but no rows in `bench/qa_results/*.jsonl` ever fire `temporal_sensitivity
|
|
# == "high"`, `contradiction_pairs`, `false_premise_hints`, or
|
|
# `out_of_corpus_risk`. This fixture fixes that.
|
|
|
|
# temporal sensitivity → stale_risk
|
|
who is the current CEO of OpenAI?
|
|
|
|
# contradiction (lexical pair: alive + dead)
|
|
which character is alive and dead at the same time?
|
|
|
|
# false-premise (presupposition pattern: "when did X stop Y?")
|
|
when did George Washington stop being president of France?
|
|
|
|
# out-of-corpus (private/uploaded reference)
|
|
what does my uploaded contract say about clause 9?
|
|
|
|
# multi-trigger: temporal + false-premise
|
|
when did the current CEO of Twitter stop being CEO of Tesla?
|
|
|
|
# control: well-formed factoid that should NOT trigger any detector
|
|
who painted the mona lisa?
|
|
|
|
# ── extended fixture (2026-05-04) — fitness-of-detector cases ──
|
|
#
|
|
# Below: edge / borderline / adversarial cases that probe the
|
|
# fitness of each detector. Some should fire, some should NOT, some
|
|
# should exhibit borderline behavior the operator can debate. The
|
|
# fixture lives long-term as a regression suite for the four
|
|
# metacog detectors. When the substrate evolves, this file is the
|
|
# pin against which behavior is measured.
|
|
|
|
# ── temporal sensitivity edge cases ──
|
|
|
|
# implicit-current via role + shape (no `current` keyword)
|
|
who is the President of the United States?
|
|
|
|
# rapid-turnover role with no temporal anchor at all (model-dependent)
|
|
who is the CEO of Twitter?
|
|
|
|
# implicit-now via "this year" — should fire HIGH
|
|
which team won this year's World Series?
|
|
|
|
# verbal future tense — model often hallucinates; not a detector
|
|
# target today but worth observing
|
|
who will be the next president after the 2028 election?
|
|
|
|
# negative control: timeless fact with role-shape — should NOT fire
|
|
who was the first president of the United States?
|
|
|
|
# ── contradiction edge cases ──
|
|
|
|
# multi-pair contradiction (alive+dead AND never+always)
|
|
which character was never alive but always dead?
|
|
|
|
# semantic contradiction without lexical pair — should NOT fire
|
|
# (out of detector scope; documents conservative-by-design choice)
|
|
which married bachelor lives in this house?
|
|
|
|
# triple-contradiction stress test
|
|
how can a celibate spouse be both unmarried and divorced?
|
|
|
|
# negative control: not a contradiction, just compound predicates
|
|
which characters appear in both Hamlet and Macbeth?
|
|
|
|
# ── false-premise edge cases ──
|
|
|
|
# "why did" relation pattern
|
|
why did Albert Einstein invent the lightbulb?
|
|
|
|
# "how did" relation pattern with subtle false premise
|
|
how did the Roman Empire conquer Australia?
|
|
|
|
# "when did" stop pattern with subtle false premise
|
|
when did Shakespeare stop writing in French?
|
|
|
|
# false premise that the lite detector probably MISSES (no
|
|
# presupposition trigger pattern) — annotates the detector ceiling
|
|
why does NASA fake the moon landings?
|
|
|
|
# negative control: legitimate "when did X stop Y" with true premise
|
|
when did the Roman Empire stop minting silver coins?
|
|
|
|
# ── out-of-corpus edge cases ──
|
|
|
|
# explicit "uploaded" reference
|
|
in the document I uploaded yesterday, what does section 5 say?
|
|
|
|
# private notes / inbox
|
|
according to my private notes, when did the meeting happen?
|
|
|
|
# borderline: claims about an existing public document the corpus
|
|
# may or may not have (NOT out-of-corpus per current detector)
|
|
what does the Wikipedia article on bipolar disorder say?
|
|
|
|
# ── multi-trigger combinatorics ──
|
|
|
|
# false-premise + out-of-corpus
|
|
when did my uploaded contract stop being valid?
|
|
|
|
# stale-risk + contradiction
|
|
which currently-alive historical figure is also dead?
|
|
|
|
# ── well-formed controls (should classify SINGULAR / well_formed) ──
|
|
|
|
what year did World War II end?
|
|
who is bilbo baggins's nephew?
|
|
list the planets of the solar system
|