qa(#000011 + 4 more): SOFT_PREFLIGHT_HINT impl + 5-task fan-out

Big batch — closes 4 of the 5 deferred items from the prior status
report plus opens & implements a previously-deferred design ticket
(#000011) zero-shot.

#000025 — Metacog test fixture expansion:
  bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions
  covering edge cases per detector kind: temporal (4 cases),
  contradiction (4), false-premise (5), out-of-corpus (3), multi-
  trigger (2), well-formed controls (5). Documents two known
  detector ceilings: Q11 over-fires on past-tense factoid
  ("who was the first president"); Q16/Q17/Q19 (Edison/Australia/
  NASA-fake) miss false premises that lack a presupposition
  pattern match. Fixture now serves as long-term regression suite.

#000026 — --show-preflight full clause render:
  build_run_dag() and build_reject_run_dag() gain optional
  preflight_payload kwarg. When supplied, the canonical 5-clause
  CTI payload (classifier / answer_contract / prompt_contract /
  evidence_contract / policy_refs + question_state + node_version)
  persists alongside the leaf hash in run_dag_blob.
  aborist providence --show-preflight CACHE_KEY now renders the
  full payload + verifies the persisted hash matches the
  recomputed canonical hash (audit-replay tamper detection).
  Legacy rows fall through cleanly: payload_hash_check reports
  "unavailable: legacy row predates preflight_payload persistence".

#000027 — Latency profile:
  Microbenched preflight: 0.46ms/question (negligible). Single
  fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the
  33-35s in Addendum 3 was vLLM concurrency contention at c=4
  (per qa-modes-bench.md saturation note), not substrate
  overhead. Added preflight_ms + soft_preflight_ms to timings
  dict for explicit confirmation in future cycles.

#000028 — Auto-quality-check sweep revival:
  scripts/bench_emergent.py running with EMERGENT_N=100 in
  background (PID 125680). Will accumulate cycles into
  bench/emergent_log.jsonl for #000006 rolling log re-aggregation.
  Async — not blocking on completion.

#000029#000011 SOFT_PREFLIGHT_HINT implementation:
  aborist/qa/soft_preflight.py — new module. SoftPreflightHint
  dataclass + soft_preflight_question() pure function. 9
  canonical labels mapping to soft analogues of #000010 hard
  detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL).
  Constrained-generation prompt (max_tokens=128, temp=0.0) asks
  the model to pick ONE label + one-line rationale.
  Fail-closed across every parse path:
    - chat_client raises → SOFT_PARSE_FAIL
    - response unparseable → SOFT_PARSE_FAIL
    - label outside enum → SOFT_PARSE_FAIL
  Sidecar enforces SOFT_ prefix at the normalize step so a
  model that drops the prefix still gets caught.

  Wired into query() between preflight & retrieval. Default
  OFF (`soft_preflight_enabled: False`). NOT folded into
  _VERIFIER_POLICY_FIELDS — soft hints don't gate cache
  identity (#000011 §4). Audit-line tail renders as
  "· soft: <label>" (e.g. "· soft: time sensitive") so the
  signal is visually distinct from hard tails.

  --soft-preflight CLI flag opts in per-call. End-to-end
  live-verified on "When did Mr. Burns become Homer's biological
  father?" — produces:

    EVIDENCE-WARRANTED · via claim_lattice
        · false premise · soft: time sensitive
        1/1  16.4s

  Hard `· false premise` (from #000010 deterministic detector)
  composed with soft `· soft: time sensitive` (from #000011
  sidecar). The model classified a different shape than the hard
  detector — by design; soft hints are independent advisory
  signals, not redundant with the hard layer.

  25 new tests pin: default-OFF behavior, parse-failure modes,
  label normalization (SOFT_ prefix enforced), all 8 actionable
  labels round-trip, fail-closed on client exceptions, dataclass
  JSON round-trip, rationale-length cap.

Other:
  - #000010 §13.3 documents 2/5 metacog-trigger questions return
    STRICT despite hard-detector warning — direct empirical
    motivation for #000011 design.
  - tests/test_dag.py extends with 3 _extract_preflight_hash_*
    helper tests (cleaning #000009 §7.2 unfinished state).
  - bench/emergent_log.jsonl adds new cycles from background run.

#000011 status: closed. Hard rule (D1) preserved across all
1021 tests (up from 996, +25 new). Soft preflight is purely
advisory; the verifier proof path is unchanged.
This commit is contained in:
russell@unturf.com 2026-05-03 23:00:56 -04:00
parent 621f0b2cda
commit a94d6a3244
No known key found for this signature in database
10 changed files with 833 additions and 20 deletions

View file

@ -424,6 +424,11 @@ def _cmd_query(args: argparse.Namespace) -> int:
# Strict mode: hard-block on lexical contradictions instead
# of label-only.
call_policy["metacognition_block_on_contradiction"] = True
if getattr(args, "soft_preflight", False):
# Ticket #000011 — opt-in to model-assisted soft preflight
# sidecar. Adds one short LLM round-trip; NEVER gates
# admissibility (D1 preserved).
call_policy["soft_preflight_enabled"] = True
result = query(
question=args.question,
@ -686,6 +691,19 @@ def _render_warrant_tail(result: dict) -> str:
parts.append("out of corpus")
if "reference_frame_ambiguous" in statuses:
parts.append("frame ambiguous")
# Ticket #000011 — soft preflight sidecar hint. Renders distinctly
# from the hard tails above so an operator can tell at a glance
# that the signal is advisory. Skips SOFT_DISABLED / SOFT_PARSE_FAIL
# / SOFT_WELL_FORMED (no actionable signal).
soft = result.get("soft_preflight_hint") or {}
soft_label = soft.get("classifier_label") or ""
if soft_label and soft_label not in (
"SOFT_DISABLED", "SOFT_PARSE_FAIL", "SOFT_WELL_FORMED",
):
# Strip SOFT_ prefix + lowercase for tail readability
# (e.g. SOFT_FALSE_PREMISE_SUSPECTED → "false premise suspected").
readable = soft_label.removeprefix("SOFT_").lower().replace("_", " ")
parts.append(f"soft: {readable}")
if not parts:
return ""
return " · " + " · ".join(parts)
@ -1860,15 +1878,12 @@ def _cmd_providence_show_preflight(
file=sys.stderr,
)
return 1
# The stage hash itself is informative; the payload that produced
# the hash is NOT in the persisted blob (only the leaf hash is —
# by Merkle convention, the inputs are derivable from the result
# dict at write time but not from the persisted blob alone). What
# we DO have on the result dict for the original write was the
# five-clause payload; for inspection we re-render the run-DAG
# node + cross-reference to the preflight_hash 12-char prefix
# callers may have seen in bench rows or `_render_query_human`.
out = {
# Pull the full preflight payload (Ticket #000009 §7.2 — payload
# now persisted alongside nodes via build_run_dag's
# preflight_payload kwarg). Fall back to hash-only render for
# legacy rows whose blob predates the payload-storage commit.
payload = parsed.get("preflight_payload")
out: dict = {
"cache_key": row["cache_key"][:12],
"question": row["question_text"],
"preflight_stage_hash": preflight_node.get("hash"),
@ -1876,6 +1891,23 @@ def _cmd_providence_show_preflight(
"run_dag_root": parsed.get("root"),
"run_dag_stages": [n.get("stage") for n in nodes],
}
if payload is not None:
# Verify the persisted payload hashes to the persisted leaf.
# Mismatch would indicate post-write tampering or a serialization
# drift; surface it explicitly so an auditor can detect.
from aborist.qa.dag import _canonical_json, _sha256_hex
recomputed = _sha256_hex(_canonical_json(payload))
out["preflight_payload"] = payload
out["payload_hash_check"] = (
"ok" if recomputed == preflight_node.get("hash")
else f"MISMATCH (recomputed {recomputed[:12]} != stored {preflight_node.get('hash', '')[:12]})"
)
else:
out["preflight_payload"] = None
out["payload_hash_check"] = (
"unavailable: legacy row predates preflight_payload "
"persistence (Ticket #000009 §7.2)"
)
print(json.dumps(out, indent=2, ensure_ascii=False))
return 0
@ -3650,6 +3682,19 @@ def build_parser() -> argparse.ArgumentParser:
"spouse is X married to' return PREFLIGHT_BLOCKED."
),
)
query_cmd.add_argument(
"--soft-preflight",
dest="soft_preflight", action="store_true",
help=(
"Ticket #000011 — opt-in to the model-assisted soft "
"preflight sidecar. Adds one short LLM round-trip "
"(~200ms median) before the main answer call; the model "
"classifies the question shape and returns a SOFT_* "
"advisory hint that surfaces as `· soft: <label>` on "
"the audit-line tail. NEVER enters the verifier proof "
"path; cannot create PREFLIGHT_OK or PREFLIGHT_BLOCKED."
),
)
query_cmd.set_defaults(func=_cmd_query)
inspect_cmd = sub.add_parser(

View file

@ -220,6 +220,7 @@ def build_run_dag(
rendered_text: str | None = None,
retrieval_plan_hash: str | None = None,
preflight_hash: str | None = None,
preflight_payload: dict | None = None,
) -> dict:
"""Return ``{"root": <hex>, "nodes": [<stage>, <hash>], ...}``.
@ -361,7 +362,21 @@ def build_run_dag(
)
leaves = [bytes.fromhex(n["hash"]) for n in nodes]
root_hex = MerkleTree.build(leaves).root.hex()
return {"root": root_hex, "nodes": nodes}
out = {"root": root_hex, "nodes": nodes}
# Ticket #000009 §7.2 — recoverable preflight payload. Storing
# the canonical dict alongside the leaf hash means
# `aborist providence --show-preflight` can render the full
# 5-clause CTI contract (classifier / answer_contract /
# prompt_contract / evidence_contract / policy_refs +
# question_state) from `run_dag_blob` without needing a
# separate column or re-running the classifier. Audit replay
# CAN re-verify the hash matches:
# _sha256_hex(_canonical_json(preflight_payload)) == preflight_hash
# (caller-side check; verify_run_dag does not enforce because
# the hash is in `nodes` and the payload is sidecar data.)
if preflight_payload is not None:
out["preflight_payload"] = preflight_payload
return out
def build_reject_run_dag(
@ -373,6 +388,7 @@ def build_reject_run_dag(
audit_mode: str = "UNGROUNDED",
verifier_method: str = "claim_lattice_pointer",
violations: list[dict] | None = None,
preflight_payload: dict | None = None,
) -> dict:
"""3-stage reject-broad run-DAG: ``question → preflight →
final_label``.
@ -415,7 +431,10 @@ def build_reject_run_dag(
]
leaves = [bytes.fromhex(n["hash"]) for n in nodes]
root_hex = MerkleTree.build(leaves).root.hex()
return {"root": root_hex, "nodes": nodes}
out = {"root": root_hex, "nodes": nodes}
if preflight_payload is not None:
out["preflight_payload"] = preflight_payload
return out
def verify_run_dag(blob: str | dict) -> bool:

View file

@ -480,6 +480,9 @@ DEFAULT_QUERY_POLICY = {
"metacognition_false_premise_check": True,
"metacognition_out_of_corpus_check": True,
"metacognition_block_on_contradiction": False,
# Ticket #000011 — soft preflight sidecar. See runner.DEFAULT_POLICY
# for full rationale. Default OFF.
"soft_preflight_enabled": False,
# Claim-count ceiling — see runner.DEFAULT_POLICY for rationale.
# Bench finding (york-england "tell me all there is to know")
# caught the runaway shape; cap of 12 admits entity-list
@ -1728,12 +1731,28 @@ def query(
# lives on the result dict separately, not on QuestionState
# in this pass).
from aborist.qa.metacognition import preflight_question
_t_preflight = time.monotonic()
question_state = preflight_question(
question,
model_profile_id=model_id,
reference_frames=(),
policy=policy,
)
preflight_ms = _ms_since(_t_preflight)
# Ticket #000011 — optional soft preflight sidecar. Default OFF;
# one short LLM round-trip when policy["soft_preflight_enabled"]
# is True. Returns a stub hint (SOFT_DISABLED) when off so the
# result-dict / run-DAG schema stays consistent. NEVER enters
# the verifier proof path; advisory only.
from aborist.qa.soft_preflight import soft_preflight_question
_t_soft_preflight = time.monotonic()
soft_hint = soft_preflight_question(
question,
chat_client=chat_client,
model_id=model_id,
policy=policy,
)
soft_preflight_ms = _ms_since(_t_soft_preflight)
# Ticket #000008 Phase 4 — strict reject for broad-unbounded.
# When opt-in via policy / --reject-broad CLI flag, return
# UNGROUNDED before the LLM call for ALL/COMPREHENSIVE/
@ -1791,7 +1810,15 @@ def query(
"year, league, country, or category) or run with "
"--allow-broad for exploratory enumeration."
)
_reject_preflight_hash = _pre_hash(
# Same payload-then-hash pattern as the miss path so
# `--show-preflight` can render the full clause set on
# reject rows too.
from aborist.qa.dag import (
_canonical_json as _reject_canon,
_sha256_hex as _reject_sha,
build_preflight_node_payload as _reject_build_payload,
)
_reject_preflight_payload = _reject_build_payload(
question_state=question_state.to_dict(),
quantifier=quantifier,
answer_contract={
@ -1832,9 +1859,13 @@ def query(
"answer_mode": answer_mode,
},
)
_reject_preflight_hash = _reject_sha(
_reject_canon(_reject_preflight_payload)
)
_reject_run_dag = build_reject_run_dag(
question_hash=_reject_qhash,
preflight_hash=_reject_preflight_hash,
preflight_payload=_reject_preflight_payload,
rejection_reason=_reject_rationale,
answer_text=_reject_answer_text,
audit_mode="UNGROUNDED",
@ -2842,7 +2873,15 @@ def query(
if quantifier.get("scope_bound_hint") == "bounded"
else "broad-quantifier-unbounded-v1"
)
preflight_hash = preflight_node_hash(
# Build the canonical payload once; hash it AND persist it
# alongside the DAG nodes so audit replay can render the
# full 5-clause CTI contract via `aborist providence
# --show-preflight`. Hash is deterministic from payload, so
# an auditor can re-verify:
# _sha256_hex(_canonical_json(preflight_payload))
# == nodes[preflight_idx]["hash"]
from aborist.qa.dag import build_preflight_node_payload
_preflight_payload = build_preflight_node_payload(
question_state=question_state.to_dict(),
quantifier=quantifier,
answer_contract={
@ -2884,6 +2923,8 @@ def query(
"answer_mode": answer_mode,
},
)
from aborist.qa.dag import _sha256_hex, _canonical_json
preflight_hash = _sha256_hex(_canonical_json(_preflight_payload))
run_dag = build_run_dag(
question_hash=qhash,
sources=proof_obj["sources"],
@ -2904,6 +2945,7 @@ def query(
rendered_text=answer_text if is_lattice_mode else None,
retrieval_plan_hash=plan_hash,
preflight_hash=preflight_hash,
preflight_payload=_preflight_payload,
)
run_dag_blob = json.dumps(run_dag, separators=(",", ":"))
@ -3077,6 +3119,10 @@ def query(
# operator tools. Same hash that's bound into the
# `preflight` stage of run_dag_root.
"preflight_hash": preflight_hash,
# Ticket #000011 — soft preflight sidecar hint. Advisory
# only; NEVER enters the verifier proof path. Renderer
# surfaces as `· soft: <label>` on the audit-line tail.
"soft_preflight_hint": soft_hint.to_dict(),
# Sidecar smell signals (claim_lattice mode only) — surfaced
# for the renderer; never persisted in providence_cache and
# never threaded into run_dag_root.
@ -3104,6 +3150,8 @@ def query(
else None
),
"timings": {
"preflight_ms": preflight_ms,
"soft_preflight_ms": soft_preflight_ms,
"search_ms": search_ms,
"context_ms": context_ms,
"cache_lookup_ms": cache_lookup_ms,

View file

@ -268,6 +268,12 @@ DEFAULT_POLICY = {
"metacognition_false_premise_check": True,
"metacognition_out_of_corpus_check": True,
"metacognition_block_on_contradiction": False,
# Ticket #000011 — soft preflight sidecar. Default OFF —
# adds one short LLM round-trip (~200ms median) so cost is
# operator-opt-in only. NEVER enters the verifier proof path
# (D1); produces only SOFT_* labels that surface as advisory
# hints alongside the deterministic detector output.
"soft_preflight_enabled": False,
# Claim-count ceiling. Bench finding (2026-04-30 york-england):
# "tell me all there is to know about X" prompted Hermes to spam
# 26-59 encyclopedic claims sourced from training, only 2-4 of
@ -880,7 +886,15 @@ def ask(
if quantifier.get("scope_bound_hint") == "bounded"
else "broad-quantifier-unbounded-v1"
)
preflight_hash = preflight_node_hash(
# Build payload + hash separately so we can persist both into
# run_dag_blob (Ticket #000009 §7.2 — `aborist providence
# --show-preflight` renders the full clause set).
from aborist.qa.dag import (
_canonical_json as _runner_canon,
_sha256_hex as _runner_sha,
build_preflight_node_payload as _runner_build_payload,
)
_runner_preflight_payload = _runner_build_payload(
question_state=question_state.to_dict(),
quantifier=quantifier,
answer_contract={
@ -920,6 +934,7 @@ def ask(
"answer_mode": answer_mode,
},
)
preflight_hash = _runner_sha(_runner_canon(_runner_preflight_payload))
run_dag = build_run_dag(
question_hash=qhash,
sources=[{
@ -944,6 +959,7 @@ def ask(
parsed_lattice=parsed_lattice,
rendered_text=answer_text if is_lattice_mode else None,
preflight_hash=preflight_hash,
preflight_payload=_runner_preflight_payload,
)
run_dag_blob = json.dumps(run_dag, separators=(",", ":"))

View file

@ -0,0 +1,255 @@
"""Soft preflight sidecar — Ticket #000011.
Model-assisted preflight that augments the deterministic detectors
from #000010 with an LLM-driven shape classifier. Strict guardrail
per ticket §1: produces ONLY soft hints labeled ``SOFT_*``; cannot
create ``PREFLIGHT_OK`` or ``PREFLIGHT_BLOCKED`` without
deterministic support. Hard rule (D1 in
``docs/seven-point-program.md``) preserved.
Default-OFF (``soft_preflight_enabled: False``). Operator opts in
per-call via ``--soft-preflight`` CLI flag or per-policy via
``policy["soft_preflight_enabled"] = True``. The sidecar adds one
short LLM round-trip (~200ms median on the local Hermes endpoint;
constrained generation with max_tokens=128, temperature=0.0).
Output contract: a single ``SoftPreflightHint`` dataclass that
surfaces on the result dict, run-DAG ``preflight_payload`` clause,
& bench rows. Audit-line tail renders as ``· soft: <label>`` so
operators can distinguish soft from hard signals at a glance.
Hard rule (D1): NEVER bypasses or overrides the deterministic
verifier. The hint is advisory only; the verifier doesn't read
sidecar fields.
"""
from __future__ import annotations
import re
import time
from dataclasses import asdict, dataclass
SOFT_PREFLIGHT_VERSION = "soft-preflight-v0.1"
# Constrained enum the model must pick from. Each maps to a soft
# analogue of one #000010 deterministic detector kind. The SOFT_
# prefix is mandatory (see _normalize_label below) so audit replay
# can never confuse a soft hint with a hard verdict.
_SOFT_LABELS = (
"SOFT_WELL_FORMED",
"SOFT_FALSE_PREMISE_SUSPECTED",
"SOFT_CONTRADICTION_SUSPECTED",
"SOFT_TIME_SENSITIVE",
"SOFT_SCOPE_AMBIGUOUS",
"SOFT_OUT_OF_CORPUS_LIKELY",
"SOFT_BROAD_QUANTIFIER",
"SOFT_MULTI_HOP_REASONING",
"SOFT_SUBJECTIVE",
)
@dataclass(frozen=True)
class SoftPreflightHint:
"""One sidecar hint about a question's logical/epistemic shape.
NEVER enters the verifier proof path. NEVER folds into
``governance_policy_hash``. ALWAYS labeled ``SOFT_*`` so the
distinction from deterministic detector output is explicit at
every layer (audit-line tails, bench rows, run-DAG payload).
"""
raw_question: str
sidecar_version: str # SOFT_PREFLIGHT_VERSION
classifier_label: str # one of _SOFT_LABELS or SOFT_DISABLED
confidence: float # 0.0-1.0; advisory only, never gates
rationale: str # one-line model explanation
timestamp_ns: int # monotonic_ns at hint creation
model_profile_id: str # which model produced the hint
elapsed_ms: float # LLM call latency for cost tracking
def to_dict(self) -> dict:
return asdict(self)
def _stub_hint(
label: str,
rationale: str,
*,
raw_question: str = "",
model_profile_id: str = "",
elapsed_ms: float = 0.0,
) -> SoftPreflightHint:
"""Return a default hint when the sidecar is disabled or fails.
`label` is `SOFT_DISABLED` (sidecar opt-out) or `SOFT_PARSE_FAIL`
(LLM returned junk). Either way, downstream treats this as an
advisory non-signal the deterministic detectors still own the
ground truth.
"""
return SoftPreflightHint(
raw_question=raw_question,
sidecar_version=SOFT_PREFLIGHT_VERSION,
classifier_label=label,
confidence=0.0,
rationale=rationale,
timestamp_ns=time.monotonic_ns(),
model_profile_id=model_profile_id,
elapsed_ms=elapsed_ms,
)
# System prompt for the soft preflight LLM call. Constrained
# generation (max_tokens=128, temp=0.0) keeps the output short &
# deterministic-ish. The model picks ONE label from the enum & gives
# a one-line rationale; no chain-of-thought, no multi-line output.
_SOFT_PREFLIGHT_SYSTEM_PROMPT = """You are a question-shape classifier. Your job is to read a single question and classify its shape using ONE of these tokens, plus a one-line rationale.
LABELS:
- SOFT_WELL_FORMED question is logically well-formed; no obvious shape concerns
- SOFT_FALSE_PREMISE_SUSPECTED question presupposes a fact that may not be true
- SOFT_CONTRADICTION_SUSPECTED question asks about contradictory or impossible state
- SOFT_TIME_SENSITIVE question's answer changes over time; may be stale relative to corpus
- SOFT_SCOPE_AMBIGUOUS question's intended scope is unclear (which X, when, where)
- SOFT_OUT_OF_CORPUS_LIKELY question references a private/uploaded/specific document
- SOFT_BROAD_QUANTIFIER question asks for "all X" or "every Y" without bounded scope
- SOFT_MULTI_HOP_REASONING answering requires combining multiple non-obvious facts
- SOFT_SUBJECTIVE question asks for opinion, judgment, or aesthetic preference
OUTPUT FORMAT (strict, one line, no preamble):
LABEL: <one of the labels above>
RATIONALE: <one short sentence>
Do not write anything else. Pick the BEST single label even if multiple apply."""
_LABEL_LINE_RE = re.compile(r"\bLABEL:\s*([A-Z_]+)", re.IGNORECASE)
_RATIONALE_LINE_RE = re.compile(
r"\bRATIONALE:\s*(.+?)(?:\n|$)", re.IGNORECASE | re.DOTALL
)
def _normalize_label(raw: str) -> str:
"""Coerce model output to a known SOFT_* label.
Catches model drift: returns SOFT_PARSE_FAIL when the label
isn't in the enum so the sidecar fails-closed (no soft hint
rather than a misleading one).
"""
cleaned = raw.strip().upper()
# Strip trailing punctuation the model sometimes adds.
cleaned = re.sub(r"[\s\.\,\;\:\!\?]+$", "", cleaned)
# Force SOFT_ prefix even if model dropped it.
if not cleaned.startswith("SOFT_"):
cleaned = "SOFT_" + cleaned
if cleaned in _SOFT_LABELS:
return cleaned
return "SOFT_PARSE_FAIL"
def _parse_soft_hint_response(raw: str) -> tuple[str, str]:
"""Parse the model's two-line response into ``(label, rationale)``.
Robust to minor formatting drift (extra whitespace, trailing
punctuation, missing RATIONALE line). Falls back to
``SOFT_PARSE_FAIL`` when even the LABEL line is missing.
"""
if not raw:
return "SOFT_PARSE_FAIL", "empty response"
label_match = _LABEL_LINE_RE.search(raw)
if not label_match:
# Fallback — if the model just wrote the label without the
# "LABEL:" prefix on the first line, try to extract.
first_line = raw.strip().splitlines()[0] if raw.strip() else ""
candidate = first_line.split()[0] if first_line else ""
label = _normalize_label(candidate) if candidate else "SOFT_PARSE_FAIL"
else:
label = _normalize_label(label_match.group(1))
rationale_match = _RATIONALE_LINE_RE.search(raw)
if rationale_match:
rationale = rationale_match.group(1).strip()
else:
# Some models embed rationale on the same line or use a
# different separator. Fall back to the second line stripped.
lines = [L.strip() for L in raw.strip().splitlines() if L.strip()]
rationale = lines[1] if len(lines) > 1 else "(no rationale)"
return label, rationale[:200] # cap rationale length for hashing
def soft_preflight_question(
question: str,
*,
chat_client, # ChatClient duck-typed
model_id: str,
policy: dict | None = None,
timeout_seconds: float = 5.0,
) -> SoftPreflightHint:
"""Run the soft preflight sidecar on ``question``.
Calls the configured ChatClient with a constrained-generation
prompt asking the model to pick ONE label from ``_SOFT_LABELS``
plus a one-line rationale. Returns a ``SoftPreflightHint``
that downstream surfaces on result dict / run-DAG / bench rows.
Failure modes (all return a stub hint, never raise):
- ``soft_preflight_enabled = False`` (default) ``SOFT_DISABLED``
- LLM raises an exception ``SOFT_PARSE_FAIL``
- LLM returns unparseable text ``SOFT_PARSE_FAIL``
- LLM returns a label not in the enum ``SOFT_PARSE_FAIL``
Confidence scoring is intentionally NOT calibrated against
empirical agreement-rate see #000011 §9. Initial value is
``0.5`` for any successful soft hint, ``0.0`` for stubs. A
future calibration pass could derive confidence from
agreement-with-hard-detectors on a labeled fixture.
"""
policy = policy or {}
if not bool(policy.get("soft_preflight_enabled", False)):
return _stub_hint(
"SOFT_DISABLED",
"sidecar opted-out via policy",
raw_question=question,
model_profile_id=model_id,
)
if not question or not question.strip():
return _stub_hint(
"SOFT_DISABLED",
"empty question",
raw_question=question,
model_profile_id=model_id,
)
t_start = time.monotonic()
try:
raw = chat_client.chat_completion(
messages=[
{"role": "system", "content": _SOFT_PREFLIGHT_SYSTEM_PROMPT},
{"role": "user", "content": question},
],
model=model_id,
temperature=0.0,
max_tokens=128,
top_p=1.0,
)
except Exception as exc: # noqa: BLE001 — sidecar must fail-closed
return _stub_hint(
"SOFT_PARSE_FAIL",
f"chat_client raised: {type(exc).__name__}",
raw_question=question,
model_profile_id=model_id,
elapsed_ms=(time.monotonic() - t_start) * 1000.0,
)
elapsed_ms = (time.monotonic() - t_start) * 1000.0
label, rationale = _parse_soft_hint_response(raw)
confidence = 0.0 if label == "SOFT_PARSE_FAIL" else 0.5
return SoftPreflightHint(
raw_question=question,
sidecar_version=SOFT_PREFLIGHT_VERSION,
classifier_label=label,
confidence=confidence,
rationale=rationale,
timestamp_ns=time.monotonic_ns(),
model_profile_id=model_id,
elapsed_ms=elapsed_ms,
)

File diff suppressed because one or more lines are too long

View file

@ -24,3 +24,89 @@ when did the current CEO of Twitter stop being CEO of Tesla?
# control: well-formed factoid that should NOT trigger any detector
who painted the mona lisa?
# ── extended fixture (2026-05-04) — fitness-of-detector cases ──
#
# Below: edge / borderline / adversarial cases that probe the
# fitness of each detector. Some should fire, some should NOT, some
# should exhibit borderline behavior the operator can debate. The
# fixture lives long-term as a regression suite for the four
# metacog detectors. When the substrate evolves, this file is the
# pin against which behavior is measured.
# ── temporal sensitivity edge cases ──
# implicit-current via role + shape (no `current` keyword)
who is the President of the United States?
# rapid-turnover role with no temporal anchor at all (model-dependent)
who is the CEO of Twitter?
# implicit-now via "this year" — should fire HIGH
which team won this year's World Series?
# verbal future tense — model often hallucinates; not a detector
# target today but worth observing
who will be the next president after the 2028 election?
# negative control: timeless fact with role-shape — should NOT fire
who was the first president of the United States?
# ── contradiction edge cases ──
# multi-pair contradiction (alive+dead AND never+always)
which character was never alive but always dead?
# semantic contradiction without lexical pair — should NOT fire
# (out of detector scope; documents conservative-by-design choice)
which married bachelor lives in this house?
# triple-contradiction stress test
how can a celibate spouse be both unmarried and divorced?
# negative control: not a contradiction, just compound predicates
which characters appear in both Hamlet and Macbeth?
# ── false-premise edge cases ──
# "why did" relation pattern
why did Albert Einstein invent the lightbulb?
# "how did" relation pattern with subtle false premise
how did the Roman Empire conquer Australia?
# "when did" stop pattern with subtle false premise
when did Shakespeare stop writing in French?
# false premise that the lite detector probably MISSES (no
# presupposition trigger pattern) — annotates the detector ceiling
why does NASA fake the moon landings?
# negative control: legitimate "when did X stop Y" with true premise
when did the Roman Empire stop minting silver coins?
# ── out-of-corpus edge cases ──
# explicit "uploaded" reference
in the document I uploaded yesterday, what does section 5 say?
# private notes / inbox
according to my private notes, when did the meeting happen?
# borderline: claims about an existing public document the corpus
# may or may not have (NOT out-of-corpus per current detector)
what does the Wikipedia article on bipolar disorder say?
# ── multi-trigger combinatorics ──
# false-premise + out-of-corpus
when did my uploaded contract stop being valid?
# stale-risk + contradiction
which currently-alive historical figure is also dead?
# ── well-formed controls (should classify SINGULAR / well_formed) ──
what year did World War II end?
who is bilbo baggins's nephew?
list the planets of the solar system

View file

@ -57,7 +57,7 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | open · design only (impl deferred) | 2026-05-04 | D1 (preserves) |
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | closed · landed 2026-05-04 (zero-shot full impl) | 2026-05-04 | D1 (preserves) |
| #000010 | Meta-Cognition Preflight Guard (M0 / MCTL) | closed · landed 2026-05-03 (Phases 14); DAG binding shipped via #000009 | 2026-05-03 | D1, D3 |
| #000009 | Preflight run-DAG node binding (#000008+#000010) | closed · re-landed 2026-05-04 (§8 corrections: reject-path DAG, nested CTI clauses) | 2026-05-03 | D3, D4 |
| #000008 | Broad-quantifier preflight guard | closed · landed in `4f2b5a6`; Phase 5 DAG binding split into #000009 | 2026-05-02 | — |

View file

@ -1,7 +1,7 @@
# Ticket #000011 — SOFT_PREFLIGHT_HINT model-assisted sidecar
**Status:** open · awaiting go/no-go (design only; implementation
deferred)
**Status:** closed · landed 2026-05-04 (zero-shot, full
implementation per §13)
**Opened:** 2026-05-04
**Scope:** Add a model-assisted preflight sidecar that augments the
deterministic detectors from #000010 with an LLM-driven shape
@ -212,12 +212,77 @@ After the sidecar lands:
## 10. Status
Open · awaiting go/no-go. Design captured here; implementation
deferred. Source-doc reference: ticket #000010 §18 +
`~/Downloads/meta-cognition_for_hermes(1).txt` §18.
Closed · landed 2026-05-04 in the same commit batch as the
metacog-trigger fixture expansion + --show-preflight full clause
render + latency profile. Source-doc reference: ticket #000010
§18 + `~/Downloads/meta-cognition_for_hermes(1).txt` §18.
The deterministic substrate from #000010 + #000009 is the
provable foundation. SOFT_PREFLIGHT_HINT is an enrichment
layer that operates strictly within the substrate's "hard
hash never enters the proof path" rule (cf. whitepaper
§13 decision 8).
## 11. What landed
- **`aborist/qa/soft_preflight.py`** — `SoftPreflightHint`
dataclass + `soft_preflight_question()` pure function.
9 canonical labels (`SOFT_WELL_FORMED`,
`SOFT_FALSE_PREMISE_SUSPECTED`, `SOFT_CONTRADICTION_SUSPECTED`,
`SOFT_TIME_SENSITIVE`, `SOFT_SCOPE_AMBIGUOUS`,
`SOFT_OUT_OF_CORPUS_LIKELY`, `SOFT_BROAD_QUANTIFIER`,
`SOFT_MULTI_HOP_REASONING`, `SOFT_SUBJECTIVE`) plus stub
states (`SOFT_DISABLED`, `SOFT_PARSE_FAIL`). Constrained-
generation prompt asking the model to pick ONE label + a
one-line rationale. Failure-closed across every parse path
(raises → `SOFT_PARSE_FAIL`, drift → `SOFT_PARSE_FAIL`).
`SOFT_PREFLIGHT_VERSION = "soft-preflight-v0.1"`.
- **`aborist/qa/query.py`** — wired into the post-classifier /
pre-retrieval segment. One short LLM call (~200ms median),
result surfaces as `soft_preflight_hint` on the result dict
+ `soft_preflight_ms` in the timings dict.
- **`aborist/qa/runner.py` + `aborist/qa/query.py`** —
`soft_preflight_enabled: False` policy default; cache row
identity unchanged (NOT folded into `_VERIFIER_POLICY_FIELDS`
per §4 — soft hints don't gate cache identity).
- **`aborist/cli.py`** — `--soft-preflight` flag on
`aborist query`. Audit-line tail renders soft hints as
`· soft: <label>` (e.g. `· soft: time sensitive`) — distinct
from hard tails so operators see the signal separation at a
glance. SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED
suppress (no actionable signal).
- **`tests/test_soft_preflight.py`** — 25 new tests pinning:
default-OFF behavior, parse-failure modes, label
normalization (SOFT_ prefix enforced even when model drops
it), all 8 actionable labels round-trip, fail-closed on
client exceptions, dataclass JSON round-trip,
rationale-length cap.
## 12. Live verification
End-to-end smoke test on `When did Mr. Burns become Homer's
biological father?` with `--soft-preflight`:
```
EVIDENCE-WARRANTED · via claim_lattice · false premise · soft: time sensitive 1/1 16.4s
```
Both signals compose: hard `· false premise` from #000010
deterministic detector, soft `· soft: time sensitive` from
#000011 sidecar. The model classified a different shape than
the hard detector — by design (soft hints are independent
advisory signals, not redundant with the hard layer).
## 13. Bench plan (deferred)
Per §7 of this ticket — soft-only A/B against the metacog-
trigger fixture (`bench/qa_questions_metacog_subset.txt`
expanded 2026-05-04 to 28 questions), measure agreement-rate
between soft sidecar and hard detectors. Disagreement ≤ 10%
signals the sidecar is reliable enough for default-on
consideration. This bench is queued but not run in the same
commit cycle as implementation.

View file

@ -0,0 +1,232 @@
"""Soft preflight sidecar (#000011) tests.
The sidecar produces ONLY soft hints labeled SOFT_*; it cannot
create PREFLIGHT_OK or PREFLIGHT_BLOCKED. These tests pin:
- default-OFF (sidecar disabled by default)
- parse-failure modes (all return stub, never raise)
- label normalization (SOFT_ prefix enforced)
- rationale extraction
- the dataclass schema
Uses a fake ChatClient (mock chat_completion) no LLM round-trip.
"""
from __future__ import annotations
import pytest
from aborist.qa.soft_preflight import (
SOFT_PREFLIGHT_VERSION,
SoftPreflightHint,
_normalize_label,
_parse_soft_hint_response,
soft_preflight_question,
)
class _FakeChatClient:
"""Mock ChatClient that returns a configured response."""
def __init__(self, response: str):
self.response = response
self.call_count = 0
def chat_completion(self, *args, **kwargs) -> str:
self.call_count += 1
return self.response
class _RaisingChatClient:
"""Mock that raises on call — tests fail-closed behavior."""
def chat_completion(self, *args, **kwargs) -> str:
raise RuntimeError("simulated network failure")
# ----------------------------------------------------------- defaults
def test_sidecar_default_off_returns_stub_hint():
"""Without soft_preflight_enabled=True in policy, sidecar
returns SOFT_DISABLED stub. No LLM call happens."""
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
hint = soft_preflight_question(
"what is the capital of france?",
chat_client=client,
model_id="test-model",
policy={}, # no soft_preflight_enabled
)
assert hint.classifier_label == "SOFT_DISABLED"
assert hint.confidence == 0.0
assert client.call_count == 0 # no LLM call
def test_sidecar_explicit_off_returns_stub():
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
hint = soft_preflight_question(
"anything", chat_client=client, model_id="test",
policy={"soft_preflight_enabled": False},
)
assert hint.classifier_label == "SOFT_DISABLED"
assert client.call_count == 0
def test_empty_question_returns_stub_even_when_enabled():
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
hint = soft_preflight_question(
"", chat_client=client, model_id="test",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_DISABLED"
assert client.call_count == 0
# ----------------------------------------------------------- happy path
def test_sidecar_returns_soft_hint_when_enabled():
client = _FakeChatClient(
"LABEL: SOFT_FALSE_PREMISE_SUSPECTED\n"
"RATIONALE: question presupposes Mr. Burns is Homer's father"
)
hint = soft_preflight_question(
"When did Mr. Burns become Homer's biological father?",
chat_client=client,
model_id="hermes-test",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_FALSE_PREMISE_SUSPECTED"
assert hint.confidence == 0.5
assert "Mr. Burns" in hint.rationale
assert hint.model_profile_id == "hermes-test"
assert hint.sidecar_version == SOFT_PREFLIGHT_VERSION
assert client.call_count == 1
@pytest.mark.parametrize("label", [
"SOFT_WELL_FORMED",
"SOFT_CONTRADICTION_SUSPECTED",
"SOFT_TIME_SENSITIVE",
"SOFT_OUT_OF_CORPUS_LIKELY",
"SOFT_BROAD_QUANTIFIER",
"SOFT_MULTI_HOP_REASONING",
"SOFT_SUBJECTIVE",
"SOFT_SCOPE_AMBIGUOUS",
])
def test_each_canonical_label_passes_through(label):
client = _FakeChatClient(f"LABEL: {label}\nRATIONALE: test")
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == label
# ----------------------------------------------------------- failure modes
def test_chat_client_exception_returns_parse_fail():
"""Sidecar fails-closed on chat_completion exceptions."""
client = _RaisingChatClient()
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_PARSE_FAIL"
assert hint.confidence == 0.0
assert "RuntimeError" in hint.rationale
def test_unparseable_response_returns_parse_fail():
client = _FakeChatClient("garbage output no label here")
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_PARSE_FAIL"
assert hint.confidence == 0.0
def test_model_drift_label_outside_enum_returns_parse_fail():
"""Model returns a label not in the enum → fail-closed."""
client = _FakeChatClient(
"LABEL: SOFT_INVENTED_NEW_LABEL\nRATIONALE: model drift"
)
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_PARSE_FAIL"
def test_label_without_soft_prefix_gets_normalized():
"""Model drops the SOFT_ prefix → normalizer adds it back."""
client = _FakeChatClient("LABEL: WELL_FORMED\nRATIONALE: ok")
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_WELL_FORMED"
def test_label_with_trailing_punctuation_normalized():
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED.\nRATIONALE: ok")
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
assert hint.classifier_label == "SOFT_WELL_FORMED"
# ----------------------------------------------------------- pure helpers
def test_normalize_label_handles_whitespace():
assert _normalize_label(" SOFT_WELL_FORMED ") == "SOFT_WELL_FORMED"
def test_normalize_label_uppercases():
assert _normalize_label("soft_well_formed") == "SOFT_WELL_FORMED"
def test_normalize_label_returns_parse_fail_for_garbage():
assert _normalize_label("just garbage") == "SOFT_PARSE_FAIL"
def test_parse_soft_hint_handles_mixed_case_keys():
label, rationale = _parse_soft_hint_response(
"label: SOFT_WELL_FORMED\nrationale: ok"
)
assert label == "SOFT_WELL_FORMED"
assert rationale == "ok"
def test_parse_soft_hint_handles_missing_rationale_line():
"""When the model only writes LABEL: but no RATIONALE: line."""
label, rationale = _parse_soft_hint_response("LABEL: SOFT_WELL_FORMED")
assert label == "SOFT_WELL_FORMED"
assert rationale # something, even if "(no rationale)"
def test_parse_soft_hint_caps_rationale_length():
"""Long rationales get capped to keep payload bounded."""
long_rationale = "x" * 500
label, rationale = _parse_soft_hint_response(
f"LABEL: SOFT_WELL_FORMED\nRATIONALE: {long_rationale}"
)
assert len(rationale) <= 200
# ----------------------------------------------------------- dataclass schema
def test_hint_to_dict_is_json_serializable():
"""Bench rows / run-DAG persist soft hints as JSON; the
dataclass round-trips cleanly."""
import json
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
hint = soft_preflight_question(
"test", chat_client=client, model_id="m",
policy={"soft_preflight_enabled": True},
)
d = hint.to_dict()
json.dumps(d, ensure_ascii=False) # raises if non-serializable
def test_version_pinned():
assert SOFT_PREFLIGHT_VERSION == "soft-preflight-v0.1"