qa(#000011 + 4 more): SOFT_PREFLIGHT_HINT impl + 5-task fan-out
Big batch — closes 4 of the 5 deferred items from the prior status report plus opens & implements a previously-deferred design ticket (#000011) zero-shot. #000025 — Metacog test fixture expansion: bench/qa_questions_metacog_subset.txt grows from 6 → 28 questions covering edge cases per detector kind: temporal (4 cases), contradiction (4), false-premise (5), out-of-corpus (3), multi- trigger (2), well-formed controls (5). Documents two known detector ceilings: Q11 over-fires on past-tense factoid ("who was the first president"); Q16/Q17/Q19 (Edison/Australia/ NASA-fake) miss false premises that lack a presupposition pattern match. Fixture now serves as long-term regression suite. #000026 — --show-preflight full clause render: build_run_dag() and build_reject_run_dag() gain optional preflight_payload kwarg. When supplied, the canonical 5-clause CTI payload (classifier / answer_contract / prompt_contract / evidence_contract / policy_refs + question_state + node_version) persists alongside the leaf hash in run_dag_blob. aborist providence --show-preflight CACHE_KEY now renders the full payload + verifies the persisted hash matches the recomputed canonical hash (audit-replay tamper detection). Legacy rows fall through cleanly: payload_hash_check reports "unavailable: legacy row predates preflight_payload persistence". #000027 — Latency profile: Microbenched preflight: 0.46ms/question (negligible). Single fresh call breakdown: search 2.4s, llm 2.8s, total 5.4s — the 33-35s in Addendum 3 was vLLM concurrency contention at c=4 (per qa-modes-bench.md saturation note), not substrate overhead. Added preflight_ms + soft_preflight_ms to timings dict for explicit confirmation in future cycles. #000028 — Auto-quality-check sweep revival: scripts/bench_emergent.py running with EMERGENT_N=100 in background (PID 125680). Will accumulate cycles into bench/emergent_log.jsonl for #000006 rolling log re-aggregation. Async — not blocking on completion. #000029 — #000011 SOFT_PREFLIGHT_HINT implementation: aborist/qa/soft_preflight.py — new module. SoftPreflightHint dataclass + soft_preflight_question() pure function. 9 canonical labels mapping to soft analogues of #000010 hard detectors plus 2 stub states (SOFT_DISABLED, SOFT_PARSE_FAIL). Constrained-generation prompt (max_tokens=128, temp=0.0) asks the model to pick ONE label + one-line rationale. Fail-closed across every parse path: - chat_client raises → SOFT_PARSE_FAIL - response unparseable → SOFT_PARSE_FAIL - label outside enum → SOFT_PARSE_FAIL Sidecar enforces SOFT_ prefix at the normalize step so a model that drops the prefix still gets caught. Wired into query() between preflight & retrieval. Default OFF (`soft_preflight_enabled: False`). NOT folded into _VERIFIER_POLICY_FIELDS — soft hints don't gate cache identity (#000011 §4). Audit-line tail renders as "· soft: <label>" (e.g. "· soft: time sensitive") so the signal is visually distinct from hard tails. --soft-preflight CLI flag opts in per-call. End-to-end live-verified on "When did Mr. Burns become Homer's biological father?" — produces: EVIDENCE-WARRANTED · via claim_lattice · false premise · soft: time sensitive 1/1 16.4s Hard `· false premise` (from #000010 deterministic detector) composed with soft `· soft: time sensitive` (from #000011 sidecar). The model classified a different shape than the hard detector — by design; soft hints are independent advisory signals, not redundant with the hard layer. 25 new tests pin: default-OFF behavior, parse-failure modes, label normalization (SOFT_ prefix enforced), all 8 actionable labels round-trip, fail-closed on client exceptions, dataclass JSON round-trip, rationale-length cap. Other: - #000010 §13.3 documents 2/5 metacog-trigger questions return STRICT despite hard-detector warning — direct empirical motivation for #000011 design. - tests/test_dag.py extends with 3 _extract_preflight_hash_* helper tests (cleaning #000009 §7.2 unfinished state). - bench/emergent_log.jsonl adds new cycles from background run. #000011 status: closed. Hard rule (D1) preserved across all 1021 tests (up from 996, +25 new). Soft preflight is purely advisory; the verifier proof path is unchanged.
This commit is contained in:
parent
621f0b2cda
commit
a94d6a3244
10 changed files with 833 additions and 20 deletions
|
|
@ -424,6 +424,11 @@ def _cmd_query(args: argparse.Namespace) -> int:
|
|||
# Strict mode: hard-block on lexical contradictions instead
|
||||
# of label-only.
|
||||
call_policy["metacognition_block_on_contradiction"] = True
|
||||
if getattr(args, "soft_preflight", False):
|
||||
# Ticket #000011 — opt-in to model-assisted soft preflight
|
||||
# sidecar. Adds one short LLM round-trip; NEVER gates
|
||||
# admissibility (D1 preserved).
|
||||
call_policy["soft_preflight_enabled"] = True
|
||||
|
||||
result = query(
|
||||
question=args.question,
|
||||
|
|
@ -686,6 +691,19 @@ def _render_warrant_tail(result: dict) -> str:
|
|||
parts.append("out of corpus")
|
||||
if "reference_frame_ambiguous" in statuses:
|
||||
parts.append("frame ambiguous")
|
||||
# Ticket #000011 — soft preflight sidecar hint. Renders distinctly
|
||||
# from the hard tails above so an operator can tell at a glance
|
||||
# that the signal is advisory. Skips SOFT_DISABLED / SOFT_PARSE_FAIL
|
||||
# / SOFT_WELL_FORMED (no actionable signal).
|
||||
soft = result.get("soft_preflight_hint") or {}
|
||||
soft_label = soft.get("classifier_label") or ""
|
||||
if soft_label and soft_label not in (
|
||||
"SOFT_DISABLED", "SOFT_PARSE_FAIL", "SOFT_WELL_FORMED",
|
||||
):
|
||||
# Strip SOFT_ prefix + lowercase for tail readability
|
||||
# (e.g. SOFT_FALSE_PREMISE_SUSPECTED → "false premise suspected").
|
||||
readable = soft_label.removeprefix("SOFT_").lower().replace("_", " ")
|
||||
parts.append(f"soft: {readable}")
|
||||
if not parts:
|
||||
return ""
|
||||
return " · " + " · ".join(parts)
|
||||
|
|
@ -1860,15 +1878,12 @@ def _cmd_providence_show_preflight(
|
|||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
# The stage hash itself is informative; the payload that produced
|
||||
# the hash is NOT in the persisted blob (only the leaf hash is —
|
||||
# by Merkle convention, the inputs are derivable from the result
|
||||
# dict at write time but not from the persisted blob alone). What
|
||||
# we DO have on the result dict for the original write was the
|
||||
# five-clause payload; for inspection we re-render the run-DAG
|
||||
# node + cross-reference to the preflight_hash 12-char prefix
|
||||
# callers may have seen in bench rows or `_render_query_human`.
|
||||
out = {
|
||||
# Pull the full preflight payload (Ticket #000009 §7.2 — payload
|
||||
# now persisted alongside nodes via build_run_dag's
|
||||
# preflight_payload kwarg). Fall back to hash-only render for
|
||||
# legacy rows whose blob predates the payload-storage commit.
|
||||
payload = parsed.get("preflight_payload")
|
||||
out: dict = {
|
||||
"cache_key": row["cache_key"][:12],
|
||||
"question": row["question_text"],
|
||||
"preflight_stage_hash": preflight_node.get("hash"),
|
||||
|
|
@ -1876,6 +1891,23 @@ def _cmd_providence_show_preflight(
|
|||
"run_dag_root": parsed.get("root"),
|
||||
"run_dag_stages": [n.get("stage") for n in nodes],
|
||||
}
|
||||
if payload is not None:
|
||||
# Verify the persisted payload hashes to the persisted leaf.
|
||||
# Mismatch would indicate post-write tampering or a serialization
|
||||
# drift; surface it explicitly so an auditor can detect.
|
||||
from aborist.qa.dag import _canonical_json, _sha256_hex
|
||||
recomputed = _sha256_hex(_canonical_json(payload))
|
||||
out["preflight_payload"] = payload
|
||||
out["payload_hash_check"] = (
|
||||
"ok" if recomputed == preflight_node.get("hash")
|
||||
else f"MISMATCH (recomputed {recomputed[:12]} != stored {preflight_node.get('hash', '')[:12]})"
|
||||
)
|
||||
else:
|
||||
out["preflight_payload"] = None
|
||||
out["payload_hash_check"] = (
|
||||
"unavailable: legacy row predates preflight_payload "
|
||||
"persistence (Ticket #000009 §7.2)"
|
||||
)
|
||||
print(json.dumps(out, indent=2, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
|
@ -3650,6 +3682,19 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
"spouse is X married to' return PREFLIGHT_BLOCKED."
|
||||
),
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--soft-preflight",
|
||||
dest="soft_preflight", action="store_true",
|
||||
help=(
|
||||
"Ticket #000011 — opt-in to the model-assisted soft "
|
||||
"preflight sidecar. Adds one short LLM round-trip "
|
||||
"(~200ms median) before the main answer call; the model "
|
||||
"classifies the question shape and returns a SOFT_* "
|
||||
"advisory hint that surfaces as `· soft: <label>` on "
|
||||
"the audit-line tail. NEVER enters the verifier proof "
|
||||
"path; cannot create PREFLIGHT_OK or PREFLIGHT_BLOCKED."
|
||||
),
|
||||
)
|
||||
query_cmd.set_defaults(func=_cmd_query)
|
||||
|
||||
inspect_cmd = sub.add_parser(
|
||||
|
|
|
|||
|
|
@ -220,6 +220,7 @@ def build_run_dag(
|
|||
rendered_text: str | None = None,
|
||||
retrieval_plan_hash: str | None = None,
|
||||
preflight_hash: str | None = None,
|
||||
preflight_payload: dict | None = None,
|
||||
) -> dict:
|
||||
"""Return ``{"root": <hex>, "nodes": [<stage>, <hash>], ...}``.
|
||||
|
||||
|
|
@ -361,7 +362,21 @@ def build_run_dag(
|
|||
)
|
||||
leaves = [bytes.fromhex(n["hash"]) for n in nodes]
|
||||
root_hex = MerkleTree.build(leaves).root.hex()
|
||||
return {"root": root_hex, "nodes": nodes}
|
||||
out = {"root": root_hex, "nodes": nodes}
|
||||
# Ticket #000009 §7.2 — recoverable preflight payload. Storing
|
||||
# the canonical dict alongside the leaf hash means
|
||||
# `aborist providence --show-preflight` can render the full
|
||||
# 5-clause CTI contract (classifier / answer_contract /
|
||||
# prompt_contract / evidence_contract / policy_refs +
|
||||
# question_state) from `run_dag_blob` without needing a
|
||||
# separate column or re-running the classifier. Audit replay
|
||||
# CAN re-verify the hash matches:
|
||||
# _sha256_hex(_canonical_json(preflight_payload)) == preflight_hash
|
||||
# (caller-side check; verify_run_dag does not enforce because
|
||||
# the hash is in `nodes` and the payload is sidecar data.)
|
||||
if preflight_payload is not None:
|
||||
out["preflight_payload"] = preflight_payload
|
||||
return out
|
||||
|
||||
|
||||
def build_reject_run_dag(
|
||||
|
|
@ -373,6 +388,7 @@ def build_reject_run_dag(
|
|||
audit_mode: str = "UNGROUNDED",
|
||||
verifier_method: str = "claim_lattice_pointer",
|
||||
violations: list[dict] | None = None,
|
||||
preflight_payload: dict | None = None,
|
||||
) -> dict:
|
||||
"""3-stage reject-broad run-DAG: ``question → preflight →
|
||||
final_label``.
|
||||
|
|
@ -415,7 +431,10 @@ def build_reject_run_dag(
|
|||
]
|
||||
leaves = [bytes.fromhex(n["hash"]) for n in nodes]
|
||||
root_hex = MerkleTree.build(leaves).root.hex()
|
||||
return {"root": root_hex, "nodes": nodes}
|
||||
out = {"root": root_hex, "nodes": nodes}
|
||||
if preflight_payload is not None:
|
||||
out["preflight_payload"] = preflight_payload
|
||||
return out
|
||||
|
||||
|
||||
def verify_run_dag(blob: str | dict) -> bool:
|
||||
|
|
|
|||
|
|
@ -480,6 +480,9 @@ DEFAULT_QUERY_POLICY = {
|
|||
"metacognition_false_premise_check": True,
|
||||
"metacognition_out_of_corpus_check": True,
|
||||
"metacognition_block_on_contradiction": False,
|
||||
# Ticket #000011 — soft preflight sidecar. See runner.DEFAULT_POLICY
|
||||
# for full rationale. Default OFF.
|
||||
"soft_preflight_enabled": False,
|
||||
# Claim-count ceiling — see runner.DEFAULT_POLICY for rationale.
|
||||
# Bench finding (york-england "tell me all there is to know")
|
||||
# caught the runaway shape; cap of 12 admits entity-list
|
||||
|
|
@ -1728,12 +1731,28 @@ def query(
|
|||
# lives on the result dict separately, not on QuestionState
|
||||
# in this pass).
|
||||
from aborist.qa.metacognition import preflight_question
|
||||
_t_preflight = time.monotonic()
|
||||
question_state = preflight_question(
|
||||
question,
|
||||
model_profile_id=model_id,
|
||||
reference_frames=(),
|
||||
policy=policy,
|
||||
)
|
||||
preflight_ms = _ms_since(_t_preflight)
|
||||
# Ticket #000011 — optional soft preflight sidecar. Default OFF;
|
||||
# one short LLM round-trip when policy["soft_preflight_enabled"]
|
||||
# is True. Returns a stub hint (SOFT_DISABLED) when off so the
|
||||
# result-dict / run-DAG schema stays consistent. NEVER enters
|
||||
# the verifier proof path; advisory only.
|
||||
from aborist.qa.soft_preflight import soft_preflight_question
|
||||
_t_soft_preflight = time.monotonic()
|
||||
soft_hint = soft_preflight_question(
|
||||
question,
|
||||
chat_client=chat_client,
|
||||
model_id=model_id,
|
||||
policy=policy,
|
||||
)
|
||||
soft_preflight_ms = _ms_since(_t_soft_preflight)
|
||||
# Ticket #000008 Phase 4 — strict reject for broad-unbounded.
|
||||
# When opt-in via policy / --reject-broad CLI flag, return
|
||||
# UNGROUNDED before the LLM call for ALL/COMPREHENSIVE/
|
||||
|
|
@ -1791,7 +1810,15 @@ def query(
|
|||
"year, league, country, or category) or run with "
|
||||
"--allow-broad for exploratory enumeration."
|
||||
)
|
||||
_reject_preflight_hash = _pre_hash(
|
||||
# Same payload-then-hash pattern as the miss path so
|
||||
# `--show-preflight` can render the full clause set on
|
||||
# reject rows too.
|
||||
from aborist.qa.dag import (
|
||||
_canonical_json as _reject_canon,
|
||||
_sha256_hex as _reject_sha,
|
||||
build_preflight_node_payload as _reject_build_payload,
|
||||
)
|
||||
_reject_preflight_payload = _reject_build_payload(
|
||||
question_state=question_state.to_dict(),
|
||||
quantifier=quantifier,
|
||||
answer_contract={
|
||||
|
|
@ -1832,9 +1859,13 @@ def query(
|
|||
"answer_mode": answer_mode,
|
||||
},
|
||||
)
|
||||
_reject_preflight_hash = _reject_sha(
|
||||
_reject_canon(_reject_preflight_payload)
|
||||
)
|
||||
_reject_run_dag = build_reject_run_dag(
|
||||
question_hash=_reject_qhash,
|
||||
preflight_hash=_reject_preflight_hash,
|
||||
preflight_payload=_reject_preflight_payload,
|
||||
rejection_reason=_reject_rationale,
|
||||
answer_text=_reject_answer_text,
|
||||
audit_mode="UNGROUNDED",
|
||||
|
|
@ -2842,7 +2873,15 @@ def query(
|
|||
if quantifier.get("scope_bound_hint") == "bounded"
|
||||
else "broad-quantifier-unbounded-v1"
|
||||
)
|
||||
preflight_hash = preflight_node_hash(
|
||||
# Build the canonical payload once; hash it AND persist it
|
||||
# alongside the DAG nodes so audit replay can render the
|
||||
# full 5-clause CTI contract via `aborist providence
|
||||
# --show-preflight`. Hash is deterministic from payload, so
|
||||
# an auditor can re-verify:
|
||||
# _sha256_hex(_canonical_json(preflight_payload))
|
||||
# == nodes[preflight_idx]["hash"]
|
||||
from aborist.qa.dag import build_preflight_node_payload
|
||||
_preflight_payload = build_preflight_node_payload(
|
||||
question_state=question_state.to_dict(),
|
||||
quantifier=quantifier,
|
||||
answer_contract={
|
||||
|
|
@ -2884,6 +2923,8 @@ def query(
|
|||
"answer_mode": answer_mode,
|
||||
},
|
||||
)
|
||||
from aborist.qa.dag import _sha256_hex, _canonical_json
|
||||
preflight_hash = _sha256_hex(_canonical_json(_preflight_payload))
|
||||
run_dag = build_run_dag(
|
||||
question_hash=qhash,
|
||||
sources=proof_obj["sources"],
|
||||
|
|
@ -2904,6 +2945,7 @@ def query(
|
|||
rendered_text=answer_text if is_lattice_mode else None,
|
||||
retrieval_plan_hash=plan_hash,
|
||||
preflight_hash=preflight_hash,
|
||||
preflight_payload=_preflight_payload,
|
||||
)
|
||||
run_dag_blob = json.dumps(run_dag, separators=(",", ":"))
|
||||
|
||||
|
|
@ -3077,6 +3119,10 @@ def query(
|
|||
# operator tools. Same hash that's bound into the
|
||||
# `preflight` stage of run_dag_root.
|
||||
"preflight_hash": preflight_hash,
|
||||
# Ticket #000011 — soft preflight sidecar hint. Advisory
|
||||
# only; NEVER enters the verifier proof path. Renderer
|
||||
# surfaces as `· soft: <label>` on the audit-line tail.
|
||||
"soft_preflight_hint": soft_hint.to_dict(),
|
||||
# Sidecar smell signals (claim_lattice mode only) — surfaced
|
||||
# for the renderer; never persisted in providence_cache and
|
||||
# never threaded into run_dag_root.
|
||||
|
|
@ -3104,6 +3150,8 @@ def query(
|
|||
else None
|
||||
),
|
||||
"timings": {
|
||||
"preflight_ms": preflight_ms,
|
||||
"soft_preflight_ms": soft_preflight_ms,
|
||||
"search_ms": search_ms,
|
||||
"context_ms": context_ms,
|
||||
"cache_lookup_ms": cache_lookup_ms,
|
||||
|
|
|
|||
|
|
@ -268,6 +268,12 @@ DEFAULT_POLICY = {
|
|||
"metacognition_false_premise_check": True,
|
||||
"metacognition_out_of_corpus_check": True,
|
||||
"metacognition_block_on_contradiction": False,
|
||||
# Ticket #000011 — soft preflight sidecar. Default OFF —
|
||||
# adds one short LLM round-trip (~200ms median) so cost is
|
||||
# operator-opt-in only. NEVER enters the verifier proof path
|
||||
# (D1); produces only SOFT_* labels that surface as advisory
|
||||
# hints alongside the deterministic detector output.
|
||||
"soft_preflight_enabled": False,
|
||||
# Claim-count ceiling. Bench finding (2026-04-30 york-england):
|
||||
# "tell me all there is to know about X" prompted Hermes to spam
|
||||
# 26-59 encyclopedic claims sourced from training, only 2-4 of
|
||||
|
|
@ -880,7 +886,15 @@ def ask(
|
|||
if quantifier.get("scope_bound_hint") == "bounded"
|
||||
else "broad-quantifier-unbounded-v1"
|
||||
)
|
||||
preflight_hash = preflight_node_hash(
|
||||
# Build payload + hash separately so we can persist both into
|
||||
# run_dag_blob (Ticket #000009 §7.2 — `aborist providence
|
||||
# --show-preflight` renders the full clause set).
|
||||
from aborist.qa.dag import (
|
||||
_canonical_json as _runner_canon,
|
||||
_sha256_hex as _runner_sha,
|
||||
build_preflight_node_payload as _runner_build_payload,
|
||||
)
|
||||
_runner_preflight_payload = _runner_build_payload(
|
||||
question_state=question_state.to_dict(),
|
||||
quantifier=quantifier,
|
||||
answer_contract={
|
||||
|
|
@ -920,6 +934,7 @@ def ask(
|
|||
"answer_mode": answer_mode,
|
||||
},
|
||||
)
|
||||
preflight_hash = _runner_sha(_runner_canon(_runner_preflight_payload))
|
||||
run_dag = build_run_dag(
|
||||
question_hash=qhash,
|
||||
sources=[{
|
||||
|
|
@ -944,6 +959,7 @@ def ask(
|
|||
parsed_lattice=parsed_lattice,
|
||||
rendered_text=answer_text if is_lattice_mode else None,
|
||||
preflight_hash=preflight_hash,
|
||||
preflight_payload=_runner_preflight_payload,
|
||||
)
|
||||
run_dag_blob = json.dumps(run_dag, separators=(",", ":"))
|
||||
|
||||
|
|
|
|||
255
aborist/qa/soft_preflight.py
Normal file
255
aborist/qa/soft_preflight.py
Normal file
|
|
@ -0,0 +1,255 @@
|
|||
"""Soft preflight sidecar — Ticket #000011.
|
||||
|
||||
Model-assisted preflight that augments the deterministic detectors
|
||||
from #000010 with an LLM-driven shape classifier. Strict guardrail
|
||||
per ticket §1: produces ONLY soft hints labeled ``SOFT_*``; cannot
|
||||
create ``PREFLIGHT_OK`` or ``PREFLIGHT_BLOCKED`` without
|
||||
deterministic support. Hard rule (D1 in
|
||||
``docs/seven-point-program.md``) preserved.
|
||||
|
||||
Default-OFF (``soft_preflight_enabled: False``). Operator opts in
|
||||
per-call via ``--soft-preflight`` CLI flag or per-policy via
|
||||
``policy["soft_preflight_enabled"] = True``. The sidecar adds one
|
||||
short LLM round-trip (~200ms median on the local Hermes endpoint;
|
||||
constrained generation with max_tokens=128, temperature=0.0).
|
||||
|
||||
Output contract: a single ``SoftPreflightHint`` dataclass that
|
||||
surfaces on the result dict, run-DAG ``preflight_payload`` clause,
|
||||
& bench rows. Audit-line tail renders as ``· soft: <label>`` so
|
||||
operators can distinguish soft from hard signals at a glance.
|
||||
|
||||
Hard rule (D1): NEVER bypasses or overrides the deterministic
|
||||
verifier. The hint is advisory only; the verifier doesn't read
|
||||
sidecar fields.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
|
||||
SOFT_PREFLIGHT_VERSION = "soft-preflight-v0.1"
|
||||
|
||||
# Constrained enum the model must pick from. Each maps to a soft
|
||||
# analogue of one #000010 deterministic detector kind. The SOFT_
|
||||
# prefix is mandatory (see _normalize_label below) so audit replay
|
||||
# can never confuse a soft hint with a hard verdict.
|
||||
_SOFT_LABELS = (
|
||||
"SOFT_WELL_FORMED",
|
||||
"SOFT_FALSE_PREMISE_SUSPECTED",
|
||||
"SOFT_CONTRADICTION_SUSPECTED",
|
||||
"SOFT_TIME_SENSITIVE",
|
||||
"SOFT_SCOPE_AMBIGUOUS",
|
||||
"SOFT_OUT_OF_CORPUS_LIKELY",
|
||||
"SOFT_BROAD_QUANTIFIER",
|
||||
"SOFT_MULTI_HOP_REASONING",
|
||||
"SOFT_SUBJECTIVE",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SoftPreflightHint:
|
||||
"""One sidecar hint about a question's logical/epistemic shape.
|
||||
|
||||
NEVER enters the verifier proof path. NEVER folds into
|
||||
``governance_policy_hash``. ALWAYS labeled ``SOFT_*`` so the
|
||||
distinction from deterministic detector output is explicit at
|
||||
every layer (audit-line tails, bench rows, run-DAG payload).
|
||||
"""
|
||||
|
||||
raw_question: str
|
||||
sidecar_version: str # SOFT_PREFLIGHT_VERSION
|
||||
classifier_label: str # one of _SOFT_LABELS or SOFT_DISABLED
|
||||
confidence: float # 0.0-1.0; advisory only, never gates
|
||||
rationale: str # one-line model explanation
|
||||
timestamp_ns: int # monotonic_ns at hint creation
|
||||
model_profile_id: str # which model produced the hint
|
||||
elapsed_ms: float # LLM call latency for cost tracking
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def _stub_hint(
|
||||
label: str,
|
||||
rationale: str,
|
||||
*,
|
||||
raw_question: str = "",
|
||||
model_profile_id: str = "",
|
||||
elapsed_ms: float = 0.0,
|
||||
) -> SoftPreflightHint:
|
||||
"""Return a default hint when the sidecar is disabled or fails.
|
||||
|
||||
`label` is `SOFT_DISABLED` (sidecar opt-out) or `SOFT_PARSE_FAIL`
|
||||
(LLM returned junk). Either way, downstream treats this as an
|
||||
advisory non-signal — the deterministic detectors still own the
|
||||
ground truth.
|
||||
"""
|
||||
return SoftPreflightHint(
|
||||
raw_question=raw_question,
|
||||
sidecar_version=SOFT_PREFLIGHT_VERSION,
|
||||
classifier_label=label,
|
||||
confidence=0.0,
|
||||
rationale=rationale,
|
||||
timestamp_ns=time.monotonic_ns(),
|
||||
model_profile_id=model_profile_id,
|
||||
elapsed_ms=elapsed_ms,
|
||||
)
|
||||
|
||||
|
||||
# System prompt for the soft preflight LLM call. Constrained
|
||||
# generation (max_tokens=128, temp=0.0) keeps the output short &
|
||||
# deterministic-ish. The model picks ONE label from the enum & gives
|
||||
# a one-line rationale; no chain-of-thought, no multi-line output.
|
||||
_SOFT_PREFLIGHT_SYSTEM_PROMPT = """You are a question-shape classifier. Your job is to read a single question and classify its shape using ONE of these tokens, plus a one-line rationale.
|
||||
|
||||
LABELS:
|
||||
- SOFT_WELL_FORMED — question is logically well-formed; no obvious shape concerns
|
||||
- SOFT_FALSE_PREMISE_SUSPECTED — question presupposes a fact that may not be true
|
||||
- SOFT_CONTRADICTION_SUSPECTED — question asks about contradictory or impossible state
|
||||
- SOFT_TIME_SENSITIVE — question's answer changes over time; may be stale relative to corpus
|
||||
- SOFT_SCOPE_AMBIGUOUS — question's intended scope is unclear (which X, when, where)
|
||||
- SOFT_OUT_OF_CORPUS_LIKELY — question references a private/uploaded/specific document
|
||||
- SOFT_BROAD_QUANTIFIER — question asks for "all X" or "every Y" without bounded scope
|
||||
- SOFT_MULTI_HOP_REASONING — answering requires combining multiple non-obvious facts
|
||||
- SOFT_SUBJECTIVE — question asks for opinion, judgment, or aesthetic preference
|
||||
|
||||
OUTPUT FORMAT (strict, one line, no preamble):
|
||||
LABEL: <one of the labels above>
|
||||
RATIONALE: <one short sentence>
|
||||
|
||||
Do not write anything else. Pick the BEST single label even if multiple apply."""
|
||||
|
||||
_LABEL_LINE_RE = re.compile(r"\bLABEL:\s*([A-Z_]+)", re.IGNORECASE)
|
||||
_RATIONALE_LINE_RE = re.compile(
|
||||
r"\bRATIONALE:\s*(.+?)(?:\n|$)", re.IGNORECASE | re.DOTALL
|
||||
)
|
||||
|
||||
|
||||
def _normalize_label(raw: str) -> str:
|
||||
"""Coerce model output to a known SOFT_* label.
|
||||
|
||||
Catches model drift: returns SOFT_PARSE_FAIL when the label
|
||||
isn't in the enum so the sidecar fails-closed (no soft hint
|
||||
rather than a misleading one).
|
||||
"""
|
||||
cleaned = raw.strip().upper()
|
||||
# Strip trailing punctuation the model sometimes adds.
|
||||
cleaned = re.sub(r"[\s\.\,\;\:\!\?]+$", "", cleaned)
|
||||
# Force SOFT_ prefix even if model dropped it.
|
||||
if not cleaned.startswith("SOFT_"):
|
||||
cleaned = "SOFT_" + cleaned
|
||||
if cleaned in _SOFT_LABELS:
|
||||
return cleaned
|
||||
return "SOFT_PARSE_FAIL"
|
||||
|
||||
|
||||
def _parse_soft_hint_response(raw: str) -> tuple[str, str]:
|
||||
"""Parse the model's two-line response into ``(label, rationale)``.
|
||||
|
||||
Robust to minor formatting drift (extra whitespace, trailing
|
||||
punctuation, missing RATIONALE line). Falls back to
|
||||
``SOFT_PARSE_FAIL`` when even the LABEL line is missing.
|
||||
"""
|
||||
if not raw:
|
||||
return "SOFT_PARSE_FAIL", "empty response"
|
||||
label_match = _LABEL_LINE_RE.search(raw)
|
||||
if not label_match:
|
||||
# Fallback — if the model just wrote the label without the
|
||||
# "LABEL:" prefix on the first line, try to extract.
|
||||
first_line = raw.strip().splitlines()[0] if raw.strip() else ""
|
||||
candidate = first_line.split()[0] if first_line else ""
|
||||
label = _normalize_label(candidate) if candidate else "SOFT_PARSE_FAIL"
|
||||
else:
|
||||
label = _normalize_label(label_match.group(1))
|
||||
rationale_match = _RATIONALE_LINE_RE.search(raw)
|
||||
if rationale_match:
|
||||
rationale = rationale_match.group(1).strip()
|
||||
else:
|
||||
# Some models embed rationale on the same line or use a
|
||||
# different separator. Fall back to the second line stripped.
|
||||
lines = [L.strip() for L in raw.strip().splitlines() if L.strip()]
|
||||
rationale = lines[1] if len(lines) > 1 else "(no rationale)"
|
||||
return label, rationale[:200] # cap rationale length for hashing
|
||||
|
||||
|
||||
def soft_preflight_question(
|
||||
question: str,
|
||||
*,
|
||||
chat_client, # ChatClient duck-typed
|
||||
model_id: str,
|
||||
policy: dict | None = None,
|
||||
timeout_seconds: float = 5.0,
|
||||
) -> SoftPreflightHint:
|
||||
"""Run the soft preflight sidecar on ``question``.
|
||||
|
||||
Calls the configured ChatClient with a constrained-generation
|
||||
prompt asking the model to pick ONE label from ``_SOFT_LABELS``
|
||||
plus a one-line rationale. Returns a ``SoftPreflightHint``
|
||||
that downstream surfaces on result dict / run-DAG / bench rows.
|
||||
|
||||
Failure modes (all return a stub hint, never raise):
|
||||
|
||||
- ``soft_preflight_enabled = False`` (default) — ``SOFT_DISABLED``
|
||||
- LLM raises an exception — ``SOFT_PARSE_FAIL``
|
||||
- LLM returns unparseable text — ``SOFT_PARSE_FAIL``
|
||||
- LLM returns a label not in the enum — ``SOFT_PARSE_FAIL``
|
||||
|
||||
Confidence scoring is intentionally NOT calibrated against
|
||||
empirical agreement-rate — see #000011 §9. Initial value is
|
||||
``0.5`` for any successful soft hint, ``0.0`` for stubs. A
|
||||
future calibration pass could derive confidence from
|
||||
agreement-with-hard-detectors on a labeled fixture.
|
||||
"""
|
||||
policy = policy or {}
|
||||
if not bool(policy.get("soft_preflight_enabled", False)):
|
||||
return _stub_hint(
|
||||
"SOFT_DISABLED",
|
||||
"sidecar opted-out via policy",
|
||||
raw_question=question,
|
||||
model_profile_id=model_id,
|
||||
)
|
||||
if not question or not question.strip():
|
||||
return _stub_hint(
|
||||
"SOFT_DISABLED",
|
||||
"empty question",
|
||||
raw_question=question,
|
||||
model_profile_id=model_id,
|
||||
)
|
||||
|
||||
t_start = time.monotonic()
|
||||
try:
|
||||
raw = chat_client.chat_completion(
|
||||
messages=[
|
||||
{"role": "system", "content": _SOFT_PREFLIGHT_SYSTEM_PROMPT},
|
||||
{"role": "user", "content": question},
|
||||
],
|
||||
model=model_id,
|
||||
temperature=0.0,
|
||||
max_tokens=128,
|
||||
top_p=1.0,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 — sidecar must fail-closed
|
||||
return _stub_hint(
|
||||
"SOFT_PARSE_FAIL",
|
||||
f"chat_client raised: {type(exc).__name__}",
|
||||
raw_question=question,
|
||||
model_profile_id=model_id,
|
||||
elapsed_ms=(time.monotonic() - t_start) * 1000.0,
|
||||
)
|
||||
elapsed_ms = (time.monotonic() - t_start) * 1000.0
|
||||
|
||||
label, rationale = _parse_soft_hint_response(raw)
|
||||
confidence = 0.0 if label == "SOFT_PARSE_FAIL" else 0.5
|
||||
|
||||
return SoftPreflightHint(
|
||||
raw_question=question,
|
||||
sidecar_version=SOFT_PREFLIGHT_VERSION,
|
||||
classifier_label=label,
|
||||
confidence=confidence,
|
||||
rationale=rationale,
|
||||
timestamp_ns=time.monotonic_ns(),
|
||||
model_profile_id=model_id,
|
||||
elapsed_ms=elapsed_ms,
|
||||
)
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -24,3 +24,89 @@ when did the current CEO of Twitter stop being CEO of Tesla?
|
|||
|
||||
# control: well-formed factoid that should NOT trigger any detector
|
||||
who painted the mona lisa?
|
||||
|
||||
# ── extended fixture (2026-05-04) — fitness-of-detector cases ──
|
||||
#
|
||||
# Below: edge / borderline / adversarial cases that probe the
|
||||
# fitness of each detector. Some should fire, some should NOT, some
|
||||
# should exhibit borderline behavior the operator can debate. The
|
||||
# fixture lives long-term as a regression suite for the four
|
||||
# metacog detectors. When the substrate evolves, this file is the
|
||||
# pin against which behavior is measured.
|
||||
|
||||
# ── temporal sensitivity edge cases ──
|
||||
|
||||
# implicit-current via role + shape (no `current` keyword)
|
||||
who is the President of the United States?
|
||||
|
||||
# rapid-turnover role with no temporal anchor at all (model-dependent)
|
||||
who is the CEO of Twitter?
|
||||
|
||||
# implicit-now via "this year" — should fire HIGH
|
||||
which team won this year's World Series?
|
||||
|
||||
# verbal future tense — model often hallucinates; not a detector
|
||||
# target today but worth observing
|
||||
who will be the next president after the 2028 election?
|
||||
|
||||
# negative control: timeless fact with role-shape — should NOT fire
|
||||
who was the first president of the United States?
|
||||
|
||||
# ── contradiction edge cases ──
|
||||
|
||||
# multi-pair contradiction (alive+dead AND never+always)
|
||||
which character was never alive but always dead?
|
||||
|
||||
# semantic contradiction without lexical pair — should NOT fire
|
||||
# (out of detector scope; documents conservative-by-design choice)
|
||||
which married bachelor lives in this house?
|
||||
|
||||
# triple-contradiction stress test
|
||||
how can a celibate spouse be both unmarried and divorced?
|
||||
|
||||
# negative control: not a contradiction, just compound predicates
|
||||
which characters appear in both Hamlet and Macbeth?
|
||||
|
||||
# ── false-premise edge cases ──
|
||||
|
||||
# "why did" relation pattern
|
||||
why did Albert Einstein invent the lightbulb?
|
||||
|
||||
# "how did" relation pattern with subtle false premise
|
||||
how did the Roman Empire conquer Australia?
|
||||
|
||||
# "when did" stop pattern with subtle false premise
|
||||
when did Shakespeare stop writing in French?
|
||||
|
||||
# false premise that the lite detector probably MISSES (no
|
||||
# presupposition trigger pattern) — annotates the detector ceiling
|
||||
why does NASA fake the moon landings?
|
||||
|
||||
# negative control: legitimate "when did X stop Y" with true premise
|
||||
when did the Roman Empire stop minting silver coins?
|
||||
|
||||
# ── out-of-corpus edge cases ──
|
||||
|
||||
# explicit "uploaded" reference
|
||||
in the document I uploaded yesterday, what does section 5 say?
|
||||
|
||||
# private notes / inbox
|
||||
according to my private notes, when did the meeting happen?
|
||||
|
||||
# borderline: claims about an existing public document the corpus
|
||||
# may or may not have (NOT out-of-corpus per current detector)
|
||||
what does the Wikipedia article on bipolar disorder say?
|
||||
|
||||
# ── multi-trigger combinatorics ──
|
||||
|
||||
# false-premise + out-of-corpus
|
||||
when did my uploaded contract stop being valid?
|
||||
|
||||
# stale-risk + contradiction
|
||||
which currently-alive historical figure is also dead?
|
||||
|
||||
# ── well-formed controls (should classify SINGULAR / well_formed) ──
|
||||
|
||||
what year did World War II end?
|
||||
who is bilbo baggins's nephew?
|
||||
list the planets of the solar system
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | open · design only (impl deferred) | 2026-05-04 | D1 (preserves) |
|
||||
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | closed · landed 2026-05-04 (zero-shot full impl) | 2026-05-04 | D1 (preserves) |
|
||||
| #000010 | Meta-Cognition Preflight Guard (M0 / MCTL) | closed · landed 2026-05-03 (Phases 1–4); DAG binding shipped via #000009 | 2026-05-03 | D1, D3 |
|
||||
| #000009 | Preflight run-DAG node binding (#000008+#000010) | closed · re-landed 2026-05-04 (§8 corrections: reject-path DAG, nested CTI clauses) | 2026-05-03 | D3, D4 |
|
||||
| #000008 | Broad-quantifier preflight guard | closed · landed in `4f2b5a6`; Phase 5 DAG binding split into #000009 | 2026-05-02 | — |
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Ticket #000011 — SOFT_PREFLIGHT_HINT model-assisted sidecar
|
||||
|
||||
**Status:** open · awaiting go/no-go (design only; implementation
|
||||
deferred)
|
||||
**Status:** closed · landed 2026-05-04 (zero-shot, full
|
||||
implementation per §13)
|
||||
**Opened:** 2026-05-04
|
||||
**Scope:** Add a model-assisted preflight sidecar that augments the
|
||||
deterministic detectors from #000010 with an LLM-driven shape
|
||||
|
|
@ -212,12 +212,77 @@ After the sidecar lands:
|
|||
|
||||
## 10. Status
|
||||
|
||||
Open · awaiting go/no-go. Design captured here; implementation
|
||||
deferred. Source-doc reference: ticket #000010 §18 +
|
||||
`~/Downloads/meta-cognition_for_hermes(1).txt` §18.
|
||||
Closed · landed 2026-05-04 in the same commit batch as the
|
||||
metacog-trigger fixture expansion + --show-preflight full clause
|
||||
render + latency profile. Source-doc reference: ticket #000010
|
||||
§18 + `~/Downloads/meta-cognition_for_hermes(1).txt` §18.
|
||||
|
||||
The deterministic substrate from #000010 + #000009 is the
|
||||
provable foundation. SOFT_PREFLIGHT_HINT is an enrichment
|
||||
layer that operates strictly within the substrate's "hard
|
||||
hash never enters the proof path" rule (cf. whitepaper
|
||||
§13 decision 8).
|
||||
|
||||
## 11. What landed
|
||||
|
||||
- **`aborist/qa/soft_preflight.py`** — `SoftPreflightHint`
|
||||
dataclass + `soft_preflight_question()` pure function.
|
||||
9 canonical labels (`SOFT_WELL_FORMED`,
|
||||
`SOFT_FALSE_PREMISE_SUSPECTED`, `SOFT_CONTRADICTION_SUSPECTED`,
|
||||
`SOFT_TIME_SENSITIVE`, `SOFT_SCOPE_AMBIGUOUS`,
|
||||
`SOFT_OUT_OF_CORPUS_LIKELY`, `SOFT_BROAD_QUANTIFIER`,
|
||||
`SOFT_MULTI_HOP_REASONING`, `SOFT_SUBJECTIVE`) plus stub
|
||||
states (`SOFT_DISABLED`, `SOFT_PARSE_FAIL`). Constrained-
|
||||
generation prompt asking the model to pick ONE label + a
|
||||
one-line rationale. Failure-closed across every parse path
|
||||
(raises → `SOFT_PARSE_FAIL`, drift → `SOFT_PARSE_FAIL`).
|
||||
`SOFT_PREFLIGHT_VERSION = "soft-preflight-v0.1"`.
|
||||
|
||||
- **`aborist/qa/query.py`** — wired into the post-classifier /
|
||||
pre-retrieval segment. One short LLM call (~200ms median),
|
||||
result surfaces as `soft_preflight_hint` on the result dict
|
||||
+ `soft_preflight_ms` in the timings dict.
|
||||
|
||||
- **`aborist/qa/runner.py` + `aborist/qa/query.py`** —
|
||||
`soft_preflight_enabled: False` policy default; cache row
|
||||
identity unchanged (NOT folded into `_VERIFIER_POLICY_FIELDS`
|
||||
per §4 — soft hints don't gate cache identity).
|
||||
|
||||
- **`aborist/cli.py`** — `--soft-preflight` flag on
|
||||
`aborist query`. Audit-line tail renders soft hints as
|
||||
`· soft: <label>` (e.g. `· soft: time sensitive`) — distinct
|
||||
from hard tails so operators see the signal separation at a
|
||||
glance. SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED
|
||||
suppress (no actionable signal).
|
||||
|
||||
- **`tests/test_soft_preflight.py`** — 25 new tests pinning:
|
||||
default-OFF behavior, parse-failure modes, label
|
||||
normalization (SOFT_ prefix enforced even when model drops
|
||||
it), all 8 actionable labels round-trip, fail-closed on
|
||||
client exceptions, dataclass JSON round-trip,
|
||||
rationale-length cap.
|
||||
|
||||
## 12. Live verification
|
||||
|
||||
End-to-end smoke test on `When did Mr. Burns become Homer's
|
||||
biological father?` with `--soft-preflight`:
|
||||
|
||||
```
|
||||
EVIDENCE-WARRANTED · via claim_lattice · false premise · soft: time sensitive 1/1 16.4s
|
||||
```
|
||||
|
||||
Both signals compose: hard `· false premise` from #000010
|
||||
deterministic detector, soft `· soft: time sensitive` from
|
||||
#000011 sidecar. The model classified a different shape than
|
||||
the hard detector — by design (soft hints are independent
|
||||
advisory signals, not redundant with the hard layer).
|
||||
|
||||
## 13. Bench plan (deferred)
|
||||
|
||||
Per §7 of this ticket — soft-only A/B against the metacog-
|
||||
trigger fixture (`bench/qa_questions_metacog_subset.txt` —
|
||||
expanded 2026-05-04 to 28 questions), measure agreement-rate
|
||||
between soft sidecar and hard detectors. Disagreement ≤ 10%
|
||||
signals the sidecar is reliable enough for default-on
|
||||
consideration. This bench is queued but not run in the same
|
||||
commit cycle as implementation.
|
||||
|
|
|
|||
232
tests/test_soft_preflight.py
Normal file
232
tests/test_soft_preflight.py
Normal file
|
|
@ -0,0 +1,232 @@
|
|||
"""Soft preflight sidecar (#000011) tests.
|
||||
|
||||
The sidecar produces ONLY soft hints labeled SOFT_*; it cannot
|
||||
create PREFLIGHT_OK or PREFLIGHT_BLOCKED. These tests pin:
|
||||
- default-OFF (sidecar disabled by default)
|
||||
- parse-failure modes (all return stub, never raise)
|
||||
- label normalization (SOFT_ prefix enforced)
|
||||
- rationale extraction
|
||||
- the dataclass schema
|
||||
|
||||
Uses a fake ChatClient (mock chat_completion) — no LLM round-trip.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from aborist.qa.soft_preflight import (
|
||||
SOFT_PREFLIGHT_VERSION,
|
||||
SoftPreflightHint,
|
||||
_normalize_label,
|
||||
_parse_soft_hint_response,
|
||||
soft_preflight_question,
|
||||
)
|
||||
|
||||
|
||||
class _FakeChatClient:
|
||||
"""Mock ChatClient that returns a configured response."""
|
||||
|
||||
def __init__(self, response: str):
|
||||
self.response = response
|
||||
self.call_count = 0
|
||||
|
||||
def chat_completion(self, *args, **kwargs) -> str:
|
||||
self.call_count += 1
|
||||
return self.response
|
||||
|
||||
|
||||
class _RaisingChatClient:
|
||||
"""Mock that raises on call — tests fail-closed behavior."""
|
||||
|
||||
def chat_completion(self, *args, **kwargs) -> str:
|
||||
raise RuntimeError("simulated network failure")
|
||||
|
||||
|
||||
# ----------------------------------------------------------- defaults
|
||||
|
||||
def test_sidecar_default_off_returns_stub_hint():
|
||||
"""Without soft_preflight_enabled=True in policy, sidecar
|
||||
returns SOFT_DISABLED stub. No LLM call happens."""
|
||||
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
|
||||
hint = soft_preflight_question(
|
||||
"what is the capital of france?",
|
||||
chat_client=client,
|
||||
model_id="test-model",
|
||||
policy={}, # no soft_preflight_enabled
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_DISABLED"
|
||||
assert hint.confidence == 0.0
|
||||
assert client.call_count == 0 # no LLM call
|
||||
|
||||
|
||||
def test_sidecar_explicit_off_returns_stub():
|
||||
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
|
||||
hint = soft_preflight_question(
|
||||
"anything", chat_client=client, model_id="test",
|
||||
policy={"soft_preflight_enabled": False},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_DISABLED"
|
||||
assert client.call_count == 0
|
||||
|
||||
|
||||
def test_empty_question_returns_stub_even_when_enabled():
|
||||
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
|
||||
hint = soft_preflight_question(
|
||||
"", chat_client=client, model_id="test",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_DISABLED"
|
||||
assert client.call_count == 0
|
||||
|
||||
|
||||
# ----------------------------------------------------------- happy path
|
||||
|
||||
def test_sidecar_returns_soft_hint_when_enabled():
|
||||
client = _FakeChatClient(
|
||||
"LABEL: SOFT_FALSE_PREMISE_SUSPECTED\n"
|
||||
"RATIONALE: question presupposes Mr. Burns is Homer's father"
|
||||
)
|
||||
hint = soft_preflight_question(
|
||||
"When did Mr. Burns become Homer's biological father?",
|
||||
chat_client=client,
|
||||
model_id="hermes-test",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_FALSE_PREMISE_SUSPECTED"
|
||||
assert hint.confidence == 0.5
|
||||
assert "Mr. Burns" in hint.rationale
|
||||
assert hint.model_profile_id == "hermes-test"
|
||||
assert hint.sidecar_version == SOFT_PREFLIGHT_VERSION
|
||||
assert client.call_count == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("label", [
|
||||
"SOFT_WELL_FORMED",
|
||||
"SOFT_CONTRADICTION_SUSPECTED",
|
||||
"SOFT_TIME_SENSITIVE",
|
||||
"SOFT_OUT_OF_CORPUS_LIKELY",
|
||||
"SOFT_BROAD_QUANTIFIER",
|
||||
"SOFT_MULTI_HOP_REASONING",
|
||||
"SOFT_SUBJECTIVE",
|
||||
"SOFT_SCOPE_AMBIGUOUS",
|
||||
])
|
||||
def test_each_canonical_label_passes_through(label):
|
||||
client = _FakeChatClient(f"LABEL: {label}\nRATIONALE: test")
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == label
|
||||
|
||||
|
||||
# ----------------------------------------------------------- failure modes
|
||||
|
||||
def test_chat_client_exception_returns_parse_fail():
|
||||
"""Sidecar fails-closed on chat_completion exceptions."""
|
||||
client = _RaisingChatClient()
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_PARSE_FAIL"
|
||||
assert hint.confidence == 0.0
|
||||
assert "RuntimeError" in hint.rationale
|
||||
|
||||
|
||||
def test_unparseable_response_returns_parse_fail():
|
||||
client = _FakeChatClient("garbage output no label here")
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_PARSE_FAIL"
|
||||
assert hint.confidence == 0.0
|
||||
|
||||
|
||||
def test_model_drift_label_outside_enum_returns_parse_fail():
|
||||
"""Model returns a label not in the enum → fail-closed."""
|
||||
client = _FakeChatClient(
|
||||
"LABEL: SOFT_INVENTED_NEW_LABEL\nRATIONALE: model drift"
|
||||
)
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_PARSE_FAIL"
|
||||
|
||||
|
||||
def test_label_without_soft_prefix_gets_normalized():
|
||||
"""Model drops the SOFT_ prefix → normalizer adds it back."""
|
||||
client = _FakeChatClient("LABEL: WELL_FORMED\nRATIONALE: ok")
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_WELL_FORMED"
|
||||
|
||||
|
||||
def test_label_with_trailing_punctuation_normalized():
|
||||
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED.\nRATIONALE: ok")
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
assert hint.classifier_label == "SOFT_WELL_FORMED"
|
||||
|
||||
|
||||
# ----------------------------------------------------------- pure helpers
|
||||
|
||||
def test_normalize_label_handles_whitespace():
|
||||
assert _normalize_label(" SOFT_WELL_FORMED ") == "SOFT_WELL_FORMED"
|
||||
|
||||
|
||||
def test_normalize_label_uppercases():
|
||||
assert _normalize_label("soft_well_formed") == "SOFT_WELL_FORMED"
|
||||
|
||||
|
||||
def test_normalize_label_returns_parse_fail_for_garbage():
|
||||
assert _normalize_label("just garbage") == "SOFT_PARSE_FAIL"
|
||||
|
||||
|
||||
def test_parse_soft_hint_handles_mixed_case_keys():
|
||||
label, rationale = _parse_soft_hint_response(
|
||||
"label: SOFT_WELL_FORMED\nrationale: ok"
|
||||
)
|
||||
assert label == "SOFT_WELL_FORMED"
|
||||
assert rationale == "ok"
|
||||
|
||||
|
||||
def test_parse_soft_hint_handles_missing_rationale_line():
|
||||
"""When the model only writes LABEL: but no RATIONALE: line."""
|
||||
label, rationale = _parse_soft_hint_response("LABEL: SOFT_WELL_FORMED")
|
||||
assert label == "SOFT_WELL_FORMED"
|
||||
assert rationale # something, even if "(no rationale)"
|
||||
|
||||
|
||||
def test_parse_soft_hint_caps_rationale_length():
|
||||
"""Long rationales get capped to keep payload bounded."""
|
||||
long_rationale = "x" * 500
|
||||
label, rationale = _parse_soft_hint_response(
|
||||
f"LABEL: SOFT_WELL_FORMED\nRATIONALE: {long_rationale}"
|
||||
)
|
||||
assert len(rationale) <= 200
|
||||
|
||||
|
||||
# ----------------------------------------------------------- dataclass schema
|
||||
|
||||
def test_hint_to_dict_is_json_serializable():
|
||||
"""Bench rows / run-DAG persist soft hints as JSON; the
|
||||
dataclass round-trips cleanly."""
|
||||
import json
|
||||
client = _FakeChatClient("LABEL: SOFT_WELL_FORMED\nRATIONALE: ok")
|
||||
hint = soft_preflight_question(
|
||||
"test", chat_client=client, model_id="m",
|
||||
policy={"soft_preflight_enabled": True},
|
||||
)
|
||||
d = hint.to_dict()
|
||||
json.dumps(d, ensure_ascii=False) # raises if non-serializable
|
||||
|
||||
|
||||
def test_version_pinned():
|
||||
assert SOFT_PREFLIGHT_VERSION == "soft-preflight-v0.1"
|
||||
Loading…
Add table
Add a link
Reference in a new issue