arborist/docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md

Ticket #000011 — SOFT_PREFLIGHT_HINT model-assisted sidecar

Status: closed · landed 2026-05-04 (zero-shot, full implementation per §13) Opened: 2026-05-04 Scope: Add a model-assisted preflight sidecar that augments the deterministic detectors from #000010 with an LLM-driven shape classifier. Strict guardrail per #000010 §18 / source doc: the sidecar produces ONLY soft hints labeled SOFT_PREFLIGHT_HINT; it cannot create PREFLIGHT_OK or PREFLIGHT_BLOCKED without deterministic support. Hard rule (D1) preserved. Audience: fox + future blackops shifts. Hard constraint: Same as #000010 §1: no schema_version, canonicalization_version, or chunking_version bumps; pure additive policy folded into governance_policy_hash. The sidecar NEVER alters the verifier's hard path; sidecar output lives on result-dict + bench rows + the run-DAG preflight payload but is clearly tagged SOFT_* so audit replay can distinguish hard from soft signal.

---

1. Premise

The deterministic detectors from #000010 (temporal, contradiction, false-premise-lite, out-of-corpus) are conservative by design — lexical pattern matching catches obvious shapes & misses subtle ones. A model-assisted preflight could catch:

• Subtle false premises that don't match the lite presupposition patterns (e.g. "Why does the Federal Reserve refuse to audit itself?" — false premise hidden in "refuse").
• Multi-hop contradictions invisible to lexical pair matching.
• Style-of-reasoning needs (compare/contrast, summarize, list) that don't surface in the quantifier classifier.
• Implicit time-anchors ("now that the AI bubble has popped") the lexical temporal detector misses.

The architectural rule from #000010 §18 forbids putting an LLM in the hard preflight path. But a sidecar — soft signal, never gates admissibility, can be wrong without breaking guarantees — is admissible under the same separation-of-concerns the substrate already maintains for hard vs soft hashes.

2. What the sidecar produces

A separate SoftPreflightHint dataclass distinct from QuestionState:

@dataclass(frozen=True)
class SoftPreflightHint:
    raw_question: str
    sidecar_version: str  # e.g. "soft-preflight-v0.1"
    classifier_label: str  # e.g. SOFT_FALSE_PREMISE_SUSPECTED
    confidence: float  # 0.0-1.0, never used for hard decisions
    rationale: str  # one-line explanation from the model
    timestamp: int  # monotonic for replay debugging
    model_profile_id: str  # which model produced the hint

The hint always carries the SOFT_ prefix on its classifier_label so it can never be mistaken for a deterministic verdict by:

• The audit-line tail renderer in arborist/cli.py:_render_warrant_tail
• The verifier (which doesn't read sidecar fields anyway)
• Bench aggregations
• Operators inspecting bench JSONL or run-DAG blobs

3. Where it lives

arborist/qa/soft_preflight.py    — sidecar implementation
                                  (new module)

Mirrors arborist/qa/inspect.py (the existing read-only sidecar diagnostic for span classification) — same architectural pattern: soft signal, never enters proof path, never written to providence cache, never bumps governance hash on use.

Wired into query() and runner.ask() as an optional pre-LLM call gated on soft_preflight_enabled (default False). Adds ~200ms of latency per call (one extra LLM round-trip on the local Hermes endpoint).

4. Cache binding

Soft sidecar hints DO NOT enter governance_policy_hash. Two cache rows with the same hard preflight contract but different soft hints SHARE a cache_key — sidecar disagreement is not a cache split.

Soft hints DO surface on the run-DAG preflight payload's question_state.soft_preflight_hint field (new optional sub-field). The hint is hash-bound for audit replay, but two runs with different sidecar outputs can still hit the same cache. This is intentional: audit can see what soft signal the sidecar emitted; cache identity remains driven by the hard policy.

5. Implementation sketch

# arborist/qa/soft_preflight.py

def soft_preflight_question(
    question: str,
    *,
    chat_client: ChatClient,
    model_id: str,
    policy: dict,
) -> SoftPreflightHint:
    """Call the LLM to produce a soft preflight hint. Pure
    sidecar: caller MUST treat the output as advisory only.
    Never enters the verifier, never gates admissibility."""
    if not policy.get("soft_preflight_enabled", False):
        return _stub_hint("SOFT_DISABLED", "sidecar off")

    # Short, focused system prompt asking the model to classify
    # the question shape from a fixed enum:
    #   SOFT_WELL_FORMED
    #   SOFT_FALSE_PREMISE_SUSPECTED
    #   SOFT_CONTRADICTION_SUSPECTED
    #   SOFT_TIME_SENSITIVE
    #   SOFT_SCOPE_AMBIGUOUS
    #   SOFT_OUT_OF_CORPUS_LIKELY
    # ...
    # Return one of the enum tokens + a one-line rationale.
    # Constrained generation (max_tokens=128, temperature=0.0
    # for deterministic-ish output, top_p=1.0).
    raw = chat_client.chat_completion(
        messages=[
            {"role": "system", "content": SOFT_PREFLIGHT_SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
        model=model_id,
        temperature=0.0,
        max_tokens=128,
    )
    label, rationale = _parse_soft_hint(raw)
    return SoftPreflightHint(
        raw_question=question,
        sidecar_version="soft-preflight-v0.1",
        classifier_label=f"SOFT_{label.upper()}",
        confidence=0.5,  # soft signals never claim hard confidence
        rationale=rationale,
        timestamp=int(time.monotonic_ns()),
        model_profile_id=model_id,
    )

6. Integration points

• arborist/qa/query.py — call soft_preflight_question() after preflight_question() (the hard call) but before the main LLM call. Soft hint surfaces on result dict as soft_preflight_hint; merges into the run-DAG preflight stage payload as a sub-field of question_state.
• arborist/qa/runner.py — same wiring.
• arborist/cli.py — new --soft-preflight flag (per-call override; default off matches policy default).
• bench/qa_sweep.py — bench rows pick up soft_preflight_label
  • soft_preflight_confidence as bounded-projection fields.
• arborist/cli.py:_render_warrant_tail — soft hints render as · soft: <label> on the audit-line (clearly differentiated from the hard · false premise etc. tails). Operator can tell at a glance whether a tail is hard or soft.

7. Bench plan

After the sidecar lands:

1. Soft-only A/B on the metacog-trigger fixture (#000010 §13.1 + the bench/qa_questions_metacog_subset.txt from the 2026-05-04 work). Measure: does the soft sidecar agree with the hard detectors on the cases where the hard detectors fire? Disagreement rate ≤ 10% signals the sidecar is reliable enough to surface to operators.
2. Soft-on subtle-cases A/B with hand-crafted fixtures of subtle false premises / multi-hop contradictions that DON'T trigger the hard detectors. Measure: does the sidecar add coverage on shapes the hard detectors miss?
3. Latency cost — the extra ~200ms per call multiplied by bench cycle counts. Bench cycle latency budget is currently ~17-35s/call (Hermes); adding 200ms is ~1% overhead. Measure to confirm.

8. What this ticket does NOT do

• Does NOT enter the verifier proof path. D1 preserved.
• Does NOT bump schema_version / canonicalization_version / chunking_version. Pure additive.
• Does NOT change cache identity. Soft hints don't fold into governance_policy_hash.
• Does NOT replace the hard detectors. Hard detectors remain the source of truth for PREFLIGHT_OK / PREFLIGHT_BLOCKED.
• Does NOT promote a soft hint to hard without deterministic support. The sidecar can claim SOFT_FALSE_PREMISE_SUSPECTED but the hard detector must agree before false_premise_suspected enters logical_statuses.

9. Open questions

• Use a separate cheap model for the sidecar (e.g. Hermes-3 itself, or a smaller model) vs the same model that produces the answer? Bench data needed.
• One-call vs streaming-batch? At 200ms/call across a 75-question bench (n=3), one-call adds ~22s; batched (e.g. 5 questions at once) could compress this significantly but complicates the per-row provenance binding.
• Confidence calibration — initial proposal is 0.5 literal for any soft hint. Future refinement might calibrate via empirical agreement-rate with the hard detectors.

10. Status

Closed · landed 2026-05-04 in the same commit batch as the metacog-trigger fixture expansion + --show-preflight full clause render + latency profile. Source-doc reference: ticket #000010 §18 + ~/Downloads/meta-cognition_for_hermes(1).txt §18.

The deterministic substrate from #000010 + #000009 is the provable foundation. SOFT_PREFLIGHT_HINT is an enrichment layer that operates strictly within the substrate's "hard hash never enters the proof path" rule (cf. whitepaper §13 decision 8).

11. What landed

• arborist/qa/soft_preflight.py — SoftPreflightHint dataclass + soft_preflight_question() pure function. 9 canonical labels (SOFT_WELL_FORMED, SOFT_FALSE_PREMISE_SUSPECTED, SOFT_CONTRADICTION_SUSPECTED, SOFT_TIME_SENSITIVE, SOFT_SCOPE_AMBIGUOUS, SOFT_OUT_OF_CORPUS_LIKELY, SOFT_BROAD_QUANTIFIER, SOFT_MULTI_HOP_REASONING, SOFT_SUBJECTIVE) plus stub states (SOFT_DISABLED, SOFT_PARSE_FAIL). Constrained- generation prompt asking the model to pick ONE label + a one-line rationale. Failure-closed across every parse path (raises → SOFT_PARSE_FAIL, drift → SOFT_PARSE_FAIL). SOFT_PREFLIGHT_VERSION = "soft-preflight-v0.1".

• arborist/qa/query.py — wired into the post-classifier / pre-retrieval segment. One short LLM call (~200ms median), result surfaces as soft_preflight_hint on the result dict

  • soft_preflight_ms in the timings dict.

• arborist/qa/runner.py + arborist/qa/query.py — soft_preflight_enabled: False policy default; cache row identity unchanged (NOT folded into _VERIFIER_POLICY_FIELDS per §4 — soft hints don't gate cache identity).

• arborist/cli.py — --soft-preflight flag on arborist query. Audit-line tail renders soft hints as · soft: <label> (e.g. · soft: time sensitive) — distinct from hard tails so operators see the signal separation at a glance. SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED suppress (no actionable signal).

• tests/test_soft_preflight.py — 25 new tests pinning: default-OFF behavior, parse-failure modes, label normalization (SOFT_ prefix enforced even when model drops it), all 8 actionable labels round-trip, fail-closed on client exceptions, dataclass JSON round-trip, rationale-length cap.

12. Live verification

End-to-end smoke test on When did Mr. Burns become Homer's biological father? with --soft-preflight:

EVIDENCE-WARRANTED · via claim_lattice · false premise · soft: time sensitive  1/1  16.4s

Both signals compose: hard · false premise from #000010 deterministic detector, soft · soft: time sensitive from #000011 sidecar. The model classified a different shape than the hard detector — by design (soft hints are independent advisory signals, not redundant with the hard layer).

13. Bench plan (deferred)

Per §7 of this ticket — soft-only A/B against the metacog- trigger fixture (bench/qa_questions_metacog_subset.txt — expanded 2026-05-04 to 28 questions), measure agreement-rate between soft sidecar and hard detectors. Disagreement ≤ 10% signals the sidecar is reliable enough for default-on consideration. This bench is queued but not run in the same commit cycle as implementation.