Two enhancements continuing the toy-Hermes design pass:
#1 Chain-segment failure localization
aborist/qa/dag.py: `localize_failure(audit_mode, n_sources, n_quotes,
n_verified)` maps a non-STRICT verdict to the pipeline stage that
introduced the failure:
retrieval no admitted sources (gate over-rejected, or corpus
genuinely lacks the topic) → ingest more / relax breadth
context sources retrieved but no quotes extracted (per-source
cap dropped relevant content, or model declined to cite)
→ raise cap / tighten prompt
answer quotes extracted but didn't all verify (model fabricated,
paraphrased inside quotes, appended citations) → mechanical
+ re-prompt repair targets exactly this case
`failure_stage` lands on the run_dag's verify node payload AND on the
result dict so an operator can read the reason at a glance — `failure_stage='answer'`
means stop tuning the verifier & fix the model behavior. Debugging
becomes typed instead of vague.
#2 Re-prompt repair (second tier of the hybrid loop)
aborist/qa/repair.py: `reprompt_repair(...)` builds a feedback message
naming the failed quotes & asks the model to rewrite using only
verbatim citations. Hard rule: only fires when
`policy["repair_max_reprompts"] > 0` (default 0); caller enforces the
cap by looping at most that many times.
aborist/qa/query.py + aborist/qa/runner.py: after mechanical repair,
if the answer is still HYBRID/UNGROUNDED with unverified quotes,
loop up to `repair_max_reprompts` times. Each iteration: build
feedback (assistant turn with current answer + user turn with failed
spans), call LLM, verify. Accept the new answer if `n_verified`
strictly improved; otherwise break (the model's not converging,
don't waste cycles).
The mechanical + re-prompt combination handles the cases each tier
declines individually:
mechanical alone: synthetic_elision split, trailing_artifact trim,
no_overlap remove
+ re-prompt: paraphrase, partial_paraphrase, interior_elision
needing semantic judgment, fabrications the
model can recognize when shown its own quote
`repair_max_reprompts` lives in DEFAULT_QUERY_POLICY +
DEFAULT_POLICY so it folds into governance_policy_hash. Default 0
preserves single-shot semantics for callers that don't opt in. Each
re-prompt iteration adds a `{action: reprompt_rewrite, diagnosis:
model_feedback_loop}` entry to repair_changes; audit chain captures
the full transition through the existing providence_repair event.
Tests:
- dag: localize_failure across all four cases (STRICT, retrieval,
context, answer); failure_stage embedded in run_dag verify node.
- repair: stub client with sequenced answers (failing first, clean
on re-prompt) — assert two LLM calls, STRICT verdict, reprompt_rewrite
in the change log.
484 tests pass (dag +5, repair +1). The pre-existing test_burn flake
under full-suite ordering remains; passes in isolation.
The system was observational: verifier classified, sidecar diagnosed,
repair plans were emitted — but no loop ever closed. This adds the
hybrid repair stage from the toy-Hermes design pass: mechanical first
(deterministic string substitution from sidecar suggestions), behind
a `policy["repair_enabled"]` flag (off by default). Re-prompt fallback
is TODO.
aborist/qa/repair.py: `mechanical_repair(answer, unverified_quotes,
context)` walks each unverified quote through the sidecar classifier &
applies its `repair` action by string sub:
synthetic_elision_inside_quote (both halves verbatim)
`"prefix [...] suffix"` → `"prefix" ... "suffix"`
Two verbatim spans the verifier can independently check; the
model's [...] ellipsis-marker becomes prose between them.
trailing_artifact
`"prose. (Source: ...)"` → `"prose."`
Verbatim prefix kept; model-appended tail dropped.
no_overlap
Drop the line containing the bad quote entirely.
Skips include_aside_for_verbatim (needs precise source-span extraction;
defer to re-prompt path), paraphrase / partial_paraphrase (need prose
rewriting). Idempotent.
aborist/qa/query.py + aborist/qa/runner.py: optional pass after first
verify. When `repair_enabled=True` AND `audit_mode != "STRICT"` AND
unverified quotes exist:
1. Run mechanical_repair on the answer text.
2. If repair produced any changes, re-verify the repaired text.
3. If post-repair verdict isn't worse (n_verified didn't decrease),
accept the repair: persist the REPAIRED answer text instead of
the model's original. Cache_key inputs unchanged.
4. Audit chain gets one `providence_repair` event with the change
log + pre/post verdict so the original→repaired transition is
reconstructable.
Result dict gains `repair_changes` (list of change records) and
`pre_repair_audit_mode` (what the original was classified as).
`policy["repair_enabled"]` enters governance_policy_hash so on/off
agents share no cache silos.
Tests:
- mechanical_repair on each diagnosis (synthetic_elision, trailing_artifact,
no_overlap), idempotence on clean text.
- query() integration: repair_enabled=False (default) leaves answer
text unchanged; repair_enabled=True promotes a HYBRID/quote
synthetic_elision case to STRICT/quote, persists the repaired text,
emits the providence_repair audit event.
479 tests pass (+6 repair).
