qa: G0 claim-lattice-pointer answer mode — CTI quote-by-pointer

New policy["answer_mode"] = "claim_lattice_pointer" (default "quote").
The runtime builds an evidence map with two-layer ids — pointer-id
(E1, E2, ...) shown to the model, sha256-derived evidence-id used by
the cache & run-DAG — and the model emits pointer-line prose
("Claim. [E12]") referencing them. Renderer interpolates literal
source spans at display time. Synthetic-elision-by-construction-
impossible: the model never types the quote string.

Pieces:

- aborist/qa/evidence.py (new): EvidenceObject + spotlight excerpt
  (claim-token-centered window into the cited span, falls back to
  leading window when no token matches).

- aborist/qa/parse_claims.py (new): pointer-line parser walks
  lines, pulls [E\d+] / [E\d+,E\d+,...] tags, returns ParsedClaim
  with PARSED / NO_EVIDENCE_POINTER status. Strict regex refuses
  fuzzy alternatives so honest UNGROUNDED beats lax acceptance.

- aborist/qa/verify.py: verify_claim_lattice. Hard checks only —
  parser succeeded, evidence_id resolves, source_role allowed, no
  manual quotes (any " char violates), claim text non-empty.
  Returns evidence_id_pairs (content-addressed, run-stable) for the
  run-DAG. Soft signals (entailment, completeness, predicate
  compatibility) stay sidecar.

- aborist/qa/dag.py: 9-stage CTI shape when evidence_map_root is
  supplied — question / retrieval / evidence_map / prompt /
  raw_answer / parsed_claim_lattice / verify / render /
  final_label. Quote mode keeps the original 7-stage shape so
  pre-G0 run_dag_root values stay valid.

- aborist/qa/query.py: per-chunk evidence-map build with role-
  weighted budget AND query-relevance ordering. Within each source,
  chunks are ranked by (distinct_query_tokens_present,
  total_mentions, doc_order_asc) so the chunk most likely to
  textually support the question gets the lowest pointer id. Without
  this re-rank Hermes-3-8B lazy-anchored on doc-order-first chunks
  regardless of relevance.

- aborist/qa/runner.py: same answer_mode branch for the single-doc
  ask() path. No iterative repair in pointer mode (one-shot
  benchmark discipline).

- aborist/store.py: verifier_method CHECK extended with
  'claim_lattice'. New _rebuild_providence_cache_claim_lattice
  migration preserves run_dag_root / run_dag_blob across the
  rebuild — older rebuilds dropped them.

- aborist/cli.py: --answer-mode {quote,claim_lattice_pointer} on
  query and ask.

- Makefile: ANSWER_MODE knob; defaults to claim_lattice_pointer for
  `make query` so the testing harness exercises the new path.
  Library DEFAULT_POLICY / DEFAULT_QUERY_POLICY stay "quote" so
  Python callers and unit tests aren't surprised.

Prompt: one-shot worked example (Apple founders) plus strict
no-quotes / no-JSON / no-markdown / plain-prose-only rules. Without
the worked example Hermes-3-8B drops the bracket protocol on roughly
half of runs; with it, JP dinosaurs benchmark went 0/13 -> 10/11 ->
16/17 verified across the iterations that hardened the pipeline.

Tests: +48 covering parser, two-layer ids, verifier failure modes
(UNKNOWN_EVIDENCE_ID / SOURCE_ROLE_BLOCKED / MANUAL_QUOTE_VIOLATION
/ NO_EVIDENCE_POINTER / SCHEMA_INVALID), spotlight rendering with
buried-term fixture, per-chunk evidence map, query-relevance
ordering, schema migration round-trip, 9-stage DAG shape divergence.
Total 438 passed, all 7 production shards report 0 chain breaks.

Known limitation: chunk boundaries can cut wikitext mid-template, so
mwparserfromhell-backed to_base() leaves orphan </ref> tags and
leading list markers in the visible spotlight excerpts. Verifier and
CTI architecture are unaffected; the leak is cosmetic. Proper fixes
are template-aware chunking (chunker bump invalidates prior records)
or an orphan-marker post-strip in aborist/wikitext.py — both out of
G0 scope.
This commit is contained in:
russell@unturf.com 2026-04-29 21:19:19 -04:00
parent 304dadfb71
commit 2337b7705c
No known key found for this signature in database
10 changed files with 2426 additions and 216 deletions

View file

@ -155,17 +155,23 @@ burn-kindergarten: bootstrap ## bust providence rows < SECONDS old [SECONDS=3600
# Multi-source RAG query against the shard cluster.
# Usage: make query Q="What is anarcho-capitalism?"
QUERY_TOP_K ?= 8
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N]; sources picked across shards
# G0 / CTI — pointer-mode is the testing default. Library DEFAULT_POLICY
# stays "quote" so Python callers aren't surprised; the Makefile
# harness ships pointer-mode-on so `make query` exercises the new path
# end-to-end. Override with ANSWER_MODE=quote to test the legacy path,
# ANSWER_MODE= (empty) to use whatever DEFAULT_QUERY_POLICY says.
ANSWER_MODE ?= claim_lattice_pointer
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N ANSWER_MODE=quote|claim_lattice_pointer]; pointer mode by default
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N]"; exit 2; \
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N ANSWER_MODE=quote|claim_lattice_pointer]"; exit 2; \
fi
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) "$(Q)"
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) "$(Q)"
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1]
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1 ANSWER_MODE=...]
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
echo "usage: make query-dry Q=\"your question\" [JSON=1 BURN=1]"; exit 2; \
echo "usage: make query-dry Q=\"your question\" [JSON=1 BURN=1 ANSWER_MODE=quote|claim_lattice_pointer]"; exit 2; \
fi
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) --dry-run $(if $(JSON),--json,) $(if $(BURN),--burn,) "$(Q)"
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) --dry-run $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) "$(Q)"
verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
$(ABORIST) --shards-dir $(SHARDS_DIR) verify -n $(VERIFY_N)