engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa: 'aborist inspect' — sidecar diagnose for unverified spans

Browse source 
Per fox: the verifier reports n/N verified but doesn't tell you WHY a
span didn't ground. Some are paraphrase, some are model-added
trailing artifacts (e.g. `(Source: https://...)` citations the model
appended to verbatim prose), some are full inventions. Operator
needs to see which is which to triage.

Read-only sidecar — no audit events, no providence_cache mutations,
no v9.8 field changes. Per the verifier-no-diagnostics memory rule:
hard chain stays binary; soft signals live in sidecar verbs that
never feed back.

Five diagnosis labels:

  verbatim_in_base       span IS in base context — verifier or
                         canonicalization bug (worth flagging loudly).
  verbatim_in_raw_only   raw wikitext match but base form differs —
                         wikitext-strip edge case.
  trailing_artifact      a >=60-char prefix matches; the tail (often a
                         model-added citation) doesn't. Surfaces the
                         tail explicitly.
  paraphrase             >=85% query tokens (>4 chars) present in
                         base context but not in this sequence.
                         Model rewrote source content.
  partial_paraphrase     40-85% coverage — mixed sourced/emergent.
  no_overlap             <40% coverage — likely full invention.

Verified live against the real Pikachu cache record (5 sources,
71KB raw -> 34KB base): two spans flagged paraphrase (token_coverage
1.0 for both, just rewritten sequence), one flagged trailing_artifact
(100-char prefix matches; tail is `(Source: https://...)` citation).
That's the actual reason HYBRID — model paraphrased + appended
citations that aren't in the corpus.

Surface:
  aborist inspect --cache-key <hex> [--qa-db ...] [--json]
  make inspect KEY=<hex> [JSON=1]

Tests: 9 (6 classifier on synthetic contexts pinning each label,
3 end-to-end with seeded fixture record + verification that inspect
writes nothing). 318 passed, 1 skipped overall.
This commit is contained in:
russell@unturf.com 2026-04-29 10:29:06 -04:00
parent 0f527b82d7
commit d3168fa236
No known key found for this signature in database
4 changed files with 613 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

8
Makefile
View file

@ -31,8 +31,8 @@ SEARCH_Q ?= computer
ingest-grok ingest-grok-media \
ingest-self ingest-git ingest-hg \
verify search stats test docs chain-check chain-check-shards \
falsify burn bootstrap-crawler test-crawler crawl-ingest recrawl-check \
clean clean-db clean-data help
falsify burn inspect bootstrap-crawler test-crawler crawl-ingest \
recrawl-check clean clean-db clean-data help
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
@ -120,6 +120,10 @@ ACTIVITY_LIMIT ?= 10
activity: bootstrap ## recent Q&A + freshly cached docs (agent timeline)
$(ABORIST) --shards-dir $(SHARDS_DIR) activity --limit $(ACTIVITY_LIMIT)
inspect: bootstrap ## sidecar diagnose unverified spans for a cache_key: make inspect KEY=hex [JSON=1]
@if [ -z "$(KEY)" ]; then echo "usage: make inspect KEY=<cache_key> [JSON=1]" >&2; exit 2; fi
$(ABORIST) --shards-dir $(SHARDS_DIR) inspect --cache-key $(KEY) $(if $(JSON),--json,)
falsify: bootstrap ## mark a cached answer wrong: make falsify KEY=hex REASON='why'
@if [ -z "$(KEY)" ]; then echo "usage: make falsify KEY=<cache_key> REASON='why'" >&2; exit 2; fi
$(ABORIST) --shards-dir $(SHARDS_DIR) providence --falsify $(KEY) --reason "$(REASON)"