docs: remove 7 non-load-bearing docs (30% reduction)
Delete redundant, superseded, or design-log docs: - qa-modes-bench-2026-04-30: prior snapshot (rolling journal is current) - verifier-semantic-gap-design: unimplemented future proposal - bench-emergent-design: stress-test design (script self-documents) - modules.md: API reference (code + docstrings are source of truth) - self-reference-design: v1 shipped, v2 scoped to future; history in git - concept-relations-design: live system documented in code - mesh-deploy: runbook for off-by-default system; mesh wire future work Reduces docs/ from 23 files to 16, keeping north-star (seven-point-program), architecture (cti, mesh.md), and operational docs (benchmarks, qa-modes-bench, bench-maxing). All deleted docs recoverable from git history.
This commit is contained in:
parent
17d637e99f
commit
bb6a89c7d4
7 changed files with 0 additions and 2573 deletions
|
|
@ -1,205 +0,0 @@
|
|||
# Bench-emergent: random-word triangulation stress test
|
||||
|
||||
**Status:** landed — `scripts/bench_emergent.py`, 2026-05-02
|
||||
**Cadence:** blue-moon — runs on demand, NOT every commit
|
||||
**Audience:** anyone hunting for failure modes the curated
|
||||
`bench/qa_questions.txt` doesn't surface
|
||||
|
||||
## Why
|
||||
|
||||
`bench/qa_questions.txt` is 71 hand-curated questions that pin
|
||||
known-good answers + known-failure shapes. It tells us whether the
|
||||
substrate handles the failure shapes we already named. It can't
|
||||
tell us what we *haven't* named.
|
||||
|
||||
The curated bench is a scoreboard. This is a sonar — random pings
|
||||
into the combinatoric space of "what happens when somebody asks
|
||||
something we never thought to ask." The 2010-11 Wikipedia corpus
|
||||
is fixed; the question space is infinite; emergent surfacing is
|
||||
how we find what we don't know.
|
||||
|
||||
## Loop shape
|
||||
|
||||
```
|
||||
/usr/share/dict/words
|
||||
│ random.sample(3)
|
||||
▼
|
||||
╔════════════════════╗
|
||||
║ generator: Hermes ║ temp=0.8
|
||||
║ "weave 3 words → ║ creative paragraph
|
||||
║ question paragraph║ with question framing
|
||||
╚════════════════════╝
|
||||
│
|
||||
▼
|
||||
╔════════════════════╗
|
||||
║ student: aborist ║ full retrieval pipeline,
|
||||
║ query() against ║ claim_lattice mode,
|
||||
║ live shards ║ burn=True (always fresh)
|
||||
╚════════════════════╝
|
||||
│
|
||||
▼
|
||||
┌────────────────────┐
|
||||
│ append to │ one JSONL line per cycle
|
||||
│ bench/emergent_ │ fields: words, question,
|
||||
│ log.jsonl │ answer, audit_mode, sources,
|
||||
│ │ timings, teacher: null
|
||||
└────────────────────┘
|
||||
│
|
||||
│ (teacher review — separate, manual)
|
||||
▼
|
||||
╔════════════════════╗
|
||||
║ teacher: Opus ║ read pending entries,
|
||||
║ (Claude 4.7, this ║ judge match / novelty /
|
||||
║ conversation) ║ bench-max signal,
|
||||
║ ║ append `teacher: {...}`
|
||||
╚════════════════════╝
|
||||
```
|
||||
|
||||
## Why "blue moon"
|
||||
|
||||
- Wall-clock cost: ~20s per cycle (Hermes generator @ 5-10s +
|
||||
aborist student @ 5-15s). N=10 ≈ 4 min, N=50 ≈ 17 min.
|
||||
- Most cycles are UNGROUNDED-by-corpus-design (random word triplets
|
||||
rarely overlap with 2010 Wikipedia coverage). The interesting
|
||||
cases are the rare HYBRID/STRICT verdicts on triplets that
|
||||
surprise us, plus the verifier-disagreement cases the teacher
|
||||
catches.
|
||||
- Curated bench (`make bench-qa{,-quick,-smoke}`) is the every-
|
||||
iteration signal. This is the once-a-month sonar.
|
||||
|
||||
## Word filter
|
||||
|
||||
`scripts/bench_emergent.py` uses `^[a-z]{5,12}$` after lowercasing
|
||||
to admit a word. Skips:
|
||||
|
||||
- length ≤ 4 (short words like "the", "and" produce trivially
|
||||
vague questions)
|
||||
- length > 12 (rare scientific terms / loanwords; Hermes struggles
|
||||
to weave them)
|
||||
- non-alpha (apostrophes, hyphens — the unix words file mixes
|
||||
these in)
|
||||
- ALLCAPS (filtered after the lowercasing step admits "Goldman" →
|
||||
"goldman" as a generic noun, which is fine; Hermes treats it
|
||||
as a name regardless)
|
||||
|
||||
Override the filter or word source via:
|
||||
|
||||
```bash
|
||||
python scripts/bench_emergent.py --words-path /custom/words.txt
|
||||
```
|
||||
|
||||
## Teacher review protocol
|
||||
|
||||
The bench script does NOT auto-invoke a teacher. Two reasons:
|
||||
|
||||
1. **Separation of concerns**: generation is automated, judgment
|
||||
is contextual. The teacher (Opus, currently) needs the loop's
|
||||
raw output PLUS access to the corpus-knowledge frame ("is this
|
||||
a 2010 Wikipedia gap or a substrate failure?"). That frame
|
||||
lives in this repo's docs, not in a per-call API spec.
|
||||
|
||||
2. **Future flexibility**: today the teacher is Claude Opus 4.7
|
||||
in this conversation. Tomorrow it might be GPT-5, Claude 5,
|
||||
or a Mixture-of-Experts review committee. Keeping teacher
|
||||
review out of the bench script means swapping teachers is a
|
||||
workflow change, not a code change.
|
||||
|
||||
### How fox brings entries to the teacher
|
||||
|
||||
```bash
|
||||
make bench-emergent-pending # print every entry with teacher==null
|
||||
# … pipe into a Claude session, paste, ask for judgment
|
||||
```
|
||||
|
||||
The teacher's output is a JSON dict to append to the same line:
|
||||
|
||||
```json
|
||||
{
|
||||
"teacher": {
|
||||
"match": true|false,
|
||||
"audit_agreement": "agree"|"disagree"|"unsure",
|
||||
"novelty_class": "known_truth_grounding"
|
||||
| "emergent_synthesis"
|
||||
| "novel_claim"
|
||||
| "no_signal",
|
||||
"score_0_5": 0..5,
|
||||
"reasoning": "<one sentence>",
|
||||
"bench_max_signal": "<retrieval | warrant | prompt | nil>",
|
||||
"reviewed_by": "claude-opus-4-7[1m]",
|
||||
"reviewed_ts": <unix>
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `bench_max_signal` field is the actionable bit: which subsystem
|
||||
should be tuned to address this kind of failure? `retrieval` /
|
||||
`warrant` / `prompt` / `nil` (no action — corpus genuinely lacks
|
||||
the answer).
|
||||
|
||||
### Fields the teacher considers
|
||||
|
||||
- **match**: did the answer address the question? Not "is the
|
||||
answer correct" — the corpus may legitimately not have the
|
||||
answer. The check is "is this answer about the same topic as
|
||||
the question?"
|
||||
- **audit_agreement**: does the substrate's audit_mode (STRICT/
|
||||
HYBRID/UNGROUNDED) or display rung (POINTER-LINKED →
|
||||
ANCHOR-WARRANTED → EVIDENCE-WARRANTED) match what the teacher
|
||||
thinks the answer's grounding deserves?
|
||||
- **novelty_class**: how does the answer relate to 2010 Wikipedia
|
||||
knowledge?
|
||||
- `known_truth_grounding`: cites well-known facts present
|
||||
verbatim in the corpus. The expected case for narrow
|
||||
factoids ("capital of France", "founder of Microsoft").
|
||||
- `emergent_synthesis`: the answer connects facts in a way no
|
||||
single source contains; the substrate did real work
|
||||
composing across sources. The interesting case.
|
||||
- `novel_claim`: the answer goes beyond what the 2010 corpus
|
||||
can ground (post-2010 science, personal opinion, made-up
|
||||
detail). The dangerous case if STRICT/EVIDENCE-WARRANTED.
|
||||
- `no_signal`: UNGROUNDED, or the question was incoherent.
|
||||
Most random-word triplets land here — that's fine.
|
||||
|
||||
## Future: multiple upstreams
|
||||
|
||||
When the substrate supports multiple inference endpoints, the
|
||||
generator and student can use different upstreams to surface
|
||||
upstream-specific failure modes. The CLI flags for this aren't
|
||||
implemented yet but the shape is clear:
|
||||
|
||||
```bash
|
||||
python scripts/bench_emergent.py \
|
||||
--generator-endpoint https://hermes.ai.unturf.com/v1 \
|
||||
--student-endpoint https://other-llm.example/v1
|
||||
```
|
||||
|
||||
Today both default to Hermes. The student is always aborist's
|
||||
`query()` against the configured shard set; only the LLM behind
|
||||
that pipeline is plugged.
|
||||
|
||||
## How to run
|
||||
|
||||
```bash
|
||||
make bench-emergent # 10 cycles, default seed (random)
|
||||
make bench-emergent EMERGENT_N=50 EMERGENT_SEED=42 # bigger sample, reproducible
|
||||
make bench-emergent-pending # print pending teacher review
|
||||
|
||||
# Direct:
|
||||
python scripts/bench_emergent.py --n 50 --seed 42
|
||||
python scripts/bench_emergent.py --print-pending
|
||||
```
|
||||
|
||||
## Append-only log
|
||||
|
||||
`bench/emergent_log.jsonl` accumulates every cycle ever run,
|
||||
across branches, across days. Never rewritten. Each line is one
|
||||
self-describing JSON record (timestamp, words, question, answer,
|
||||
audit, sources, timings, teacher). The log is the substrate's
|
||||
own version of `/var/log/syslog` — every interaction with the
|
||||
emergent harness leaves a trace, and a future you can grep for
|
||||
"every entry where the answer mentioned X" or "every UNGROUNDED
|
||||
result on triplets containing Y."
|
||||
|
||||
A line whose `teacher` field is `null` is awaiting review. A line
|
||||
with a populated `teacher` field is closed. There is no DELETE
|
||||
path; corrections add new lines that supersede old ones.
|
||||
|
|
@ -1,267 +0,0 @@
|
|||
# Concept relations: corpus-derived synonym & rivalry layer
|
||||
|
||||
**Status:** landed — `aborist/concepts/` package, 2026-05-01 (commit
|
||||
`5fd458a`)
|
||||
**Audience:** anyone editing retrieval, storage budget, or planning
|
||||
new concept extractors
|
||||
**Hard constraint:** writes to `concept_relations` MUST NOT affect
|
||||
`document_root`, `chunk_root`, or `cache_key`. The layer is a
|
||||
secondary index over the Merkle-committed corpus; backfilling it is
|
||||
safe across the entire corpus without invalidating any cached answer.
|
||||
|
||||
## Why
|
||||
|
||||
Pre-2026-05-01 the synonym & rivalry data lived as hand-curated
|
||||
`frozenset`s in `aborist/qa/concepts.py`:
|
||||
|
||||
```python
|
||||
SYNONYM_GROUPS = [
|
||||
frozenset({"amd", "athlon", "duron", ...}),
|
||||
frozenset({"intel", "pentium", ...}),
|
||||
... # 7 groups total
|
||||
]
|
||||
RIVALRIES = [(0, 1), (4, 5)]
|
||||
```
|
||||
|
||||
The original commit message even flagged the limit:
|
||||
|
||||
> *Phase 1: hand-curated. Phase 2 idea: derive from Wikipedia's
|
||||
> category graph or "See also" sections.*
|
||||
|
||||
Phase 1 didn't scale. Adding a domain meant editing Python source,
|
||||
committing, pushing, redeploying. A 3.47M-doc corpus has thousands
|
||||
of concept families; hand-curating them is a fool's errand. The
|
||||
corpus already encodes the relationships we'd be hand-rebuilding —
|
||||
"See also" sections, category links, internal-link clusters.
|
||||
Phase 2 reads what's already there.
|
||||
|
||||
## Architecture
|
||||
|
||||
A per-shard `concept_relations` SQLite table (live alongside
|
||||
`documents` and `chunks`):
|
||||
|
||||
```sql
|
||||
CREATE TABLE concept_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_root TEXT NOT NULL,
|
||||
relation_kind TEXT NOT NULL CHECK (relation_kind IN
|
||||
('synonym','antonym','rivalry','category')),
|
||||
token TEXT NOT NULL,
|
||||
target TEXT NOT NULL,
|
||||
evidence_kind TEXT NOT NULL,
|
||||
confidence REAL NOT NULL DEFAULT 1.0,
|
||||
derived_at INTEGER NOT NULL,
|
||||
derived_from TEXT,
|
||||
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
|
||||
);
|
||||
```
|
||||
|
||||
**Append-only.** UNIQUE on `(source_root, relation_kind, token,
|
||||
target, evidence_kind)` makes re-derivation idempotent — re-running
|
||||
an extractor adds nothing if no new relations have appeared since
|
||||
last run. `INSERT OR IGNORE` is the only write path.
|
||||
|
||||
**Per-shard.** Concept relations live in the shard whose document
|
||||
they were derived from. Mesh sync moves shards between peers;
|
||||
relations come along automatically.
|
||||
|
||||
**Cross-shard lookup.** `aborist.concepts.query.synonyms_for` walks
|
||||
every shard via `connect_query`'s UNION view (the same pattern as
|
||||
cross-shard FTS5 search). Concept relations from shard 003 are
|
||||
visible to a query routed at shard 000.
|
||||
|
||||
**Public API stable.** `synonym_expand(tokens, *, shards_dir=...)`
|
||||
& `rivalry_excluded(tokens, ..., shards_dir=...)` keep the legacy
|
||||
shape from `aborist/qa/concepts.py`. The old module is now a
|
||||
back-compat shim that delegates to `aborist.concepts.query`. Call
|
||||
sites in `qa/query.py` thread `shards_dir` through but did not
|
||||
otherwise change.
|
||||
|
||||
## Extractors
|
||||
|
||||
`aborist/concepts/extract.py` ships an extractor framework. Each
|
||||
extractor reads existing corpus rows (no new crawling) & emits
|
||||
concept relations under a stable `evidence_kind` string. An operator
|
||||
can `purge --evidence-kind X` to revoke a single extractor's output
|
||||
without touching manual or other-extractor rows.
|
||||
|
||||
**`link_reciprocity_synonym`** (built-in). For any reciprocal edge
|
||||
pair (A→B AND B→A) in the existing `edges` table, emit a synonym
|
||||
edge between every (title-token-of-A, title-token-of-B) pair.
|
||||
Title-tokens are filtered to ≥4 chars + stopword-stripped to keep
|
||||
generic words like "the" or "and" from generating noise.
|
||||
|
||||
The `edges` table is already populated on ingest. For Wikipedia,
|
||||
the wikitext parser pulls every `[[link]]` as an edge row. For HTML,
|
||||
`aborist/sources/html_page.py:parse_html` pulls every `<a href>`.
|
||||
So the link reciprocity extractor works for any corpus that flows
|
||||
through aborist's standard ingest path: Wikipedia, crawled HTML
|
||||
sites (russell.ballestrini.net pattern), or any other document
|
||||
graph the corpus already encodes.
|
||||
|
||||
## Storage cost — measured
|
||||
|
||||
Backfill on 6 GB of Wikipedia (2003 cur dump, 4 shards, 3.47M docs):
|
||||
|
||||
| Shard | Docs | Resolved edges | Reciprocal pairs | Synonyms inserted | Backfill time | Storage |
|
||||
|------------|------------|----------------|------------------|-------------------|---------------|---------|
|
||||
| `000.db` | ~870k | 2,703,287 | 13,562 | 71,288 | 50.6s | 23.52 MB |
|
||||
| `001.db` | ~870k | 2,571,390 | 14,078 | 73,351 | 47.7s | 24.20 MB |
|
||||
| `002.db` | ~870k | 2,772,156 | 13,708 | 72,576 | 1m34s | 23.92 MB |
|
||||
| `003.db` | ~870k | 2,707,627 | 13,800 | 72,633 | 1m03s | 23.94 MB |
|
||||
| **total** | **3.47M** | **10,754,460** | **55,148** | **289,848** | **4m16s** | **95.58 MB** |
|
||||
|
||||
**Per-row cost: ~330 bytes.** The bulk is the 64-char hex
|
||||
`source_root` stored both in the table & in the UNIQUE auto-index
|
||||
that enforces the idempotent-re-derivation key. Per-shard breakdown:
|
||||
|
||||
| Component | Size per shard | Purpose |
|
||||
|--------------------------------|----------------|---------|
|
||||
| `concept_relations` table | ~10.0 MB | rows |
|
||||
| `sqlite_autoindex` (UNIQUE) | ~9.3 MB | enforces idempotent re-derivation |
|
||||
| `idx_concept_evid` | ~1.8 MB | for `purge --evidence-kind X` |
|
||||
| `idx_concept_kind` | ~1.0 MB | filter by relation_kind |
|
||||
| `idx_concept_target` | ~1.0 MB | reverse lookup |
|
||||
| `idx_concept_token` | ~1.0 MB | forward lookup |
|
||||
|
||||
**95.58 MB on a 6 GB corpus = 1.6% storage tax for the entire
|
||||
denormalization.** Full backfill took 4m16s wall-clock across the
|
||||
4 wiki shards — the cost is paid once.
|
||||
|
||||
## Storage choice — keep, don't compact
|
||||
|
||||
We considered three compactions & rejected all three. Documented
|
||||
here so a future maintainer doesn't reopen the question without a
|
||||
measured reason.
|
||||
|
||||
### Option A — drop `idx_concept_evid` (saves ~1.8 MB / shard, ~7 MB total)
|
||||
|
||||
`idx_concept_evid` exists to make `purge --evidence-kind X` cheap
|
||||
(one index lookup per evidence_kind instead of a full table scan).
|
||||
Without it, purge becomes O(N) over the whole table — acceptable
|
||||
for a one-shot cleanup, painful for a tight extractor-development
|
||||
loop where an operator runs purge then re-derive many times.
|
||||
|
||||
**Decision:** keep. The 1.8 MB saving doesn't justify the painful
|
||||
debugging loop.
|
||||
|
||||
### Option B — store `source_root` as 32-byte BLOB instead of 64-char hex TEXT (saves ~5 MB / shard, ~20 MB total)
|
||||
|
||||
The `source_root` column is the largest single contributor to
|
||||
storage (~50% of per-row cost). Storing as BLOB instead of hex TEXT
|
||||
halves it.
|
||||
|
||||
**Decision:** keep TEXT. The rest of the schema (documents.document_root,
|
||||
chunks.leaf_hash, providence_cache.source_root, audit_events.subject_root)
|
||||
all use hex TEXT. Mixing TEXT vs BLOB across tables hurts schema
|
||||
legibility & complicates joins. 5 MB saving doesn't justify the
|
||||
inconsistency.
|
||||
|
||||
### Option C — normalize `source_root` into a `source_lookup(id, hex)` foreign key (saves ~7 MB / shard, ~28 MB total)
|
||||
|
||||
Replace the 64-char hex `source_root` column with a 4-8 byte
|
||||
`source_id` integer pointing at a `source_lookup` table that maps
|
||||
id ↔ hex.
|
||||
|
||||
**Decision:** keep flat. The savings are real but every concept
|
||||
lookup adds a JOIN. The lookup is in the retrieval hot path
|
||||
(`synonym_expand` is called per-query). Adding a JOIN for a 0.5%
|
||||
storage win is the wrong direction.
|
||||
|
||||
### Final verdict
|
||||
|
||||
**1.6% storage tax is fine.** Concept relations are a query-time
|
||||
performance accelerator over an already-3.5GB-per-shard corpus.
|
||||
The tax is paid once at backfill & every retrieval-time lookup
|
||||
benefits. If a future shard layout pushes total storage to a
|
||||
different cost regime (e.g. compressed columnstore), revisit Option
|
||||
B at that point — the boundary changes the tradeoff math.
|
||||
|
||||
## How to use
|
||||
|
||||
### Backfill the existing corpus
|
||||
|
||||
```bash
|
||||
make backfill-concepts # all extractors × all shards × 4 workers
|
||||
make backfill-concepts CONCEPTS_WORKERS=2 # tune parallelism
|
||||
```
|
||||
|
||||
Driven by `scripts/backfill_concepts.py` — runs each registered
|
||||
extractor in `aborist.concepts.extract.EXTRACTORS` (`link_reciprocity`,
|
||||
`token_idf`, `documents_fts`) across every numeric-stem shard in
|
||||
parallel. Idempotent — re-running on already-backfilled shards is
|
||||
~no-op-cost (INSERT OR IGNORE / DELETE-and-rebuild semantics depending
|
||||
on extractor).
|
||||
|
||||
(CLI commands `aborist concepts {seed,list,add,derive,purge}` deferred
|
||||
to a follow-on commit.)
|
||||
|
||||
### Add a manual relation (e.g. when a domain expert sees a gap)
|
||||
|
||||
```python
|
||||
from aborist.concepts.store import add_concept_relation
|
||||
from aborist.store import connect
|
||||
|
||||
conn = connect('~/.aborist/shards/000.db')
|
||||
add_concept_relation(
|
||||
conn,
|
||||
source_root='__manual__',
|
||||
relation_kind='synonym',
|
||||
token='telepathy',
|
||||
target='neurotechnology',
|
||||
evidence_kind='manual',
|
||||
derived_from='fox 2026-05-01: brain-tech retrieval gap',
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
```
|
||||
|
||||
### Revoke a buggy extractor's output
|
||||
|
||||
```python
|
||||
from aborist.concepts.store import purge_by_evidence_kind
|
||||
from aborist.store import connect
|
||||
|
||||
conn = connect('~/.aborist/shards/000.db')
|
||||
n = purge_by_evidence_kind(conn, 'broken_extractor_v1')
|
||||
conn.commit()
|
||||
print(f'removed {n} rows')
|
||||
```
|
||||
|
||||
`evidence_kind='manual'` rows are NOT touched by a purge of any
|
||||
other kind — manual contributions are safe.
|
||||
|
||||
## Adding new extractors
|
||||
|
||||
1. Implement a callable
|
||||
```python
|
||||
def my_extractor(conn, *, derived_from=None) -> dict[str, int]:
|
||||
...
|
||||
return {"items_inserted": N, "items_skipped": K}
|
||||
```
|
||||
that walks the shard's existing rows (`documents`, `chunks`,
|
||||
`edges`, `derivations`, …) & calls
|
||||
`aborist.concepts.store.add_concept_relation` for each finding.
|
||||
|
||||
2. Pick a stable `evidence_kind` string. Don't reuse an existing
|
||||
one unless your extractor genuinely produces the same kind of
|
||||
output as that one (so `purge --evidence-kind X` semantics
|
||||
stay clean).
|
||||
|
||||
3. Register in `aborist/concepts/extract.py:EXTRACTORS`.
|
||||
|
||||
4. Document the trade-offs in this file alongside `link_reciprocity`.
|
||||
|
||||
## Deferred follow-ons
|
||||
|
||||
- **CLI commands** — `aborist concepts {seed,list,add,derive,purge}`.
|
||||
Today the helpers are accessible via `python -c "..."`.
|
||||
- **Wikipedia See-also extractor** — parses `==See also==` sections
|
||||
in chunked wikitext beyond what `edges` already captures (some
|
||||
See-also entries are wikilinks already in edges; some are bullet
|
||||
lists with annotations that aren't).
|
||||
- **Wikipedia category extractor** — parses `[[Category:X]]` tails
|
||||
& emits `relation_kind='category'` rows.
|
||||
- **Hatnote / disambiguation extractor** — parses `{{about|...}}`
|
||||
& `{{not to be confused with|...}}` templates as antonym /
|
||||
rivalry signals.
|
||||
|
|
@ -1,257 +0,0 @@
|
|||
# Mesh deploy — two-host runbook
|
||||
|
||||
This is the operator runbook for standing up an aborist mesh between two
|
||||
real hosts. The protocol is documented in `docs/mesh.md`; this file
|
||||
covers what to type, in what order, with what to verify at each step.
|
||||
|
||||
> Every step assumes both peers have aborist installed and have already
|
||||
> ingested some local corpus. If you only want to test the wire on
|
||||
> localhost-loopback, see `tests/test_mesh_wire_e2e.py` instead.
|
||||
|
||||
## Naming
|
||||
|
||||
Two hosts in this runbook:
|
||||
|
||||
- **`alice`** — admin, founder of group `myteam`. Runs `mesh serve`.
|
||||
- **`bob`** — joining member. Runs `mesh sync` outbound to alice.
|
||||
|
||||
Substitute hostnames / IPs / ports per your environment.
|
||||
|
||||
## 0. Pre-flight on each host
|
||||
|
||||
```sh
|
||||
# Confirm aborist + extras + tests:
|
||||
make bootstrap
|
||||
make test
|
||||
.venv/bin/aborist --version
|
||||
```
|
||||
|
||||
Both hosts should be on the same git commit. Mismatched
|
||||
`schema_version` / `chunking_version` / `canonicalization_version`
|
||||
will be rejected by the v9.8 admissibility check; aborist refuses
|
||||
silent corruption.
|
||||
|
||||
## 1. Initialize mesh on each peer
|
||||
|
||||
```sh
|
||||
# alice — first peer becomes founder of epoch 0:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id alice
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
|
||||
|
||||
# bob — initializes a separate identity in the same group name:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id bob
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
|
||||
```
|
||||
|
||||
At this point alice and bob both think they're the sole member of
|
||||
`myteam`. The next step adds bob to alice's roster — and vice versa —
|
||||
so both rosters carry both pubkeys at the same epoch.
|
||||
|
||||
## 2. Exchange pubkeys (out-of-band)
|
||||
|
||||
bob's public keys must reach alice via a trusted channel. Mesh has no
|
||||
built-in introduction protocol; signal, in-person hand-off, or a
|
||||
signed file all work.
|
||||
|
||||
```sh
|
||||
# On bob:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
|
||||
# -> { "member_id": "bob", "sign_pub_hex": "...", "dh_pub_hex": "..." }
|
||||
```
|
||||
|
||||
Send the two hex strings to alice. Verify the channel out of band.
|
||||
|
||||
Same direction in reverse so bob can enroll alice:
|
||||
|
||||
```sh
|
||||
# On alice:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
|
||||
```
|
||||
|
||||
## 3. Mutual enrollment
|
||||
|
||||
```sh
|
||||
# On alice (admin) — enroll bob at alice's epoch:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
|
||||
--member-id bob \
|
||||
--sign-pub <bob_sign_pub_hex> \
|
||||
--dh-pub <bob_dh_pub_hex>
|
||||
|
||||
# On bob — enroll alice at bob's epoch (so bob's local roster knows
|
||||
# alice's pubkey for signature verification when alice's announces
|
||||
# arrive):
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
|
||||
--member-id alice \
|
||||
--sign-pub <alice_sign_pub_hex> \
|
||||
--dh-pub <alice_dh_pub_hex>
|
||||
```
|
||||
|
||||
Each `mesh add` bumps the local epoch and writes a `mesh_epoch_rotate`
|
||||
audit event. Verify:
|
||||
|
||||
```sh
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh members
|
||||
```
|
||||
|
||||
Both peers should now show 2 members at epoch 1.
|
||||
|
||||
## 4. Start alice's gossip server
|
||||
|
||||
```sh
|
||||
# On alice — bind on a routable interface:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh serve \
|
||||
--host 0.0.0.0 --port 8400
|
||||
# Stdout: {"status": "serving", "url": "http://0.0.0.0:8400", ...}
|
||||
```
|
||||
|
||||
Leave this running. Open the firewall to allow bob to reach
|
||||
`alice:8400`. Production deployments should put a TLS terminator
|
||||
(Caddy / nginx) in front; the wire speaks plain HTTP — TLS is the
|
||||
operator's choice.
|
||||
|
||||
Verify reachability from bob:
|
||||
|
||||
```sh
|
||||
# On bob:
|
||||
curl http://alice.example.com:8400/mesh/info | jq
|
||||
# -> { "v": 1, "member_id": "alice", "group_name": "myteam",
|
||||
# "current_epoch": 1, "sign_pub_hex": "..." }
|
||||
```
|
||||
|
||||
If `member_id` matches what bob has in his roster for "alice", and the
|
||||
`sign_pub_hex` matches what bob enrolled, you're good. If they don't
|
||||
match, bob is talking to the wrong peer or alice's enrollment was wrong;
|
||||
do not push gossip until the discrepancy is resolved.
|
||||
|
||||
## 5. Bob announces his local roots to alice
|
||||
|
||||
```sh
|
||||
# On bob:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
|
||||
--peer http://alice.example.com:8400 \
|
||||
--limit 100
|
||||
```
|
||||
|
||||
Output: JSON status, count of acked vs errored announces.
|
||||
|
||||
Each accepted announce writes one `mesh_received` event into alice's
|
||||
audit chain. Verify on alice:
|
||||
|
||||
```sh
|
||||
sqlite3 ~/.aborist/aborist.db \
|
||||
"SELECT COUNT(*) FROM audit_events WHERE event_type='mesh_received'"
|
||||
```
|
||||
|
||||
Cross-check chain integrity:
|
||||
|
||||
```sh
|
||||
make chain-check
|
||||
```
|
||||
|
||||
Should return `0` chain breaks.
|
||||
|
||||
## 6. Make sync bidirectional
|
||||
|
||||
The current `mesh sync` only pushes outbound. To get alice's roots
|
||||
onto bob, run `mesh sync` from alice pointed at bob:
|
||||
|
||||
```sh
|
||||
# On alice (in a new terminal — keep mesh serve running):
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
|
||||
--peer http://bob.example.com:8400 --limit 100
|
||||
```
|
||||
|
||||
This requires bob to also be running `mesh serve`. Symmetric setup is
|
||||
the standard mode.
|
||||
|
||||
## 7. Pulling missing bodies (when shipped)
|
||||
|
||||
Today only ANNOUNCE_ROOT is exercised by `mesh sync`. The wire layer
|
||||
already implements REQUEST_BODY/DELIVER_BODY with Merkle verification
|
||||
on the client side; the CLI hookup (`mesh pull --root <hex> --peer
|
||||
<url>`) is on the queue. Once shipped, the pull-on-miss workflow is:
|
||||
|
||||
```sh
|
||||
# bob discovers via announces that alice has document X he doesn't:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh pull \
|
||||
--root <document_root_hex> \
|
||||
--peer http://alice.example.com:8400
|
||||
# bob's local store now has X (Merkle-verified at receive).
|
||||
```
|
||||
|
||||
## 8. Eviction protocol
|
||||
|
||||
If bob leaves the team, alice (admin) kicks. This bumps alice's epoch
|
||||
and rewraps the next epoch secret to the post-bob roster:
|
||||
|
||||
```sh
|
||||
# On alice:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh kick \
|
||||
--member-id bob \
|
||||
--reason "left team 2026-04-28"
|
||||
```
|
||||
|
||||
Bob's prior signatures stay verifiable forever (his roster row at
|
||||
older epochs is preserved on disk). Any AEAD-protected gossip from
|
||||
epoch+1 onward is opaque to him — that's the eviction guarantee.
|
||||
|
||||
Important: kicking only mutates alice's local state. If bob's host is
|
||||
still running `mesh serve`, alice should also stop sending to him (or
|
||||
he'll keep accepting alice's announces under the OLD epoch she's no
|
||||
longer broadcasting from). Operationally, kicks should pair with
|
||||
either: (a) dropping bob's URL from alice's sync targets, OR (b)
|
||||
announcing the kick to remaining members so they update their
|
||||
rosters.
|
||||
|
||||
## 9. Operational checks
|
||||
|
||||
Every chain-write op (`mesh init`, `mesh add`, `mesh kick`, `mesh
|
||||
rotate`, every received announce) extends the local audit chain. The
|
||||
fast probe is:
|
||||
|
||||
```sh
|
||||
make chain-check # single db
|
||||
make chain-check-shards # every db in $(SHARDS_DIR)
|
||||
```
|
||||
|
||||
Both should always print `0`. Non-zero = operator must investigate
|
||||
before further operations; data has diverged from the audit trail.
|
||||
|
||||
## 10. Reset / teardown
|
||||
|
||||
To take a peer fully out of the mesh (irreversible — fresh keys
|
||||
needed to rejoin):
|
||||
|
||||
```sh
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh disable
|
||||
# Identity + roster history stay on disk for forensics. Re-enable
|
||||
# requires another peer to re-add this member at a fresh epoch.
|
||||
```
|
||||
|
||||
For full local nuke (developers only — destroys identity + chain):
|
||||
|
||||
```sh
|
||||
sqlite3 ~/.aborist/aborist.db <<'SQL'
|
||||
DELETE FROM mesh_identity;
|
||||
DELETE FROM mesh_roster;
|
||||
DELETE FROM mesh_epochs;
|
||||
SQL
|
||||
```
|
||||
|
||||
## What's not yet shipped
|
||||
|
||||
These features are documented in the protocol but not wired into the
|
||||
CLI yet:
|
||||
|
||||
- **`mesh pull <root>`** — fetch a body on cache miss.
|
||||
- **Per-peer audit-chain merge** — receiver tracks each sender's last
|
||||
known event_hash and rejects fork events. Today the receiver verifies
|
||||
signatures only; chain-of-claims tracking is deferred.
|
||||
- **AEAD body encryption** — wire signs envelope bytes for integrity;
|
||||
body confidentiality (encrypting against the per-epoch shared
|
||||
secret) is optional in the contract and not yet wired.
|
||||
|
||||
When these land, this runbook will gain steps for them.
|
||||
1211
docs/modules.md
1211
docs/modules.md
File diff suppressed because it is too large
Load diff
|
|
@ -1,240 +0,0 @@
|
|||
# Pointer / Quote / JSON answer modes — bench, failure analysis, roadmap
|
||||
|
||||
**Date:** 2026-04-30
|
||||
**Bench:** `bench/qa_sweep.py`, 22 questions × 3 samples × 3 modes = 198 LLM calls
|
||||
**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx)
|
||||
**Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.aborist/shards`
|
||||
**Verifier hardening at the time of bench:** chunk cap=2, pointer cap=2, coverage threshold=0.30, manual_quote rule removed, partial-grounding split, bare-name guard (≥2 content tokens), lazy-anchor demote, noisy-marker tie-ins.
|
||||
|
||||
## Aggregate
|
||||
|
||||
| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | grounded (S+H) | mean ratio | mean latency |
|
||||
|------|------|--------|--------|------------|-----|-------------|----------------|------------|--------------|
|
||||
| `quote` | 66 | 31 | 20 | 15 | 0 | **47%** | 51 | 0.70 | 7.7s |
|
||||
| `claim_lattice_pointer` | 66 | 14 | 34 | 18 | 0 | 21% | 48 | 0.54 | **4.4s** |
|
||||
| `claim_lattice` (JSON, guided_json) | 66 | 26 | 12 | 9 | **19** | 39%* | 38 | 0.51 | 4.4s |
|
||||
|
||||
\* JSON strict-rate is 26/66 over all runs but 26/47 (55%) over error-free runs. The 19 errors are operationally visible to the user.
|
||||
|
||||
### Post-retry / post-trim-and-verify rerun (same day)
|
||||
|
||||
After landing two improvements derived from the analysis below — HTTP retry on transient 5xx in `OpenAICompatibleClient` and pointer-cap trim-and-verify in `verify_claim_lattice` — the bench was rerun on the same 22-question set:
|
||||
|
||||
| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | grounded (S+H) | mean ratio | mean latency |
|
||||
|------|------|--------|--------|------------|-----|-------------|----------------|------------|--------------|
|
||||
| `quote` | 66 | 31 | 18 | 17 | 0 | 47% | 49 | 0.70 | 5.9s |
|
||||
| `claim_lattice_pointer` | 66 | 16 | 40 | 10 | 0 | 24% | **56** | 0.59 | **4.6s** |
|
||||
| `claim_lattice` (JSON, guided_json + retry) | 66 | **33** | 21 | 12 | **0** | **50%** | 54 | 0.70 | 5.8s |
|
||||
|
||||
**Deltas vs the pre-improvement bench above:**
|
||||
- JSON errors: **19 → 0** — retry cleared the 502 cluster entirely. The errors *were* upstream vLLM 502s, not Hermes-can't-produce-JSON failures (the pre-improvement diagnosis above was wrong; HTTP-status inspection proved it).
|
||||
- JSON strict-rate: 39% → **50%** — now leads all three modes (was lowest).
|
||||
- JSON grounded count: 38 → 54 (+16) — directly from the recovered runs.
|
||||
- Pointer STRICT: 14 → 16 (+2) — trim-and-verify rescued correct over-cited claims (Mona Lisa case).
|
||||
- Pointer grounded: 48 → 56 (+8) — UNGROUNDED 18 → 10.
|
||||
- Quote: largely unchanged within sampling noise, latency dropped 7.7s → 5.9s.
|
||||
|
||||
**The picture flipped.** Pre-improvement, the recommendation was "don't switch default to JSON, the 29% error rate is unacceptable." Post-improvement, JSON has the **highest strict-rate**, **highest grounded count tied with pointer**, **zero errors**, and similar latency. JSON is now the strongest default candidate.
|
||||
|
||||
The `make query` target's default `ANSWER_MODE` was flipped from `claim_lattice_pointer` to `claim_lattice` on 2026-04-30 to reflect this. The library-level `DEFAULT_ANSWER_MODE` stays `"quote"` so unit tests using `StubClient` aren't disrupted; pointer mode is still available via `ANSWER_MODE=claim_lattice_pointer`.
|
||||
|
||||
### Architectural fix — JSON mode uses pointer IDs (E1, E2, …) instead of content-addressed evidence_ids
|
||||
|
||||
A separate failure mode surfaced after the retry/trim work: cross-document relationship questions consistently landed `UNGROUNDED 0/1` in JSON mode despite the model writing the correct answer text. Diagnosis of `who is homer simpson's boss?` in JSON mode showed:
|
||||
|
||||
```
|
||||
{"claims":[{"text":"Homer Simpson's boss is Mr. Burns.","evidence_ids":["E1b6e396"]}]}
|
||||
```
|
||||
|
||||
The runtime had `Eed1b6e396` for that chunk; Hermes-3-8B emitted `E1b6e396` — a plausible-looking near-miss the verifier rightly flagged as `UNKNOWN_EVIDENCE_ID`. The model was *fabricating* content-addressed evidence_ids when the real ones felt awkwardly long.
|
||||
|
||||
Fix landed in commit `bb8450d`: the JSON-mode prompt-facing surface switched from content-addressed evidence_ids (`Eed1b6e396`) to pointer IDs (`E1`, `E2`, …) — same as `claim_lattice_pointer` mode. The runtime still resolves each pointer_id to its content-addressed evidence_id internally and stores **that** in `evidence_id_pairs` for cache & run-DAG continuity. Only the prompt-facing string changes. After the fix, `who is homer simpson's boss?` lands `STRICT 1/1` in JSON mode.
|
||||
|
||||
Why pointer IDs work where content-addressed didn't:
|
||||
- short, enumerable, fabrication-obvious — if only `E1`-`E10` are shown, an emitted `E27` reads as a schema violation at a glance
|
||||
- distribution-natural for small models (citation-style is heavily represented in training)
|
||||
- the proof path stays content-addressed (the verifier's audit chain still hashes content-addressed ids), so the human/model surface change doesn't weaken the v9.8 admissibility ledger
|
||||
|
||||
### Token-runaway guard — JSON-mode stop-sequence
|
||||
|
||||
Post-pointer-ID-switch bench (2026-04-30T19-55-11Z) found a residual JSON-mode failure on broad-descriptive questions: ~4 of 66 runs landed `UNGROUNDED 0/0` at 12-15s instead of the normal 2-5s. Inspection: Hermes emitted a valid claim object then kept generating whitespace / blank lines until `max_tokens=512` exhausted. The truncated payload didn't parse and the lenient pre-parser returned no claims.
|
||||
|
||||
Concrete instances:
|
||||
- `tell me about the apollo program` — 3/3 samples runaway
|
||||
- `tell me about the python programming language` — 1/3 runaway
|
||||
|
||||
Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well-formed JSON-mode output never contains a blank line — the model emits one object on a single line (or with simple internal newlines), never `\n\n`. The stop sequence is the runaway signature itself; legitimate output is never truncated. Folds into `governance_policy_hash` via `claim_lattice_json_stop_sequences` policy field so changing the list invalidates prior cached records.
|
||||
|
||||
### Bench progression summary
|
||||
|
||||
| run | bench | quote STRICT | pointer STRICT | JSON STRICT | JSON err | JSON grounded |
|
||||
|-----|-------|--------------|----------------|-------------|----------|----------------|
|
||||
| baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 |
|
||||
| post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 |
|
||||
| post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 |
|
||||
| post-stop-sequence | late-evening n=3 | 29 | 14 | **37** | 0 | **60** |
|
||||
|
||||
JSON mode net: 26 → 37 STRICT (+11), 19 → 0 errors, 38 → 60 grounded (+22) over the day. Strict-rate climbed 39% → 56% (+17pp). Each of the four improvements addressed a named failure mode surfaced by the prior bench.
|
||||
|
||||
Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure.
|
||||
|
||||
**Headline:** the three modes occupy distinct points on a strict-vs-honest-vs-stable trade-off:
|
||||
|
||||
- `quote` — highest strict-rate but rests on the older verifier path (substring quote-pair extraction). Includes false-STRICT cases the pointer-mode hardening discovered (e.g. claims that pass token-coincidence but cite the wrong source). Slowest of the three.
|
||||
- `claim_lattice_pointer` — hardened path; lowest strict-rate because previously-bogus STRICTs were honestly demoted to HYBRID. "Most truthful" mode but visibly fewer perfect-confidence ratings.
|
||||
- `claim_lattice` (JSON) — fastest strong-form mode on narrow factoids (5-for-5 perfect strict on the simplest questions) but **brittle**: 29% raw error rate driven by Hermes-3-8B failing to satisfy `guided_json` constraints on certain question shapes.
|
||||
|
||||
## Per-question shape × mode breakdown
|
||||
|
||||
Verdict notation: `S=STRICT H=HYBRID U=UNGROUNDED e=error`. Three samples per cell; majority shown.
|
||||
|
||||
### Narrow factoids — 5 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| who founded apple computer? | 0/3 STRICT (always UNGROUNDED) | 3/3 STRICT | **3/3 STRICT** |
|
||||
| what is the capital of france? | 2 STRICT, 1 UNGROUNDED | 3/3 STRICT | **3/3 STRICT** |
|
||||
| who wrote the linux kernel? | 2 STRICT, 1 HYBRID | 3/3 HYBRID | **3/3 STRICT** |
|
||||
| when was the python programming language created? | 3/3 STRICT | 0/3 STRICT (3 UNGROUNDED) | **3/3 STRICT** |
|
||||
| who painted the mona lisa? | 0/3 STRICT (1H, 2U) | 0/3 STRICT (3 UNGROUNDED) | **3/3 STRICT** |
|
||||
|
||||
JSON dominates narrow factoids. Pointer mode is unstable here — Mona Lisa, Apollo Python date, both 0/3 STRICT in pointer because Hermes paraphrases the source span enough that the coverage check trips. `quote` is similarly variable. JSON's grammar-constrained output makes it cleanest because the model emits short structured `{"text":"X","evidence_ids":[...]}` records that don't need to negotiate prose-style coverage thresholds.
|
||||
|
||||
### Broad descriptive — 4 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| tell me about connecticut | 3/3 STRICT | 3/3 HYBRID | 3/3 UNGROUNDED |
|
||||
| tell me about python language | 3/3 STRICT | 3/3 HYBRID | 2 HYBRID, 1 UNGROUNDED |
|
||||
| tell me about charles babbage | 3/3 STRICT | 3/3 HYBRID | 1 HYBRID, 2 UNGROUNDED |
|
||||
| tell me about the apollo program | 3/3 STRICT | 3/3 HYBRID | 3/3 UNGROUNDED |
|
||||
|
||||
`quote` mode wins these on aggregate — but the win is partly because the verifier's looser quote-pair check passes claims the pointer / JSON paths reject. Pointer mode produces honest HYBRID (some claims grounded, some not) which is closer to the truth of what Hermes is actually doing on encyclopedic prompts. JSON mode collapses heavily on descriptives: the model often emits no claims it considers solidly grounded → empty-or-near-empty `claims` array → UNGROUNDED.
|
||||
|
||||
### Entity list — 3 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| list the founders of microsoft | 2 STRICT, 1 UNGROUNDED | 3/3 HYBRID | **3/3 STRICT** |
|
||||
| what dinosaurs were in the first jurassic park film? | 1 STRICT, 2 HYBRID | 0/3 STRICT (1H, 2U) | **3/3 HYBRID** |
|
||||
| who are the members of the beatles? | 3/3 STRICT | 3/3 HYBRID | 3/3 HYBRID |
|
||||
|
||||
JSON handles list shapes well — structured output is a natural fit. Pointer mode's bare-name guard correctly catches the JP-dinosaurs case (2 UNGROUNDED) where the model emits one-token names; JSON mode sidesteps the bare-name guard because each claim text has the surrounding context required by the schema.
|
||||
|
||||
### Relationship / multi-fact — 3 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| who is supermans girlfriend? | 0 STRICT, 2 HYBRID, 1 UNGROUNDED | 0 STRICT, 2 HYBRID, 1 UNGROUNDED | 2 STRICT, 1 ERROR |
|
||||
| who is bilbo baggins's nephew? | 0/3 STRICT (3 UNGROUNDED) | **3/3 STRICT** | 0/3 (3 ERRORS) |
|
||||
| what is the relationship between linux and unix? | 1 STRICT, 2 HYBRID | 3/3 HYBRID | 0/3 (3 ERRORS) |
|
||||
|
||||
Pointer mode shines on `bilbo's nephew` (3/3 STRICT). Quote and JSON both fail. Mixed picture overall.
|
||||
|
||||
### Comparison — 2 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| what's the difference between linux and bsd? | 2 STRICT, 1 HYBRID | 0/3 STRICT (3 UNGROUNDED) | 0/3 (3 ERRORS) |
|
||||
| how does intel compare to amd? | 0/3 STRICT (3 UNGROUNDED) | 0/3 STRICT (3 UNGROUNDED) | 0/3 (3 ERRORS) |
|
||||
|
||||
Comparison questions break JSON mode catastrophically (6 errors / 6 runs). Quote handles linux-vs-bsd; pointer fails it. Intel-vs-AMD is hard for everyone — corpus likely thin or rivalry-exclusion is too aggressive.
|
||||
|
||||
### Niche / partial — 2 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| what is the boltzmann constant? | 1 STRICT, 2 HYBRID | 2 STRICT, 1 HYBRID | 0/3 (3 ERRORS) |
|
||||
| who invented the doppler effect? | 2 STRICT, 1 HYBRID | **3/3 STRICT** | 0/3 (3 ERRORS) |
|
||||
|
||||
Pointer wins niche-factoid where corpus content is solid. JSON fails completely — these queries return very fast (0.8–1.5s) with errors, suggesting Hermes is producing malformed JSON rather than schema-valid output.
|
||||
|
||||
### Adversarial / honest-refusal — 3 questions
|
||||
|
||||
| question | quote | pointer | json |
|
||||
|----------|-------|---------|------|
|
||||
| isn't it true that the great wall of china is visible from space? | 3/3 HYBRID | 3/3 HYBRID | 3/3 STRICT |
|
||||
| who is the prime minister of mars? | 0 STRICT, 2 HYBRID, 1 UNGROUNDED | 3/3 UNGROUNDED | **3/3 STRICT** ← honest "no PM exists" + cited a definitional chunk |
|
||||
| what year did the cold fusion breakthrough happen? | 3/3 HYBRID | 3/3 HYBRID | 3/3 HYBRID |
|
||||
|
||||
JSON's strict-form on Mars is a *correct refutation*: model wrote "There is no prime minister of Mars, as Mars is not a sovereign nation" and cited a chunk explaining what "prime minister" means (Canadian PM definition span). Coverage passed because the term `prime minister` overlapped. This is the **right** behavior for a refute-the-premise question — the verifier passed honest definitional grounding.
|
||||
|
||||
## Failure modes, by class
|
||||
|
||||
### F1 — JSON-mode parse-failure errors (29% of JSON runs)
|
||||
|
||||
Six question shapes consistently 3/3 ERROR in JSON mode at 0.8–1.5s latency:
|
||||
|
||||
- comparison (linux vs unix, linux vs bsd, intel vs amd)
|
||||
- technical-term niche (boltzmann constant, doppler effect)
|
||||
- relationship (bilbo's nephew)
|
||||
- partial (supermans girlfriend, 1 of 3)
|
||||
|
||||
Pattern: very fast latency = Hermes returns text that fails JSON parsing or violates the `guided_json` schema. Same questions in pointer mode often succeed (bilbo: pointer 3/3 STRICT vs JSON 3/3 ERROR; doppler: pointer 3/3 STRICT vs JSON 3/3 ERROR).
|
||||
|
||||
**Root cause hypothesis:** vLLM's `guided_json` is forcing a particular output structure, but Hermes-3-8B's training distribution for these question shapes wants prose. The grammar mask collides with the model's natural completion → degenerate output → schema violation → `_lenient_json_parse` produces `json_fixups` warnings or fails outright.
|
||||
|
||||
### F2 — Pointer-mode mode-collapse on broad descriptives
|
||||
|
||||
Confirmed: connecticut, python, charles babbage, apollo program — all 3/3 HYBRID with claim ratios in 0.33–0.78 range. Hermes writes encyclopedic paragraphs that paraphrase source but each claim text has more tokens than the cited span supports → coverage threshold passes some claims, rejects others.
|
||||
|
||||
This is *correct* honest behavior given the chunk cap (2 chunks per source). Quote mode "wins" these only because its verifier has more lenient acceptance.
|
||||
|
||||
### F3 — Pointer-mode strict-rate softness on narrow factoids
|
||||
|
||||
Surprising: pointer mode hits 0/3 STRICT on `mona lisa`, `python date`, on the same questions JSON nails 3/3 STRICT. The model writes narrow correct claims but the coverage threshold or claim-shape check fires.
|
||||
|
||||
Inspection of failed mona lisa pointer answers needed — likely the model wrote a multi-clause claim like "The Mona Lisa was painted by Leonardo da Vinci, an Italian Renaissance artist." (~6 content tokens) and the cited span has only "Leonardo da Vinci" verbatim — coverage 2/6 = 33% → just over threshold but the claim contains "Italian", "Renaissance", "artist" not in the span. Need to verify.
|
||||
|
||||
### F4 — Quote-mode false-STRICT on bogus citations
|
||||
|
||||
Quote-mode's strict-rate (47%) is the highest but includes false-positives we discovered while hardening pointer mode. Specifically: claims wrapped in `"..."` whose quoted content contains tokens that exist somewhere in context but not in a span that supports the claim. Pre-pointer-hardening, a similar test set would show pointer at 50% STRICT too — the pointer hardening exposed and rejected the false-STRICTs that quote still credits.
|
||||
|
||||
This means quote-mode's strict-rate over-reports honest grounding. The correct comparison metric is **strict-rate over an audited subset** which is laborious but is the only honest scoreboard.
|
||||
|
||||
### F5 — Lazy-anchor smell still active
|
||||
|
||||
JP-dinosaurs in pointer mode: 0/3 STRICT (1 HYBRID, 2 UNGROUNDED) — the bare-name guard correctly rejected single-word claims. JSON mode: 3/3 HYBRID — JSON's structured `text` field carries enough context to clear the bare-name guard, but the citations are still anchoring on game tie-in chunks (`Jurassic Park (NES game)` made it into top-K despite the noisy-marker addition for "operation genesis" + "the game" + "video games" — "(NES game)" parens form passes the substring check but not in the way I'd hoped — the marker `"the game"` matches `"the NES game"` only if the title is exactly that, which it isn't here. Need to extend markers further or generalize via regex).
|
||||
|
||||
## Conclusions
|
||||
|
||||
1. **Don't switch the default to JSON.** The 29% raw error rate on diverse questions is operationally unacceptable — 6 of 22 question shapes catastrophically fail. JSON wins narrow factoids cleanly but Hermes-3-8B can't sustain valid JSON output on comparison, technical, and relationship questions.
|
||||
|
||||
2. **Pointer mode is the most honest default.** Lower strict-rate, but every "lost" STRICT vs quote mode was a false-positive caught by the hardening (chunk cap, coverage threshold, bare-name guard, lazy-anchor demote). The honest narrative beats the optimistic one in a verifier substrate.
|
||||
|
||||
3. **Quote mode's strict-rate is misleading.** The 47% includes false-STRICTs the pointer hardening exposed; those same patterns would demote to HYBRID under the pointer verifier's checks.
|
||||
|
||||
4. **JSON mode is the right answer for narrow-factoid hot paths.** If a deployment can detect "narrow factoid" question shape ahead of time, routing to JSON gives the cleanest perfect-grounding rate at 4.4s mean latency. For the rest, pointer is more robust.
|
||||
|
||||
5. **A fallback strategy makes JSON viable as default.** Try JSON first; on parse error, fall back to pointer mode with the same retrieval. Captures JSON's 5/5 narrow-factoid wins while bounding the worst case at pointer mode's 21% strict-rate / 48 grounded.
|
||||
|
||||
6. **Mode-collapse on broad descriptives is structural to Hermes-3-8B.** No prompt or policy tweak fully cures it. Larger models with better instruction-following (Hermes 70B class, Qwen 3.6 reasoner) would probably help. The chunk cap and bare-name guard make the failure honest rather than hidden.
|
||||
|
||||
## Roadmap — making solutions better
|
||||
|
||||
### Highest leverage
|
||||
|
||||
1. **JSON-fallback dispatch in the runtime.** When `answer_mode="claim_lattice"` returns a parse error / schema violation, retry once with `answer_mode="claim_lattice_pointer"` against the same retrieved context (no second LLM round trip needed if JSON failure is detected pre-call; otherwise one extra call). Result records both attempts in the run-DAG so the audit chain documents what happened. Estimated effort: small (one new dispatch path in `query.py`).
|
||||
2. **Stronger JSON prompt — or: relax `guided_json` for diverse shapes.** Investigate whether `guided_json: false` + lenient JSON parsing (which the codebase already supports) reduces the error rate below 29%. May trade some narrow-factoid wins for fewer errors.
|
||||
3. **Pointer narrow-factoid recovery.** Mona Lisa / python-date pointer flunks need root-cause diagnosis. Likely the bare-name guard is too aggressive on legitimate narrow claims that have only 2 content tokens after the spotlight stopword filter, or the coverage threshold is biting on multi-clause claims. Specific fix candidates:
|
||||
- Add a "narrow-factoid" detection (≤3 source chunks visible, question has a single clear interrogative) → relax coverage to 0.20 for these.
|
||||
- OR: move the bare-name guard threshold down to 1 (only catches truly empty claims like "Triceratops" with 0 tokens), and rely on coverage threshold alone for the JP-dinosaurs failure mode.
|
||||
|
||||
### Medium leverage
|
||||
|
||||
4. **Extend noisy markers to handle parenthetical disambiguation.** `"(NES game)"`, `"(arcade game)"`, `"(Sega adaptation)"` are common Wikipedia spinoff disambiguations. A regex `\((.*\bgame\b.*)\)` would catch all `(X game)` parens variants. Folds into `governance_policy_hash`.
|
||||
5. **Per-source chunk cap responsive to source role.** Primary answer source gets 4 chunks; background/noisy sources get 0–1. Already partly there via `SOURCE_ROLE_BUDGET_WEIGHTS` but the count cap is uniform. Combining char budget + chunk count + role weighting.
|
||||
6. **Comparison-question repair tier.** Detect "X vs Y" / "compare X and Y" / "difference between X and Y" patterns and retrieve sources for both X and Y before assembling context. Currently retrieval treats the full query as one keyword bag and rivalry-exclusion fights against this.
|
||||
|
||||
### Long-tail / experimental
|
||||
|
||||
7. **Larger model class for descriptives.** Route broad-descriptive questions to a 70B-class model (Hermes 70B, Qwen 3.6 reasoner) with `claim_lattice` JSON mode + grammar guidance. Honesty gains on connecticut/apollo would be measurable.
|
||||
8. **Question-shape classifier.** Detect narrow-factoid / broad-descriptive / list / comparison / niche before retrieval; route per-shape to optimal mode + chunk cap + repair tier. Expensive in code but each individual rule is cheap.
|
||||
9. **Verifier semantic check (soft signal).** A "did the claim's predicate match the cited span's frame?" signal that runs alongside the lexical checks but never enters the proof path. Could catch the JP-dinos "Triceratops in JP1 cited to Operation Genesis" failure (predicate mismatch). NER + relation extraction territory; substantial.
|
||||
|
||||
## Persistence
|
||||
|
||||
- Bench: `bench/qa_results/2026-04-30T17-05-28Z.{jsonl,md}` (3-mode merged not yet — quote+pointer in 17-05-28, JSON-only in a separate timestamp)
|
||||
- Code state at bench time: commit `873cfa8` (bare-name + smell-demote + game tie-in markers) on top of `c3da725` (JSON-mode runner wiring + partial-grounding split).
|
||||
- Endpoint: live `hermes.ai.unturf.com/v1` — Hermes-3 Llama-3.1-8B-FP8-Dynamic, no auth.
|
||||
- Reproduce: `make bench-qa` (default modes now = quote + pointer + JSON) or `make bench-qa BENCH_QA_MODES=claim_lattice` for JSON-only.
|
||||
|
|
@ -1,230 +0,0 @@
|
|||
# Self-reference design — recursive grounding on the providence ledger
|
||||
|
||||
**Date opened:** 2026-05-01
|
||||
**Status:** v1 (flat MVP) shipped — `aborist/sources/providence.py`, `make ingest-self-providence`, allowlist update for `claim_lattice_allowed_source_roles`. v2 (fact-Core distillation) — design proposal, implementation scoped to a follow-on commit pass.
|
||||
**Audience:** fox + future blackops shifts.
|
||||
**Hard constraint:** STRICT records are trusted as fact unless a verifier falsifies them. Other audit_modes stay opaque to retrieval until promoted. v2 extends the audit chain recursively without schema change.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Aborist's namesake is "tends trees and forests of cross-linked information." Today the system tends Wikipedia trees but never grafts its own past Q&A records into the forest. Every query starts from cold corpus retrieval; prior providence records sit in `providence_cache` unread. The system answers a question, stores the answer, and never looks at that answer again unless someone re-asks the exact same question (cache_key match).
|
||||
|
||||
The "kindergarten thought chains" framing names the gap: the system has a kindergarten of thoughts (early STRICT records) that should mature into citable substrate as they cool, then serve as anchors for new thoughts. Without that loop the substrate is a one-shot answerer, not a recursively-deepening reasoner.
|
||||
|
||||
Concretely: ask "what is verify_claim_lattice?" today and Hermes guesses from training. The right primary source — the verify.py source code or the design docs in `docs/` — isn't in the corpus. Even after `make ingest-self`, only the *code* gets ingested; the system's *answers about its own code* stay invisible to retrieval.
|
||||
|
||||
Two design layers solve this. **v1 flat MVP** lands the surface step (records become flat documents). **v2 fact-Core distillation** lands the deep step (records become Merkle-bound facts that compose).
|
||||
|
||||
---
|
||||
|
||||
## 2. v1 — flat MVP (shipped)
|
||||
|
||||
### 2.1 Architecture
|
||||
|
||||
A new `Source` subclass — `ProvidenceSource` — iterates `providence_cache` records and yields each as a `Document`:
|
||||
|
||||
- **URI**: `aborist://providence/<cache_key>` — content-addressed, stable, distinguishable from Wikipedia / external URIs at retrieval time.
|
||||
- **Title**: the question text (truncated to ~120 chars).
|
||||
- **Content**: a canonical layout of `Q: <question>` then `A: <answer_text>` then per-claim line `[E#: cite]` if available. The content gets chunked + Merkle-rooted via the standard ingest pipeline.
|
||||
- **source_type**: `"providence"`.
|
||||
|
||||
Records are filtered at iteration time by:
|
||||
|
||||
1. **`audit_mode == "STRICT"`** — only fully-grounded records become substrate. HYBRID and UNGROUNDED stay opaque to retrieval (noisy or speculative).
|
||||
2. **`falsification_state == "live"`** — falsified records (state ∈ {failed, stale, quarantined}) are excluded. The existing falsification machinery is the verifier-falsification mechanism: when `aborist providence --falsify` flips a record's state, it stops being substrate on next ingest.
|
||||
3. **`now - created_at >= kindergarten_seconds`** (default 3600s = 1h). Fresh thoughts cool before they're recyclable. Mirrors the mesh-sync kindergarten window. Without this, the system would self-cite a record from 30 seconds ago — echo-chamber loops.
|
||||
4. **Anti-recursion**: records whose own answer cited a `self_reference_source` are excluded — first-generation only. A wrong-but-STRICT record otherwise repeatedly recompiles into deeper claims and the chain rots silently.
|
||||
|
||||
### 2.2 Retrieval integration
|
||||
|
||||
The source-role classifier in `aborist/qa/query.py:_classify_source_role` recognizes the `aborist://providence/` URI scheme and tags those documents `self_reference_source`. This role gets added to the default `claim_lattice_allowed_source_roles` allowlist so claims can verify against self-reference spans. `SOURCE_ROLE_BUDGET_WEIGHTS` for `self_reference_source` = 1.0 (same as background; deliberately not boosted — Wikipedia is still the canonical primary).
|
||||
|
||||
### 2.3 Recursive Merkle proof
|
||||
|
||||
When Q2 cites Q1's answer span, the audit chain becomes "Q2 → Q1 → Wikipedia chunk." The original `chunk_root` remains the leaf; Q1's `run_dag_root` becomes an intermediate node. v9.8's admissibility ledger already supports this layering — the recursive-cores insight ("planet toward center compression" in CLAUDE.md). No schema change needed; the providence record's `merkle_proof` blob carries the parent chain.
|
||||
|
||||
### 2.4 Operational model
|
||||
|
||||
- `make ingest-self-providence` — promote STRICT-live providence records older than the kindergarten window into the document corpus.
|
||||
- Run on a cron (every hour, mirroring the mesh kindergarten window).
|
||||
- Idempotent: same record → same document_root → no-op insert. Replaced records get a `supersedes` edge linking new → old.
|
||||
|
||||
### 2.5 What v1 does NOT do
|
||||
|
||||
- **Aggregation** of multiple Q&A records into a synthesized "summary" record.
|
||||
- **Cross-shard self-reference**: each shard self-promotes within itself; cross-shard cites work via the existing `--shards-dir` UNION. No new code.
|
||||
- **Live retrieval from `providence_cache`** (Option B from the design discussion). The MVP uses snapshot ingestion (Option A) for simplicity.
|
||||
- **HYBRID-record self-promotion**. STRICT only.
|
||||
- **Compositional reasoning** — records become reachable, but they don't yet COMPOSE into new claims. v2 addresses the composition gap.
|
||||
|
||||
### 2.6 v1 risks (still open)
|
||||
|
||||
| risk | mitigation |
|
||||
|------|-----------|
|
||||
| Lazy-anchor false-STRICT compounds — Q2 inherits Q1's bogus cite. | `lazy_anchor_demoted` records skipped on promotion. NLI sidecar (`docs/verifier-semantic-gap-design.md`) adds another gate when it lands. |
|
||||
| Echo-chamber: same fact recycled across many records. | Kindergarten window (1h) + first-gen-only anti-recursion check. |
|
||||
| Falsified record stays in retrieval until re-ingest. | Verifier checks `falsification_state` of the cited source_root at verify time; non-live cites get rejected. Fail-closed. |
|
||||
| Storage bloat: every Q&A becomes a document. | Same chunker + Merkle as everything else; per-record cost is small. Periodic `burn-kindergarten` trims. |
|
||||
|
||||
---
|
||||
|
||||
## 3. v2 — fact-Core distillation (proposal)
|
||||
|
||||
### 3.1 Where v1 falls short
|
||||
|
||||
v1 makes records "another flat document source." The architecture wants them to be **Merkle-bound facts that compose into new claims**. v1 retrieves a record's text; it doesn't let the record's *evidence chain* attach as substrate for new reasoning. New claims about the same topic don't compose with old claims; they just see them as more context.
|
||||
|
||||
The deeper play: STRICT claims become **Cores** via the existing distillation pipeline. New claims derive from cores via the recursive-cores layer. The fact-graph grows.
|
||||
|
||||
### 3.2 The existing infrastructure already does most of this
|
||||
|
||||
The Surface → Core → Recursive-Core layering is in `aborist/distill/`:
|
||||
|
||||
```
|
||||
aborist/distill/
|
||||
├── base.py # Distiller ABC + DistillationResult
|
||||
├── first_sentence.py # FirstSentenceDistiller (no-ML stub)
|
||||
├── tfidf.py # TfidfKeywordDistiller (pure-Python TF-IDF)
|
||||
└── runner.py # batched: derive + per-contrib-chunk proofs
|
||||
```
|
||||
|
||||
Contract:
|
||||
|
||||
```python
|
||||
class Distiller(ABC):
|
||||
name: str
|
||||
def distill(self, source: Document, source_chunks: list[str]) -> DistillationResult:
|
||||
...
|
||||
```
|
||||
|
||||
The runner takes a surface Document, runs the Distiller, returns a Core Document plus `contributing_chunk_indices`. Per-chunk Merkle inclusion proofs against `document_root` get stored in `derivations.proof_blob` — Cores are **cryptographically bound** to their source surfaces.
|
||||
|
||||
What this gives us for free:
|
||||
- A Core is itself a Document with its own `document_root`, chunkable + retrievable like Wikipedia content.
|
||||
- The recursive-core mode (cores derive from cores) already exists in the runner.
|
||||
- Every Core carries explicit lineage back to its source via per-chunk proofs.
|
||||
|
||||
The missing piece: a `ProvidenceDistiller` that takes a STRICT providence record and produces a Core, where the "source chunks" are the cited evidence spans the record verified against.
|
||||
|
||||
### 3.3 ProvidenceDistiller — cited-evidence-bound (Option B)
|
||||
|
||||
The Distiller contract today maps `(surface Document, surface chunks) → core Document`. For self-reference we want `(STRICT record, cited evidence spans from OTHER documents) → fact-Core`. The "source chunks" the Core derives from aren't the providence record's own chunks — they're the EVIDENCE SPANS the record cited in its `claim_statuses`.
|
||||
|
||||
Two options were considered:
|
||||
|
||||
- **Option A — distill from the providence record's own content.** ProvidenceDistiller treats the record's `Q: ... A: ...` text as the surface, distills it to a Core. Simple, fits the Distiller contract directly.
|
||||
- **Option B — distill from the cited evidence spans, with the record as an indirection.** ProvidenceDistiller looks up the record's `claim_statuses[].evidence_ids`, fetches the cited evidence chunks from THEIR source documents, treats those as the "source chunks," and emits a Core that's bound by inclusion proof to the cited chunks of the cited Wikipedia documents.
|
||||
|
||||
**Option B is the right deep version.** A fact-Core derived from a STRICT claim is a Merkle-bound assertion that "claim text C is supported by chunk_root Cr1 in document_root Dr1." Future claims attaching to this Core inherit that evidence chain transparently. Option A would make Cores derive from the record's text (which already says what the answer is), losing the direct connection to the underlying Wikipedia facts.
|
||||
|
||||
Option B's cross-document fetch is supported today: `--shards-dir` UNION views let a single `connect()` see all shards as one read connection.
|
||||
|
||||
### 3.4 Core content shape
|
||||
|
||||
Three candidate shapes for the fact-Core's content:
|
||||
|
||||
```text
|
||||
SHAPE A — claim text only
|
||||
"Joey Potter is the girl across the creek in Dawson's Creek."
|
||||
|
||||
SHAPE B — claim + per-source pointer
|
||||
"Joey Potter is the girl across the creek in Dawson's Creek."
|
||||
[Dawson Leery (Wikipedia): "...the central fictional character..."]
|
||||
|
||||
SHAPE C — structured triple
|
||||
SUBJECT: Joey Potter
|
||||
PREDICATE: is the girl across the creek
|
||||
OBJECT: in Dawson's Creek
|
||||
EVIDENCE: chunk_root=ab12... offset_start=4032 offset_end=4189
|
||||
```
|
||||
|
||||
**Recommendation: Shape B for MVP.** Pure prose with a tagged citation. Retrieval finds the prose; the cited span is right there. Shape C lands later if a fact-graph traversal becomes a real need (NER + relation extraction — out of scope today).
|
||||
|
||||
### 3.5 Recursive cores — facts grow new ideas
|
||||
|
||||
```text
|
||||
Surface (Wikipedia chunk)
|
||||
↓ TfidfKeywordDistiller
|
||||
Core-tfidf (keywords from Wikipedia chunk)
|
||||
|
||||
STRICT providence record
|
||||
↓ ProvidenceDistiller (Option B — bound to cited Wikipedia chunks)
|
||||
Fact-Core
|
||||
|
||||
[Future] N related Fact-Cores
|
||||
↓ ?CompositionDistiller (deferred)
|
||||
Composite-Fact-Core (claims combining multiple facts)
|
||||
```
|
||||
|
||||
The v2 MVP does only the first ProvidenceDistiller pass. **CompositionDistiller is the future shape that makes facts compose into new ideas — that's where "the substrate forms new claims from its own facts" lives.** CompositionDistiller is hard because deciding which facts to compose, and how, is the actual reasoning step. Today's Hermes doesn't do that reliably. Defer.
|
||||
|
||||
What v2 ships: each STRICT claim becomes a Merkle-bound Fact-Core whose proof chain reaches all the way back to a Wikipedia chunk_root. Retrieval over Cores returns Fact-Cores alongside Wikipedia surfaces — the lattice grows. Composition is deferred but the substrate is in shape for it when we land it.
|
||||
|
||||
### 3.6 Recursive Merkle proof (the part already free)
|
||||
|
||||
When a future Q3 cites a Fact-Core that derived from a STRICT Q1 record citing Wikipedia chunk Cr1:
|
||||
|
||||
```
|
||||
Q3 claim → Fact-Core → Cr1 → Dr1 → Sr1
|
||||
```
|
||||
|
||||
Each link is a Merkle inclusion proof or a content-addressed lookup. No new schema is needed — `derivations.proof_blob` already holds the per-chunk inclusion proofs; the per-claim → Fact-Core → derivation walk just composes existing primitives. v9.8's audit chain extends naturally; we don't need v9.9.
|
||||
|
||||
### 3.7 Trust + falsification (sharper than v1)
|
||||
|
||||
- A Fact-Core is created only from a STRICT-live providence record past the kindergarten window.
|
||||
- If the record is later falsified (`falsification_state != live`), the Fact-Core's `derivations` row is marked stale on next promotion run. Idempotent: same record → same Core hash. Falsified records don't promote.
|
||||
- A future Q3 citing a stale Fact-Core fails verification at the source-state check (verifier checks `falsification_state` of the cited source's underlying records, not just the surface document).
|
||||
- **Fail-closed: a falsified Fact-Core CANNOT serve as substrate even if it's still in the documents table.**
|
||||
|
||||
The key trust-model add over v1: **falsification cascades**. Falsifying Q1 stales Q1's Fact-Core, which stales Q2 if Q2 had cited the Fact-Core. The Merkle chain makes the cascade traceable.
|
||||
|
||||
### 3.8 Anti-recursion (kept from v1)
|
||||
|
||||
A providence record whose own answer text already cites a Fact-Core — i.e. a record answered by composing existing facts — gets ONE level of self-reference but cannot itself be promoted to a NEW Fact-Core. First-generation only. This kills echo-chamber chains where a wrong-but-STRICT record keeps recompiling itself into deeper claims. Conservative; the right relaxation is "promote when the lazy-anchor sidecar AND the NLI sidecar both pass" — but that's after both signals are in place.
|
||||
|
||||
---
|
||||
|
||||
## 4. v2 implementation plan (8 steps)
|
||||
|
||||
1. **`aborist/distill/providence.py`** — new module, `ProvidenceDistiller(Distiller)`. Reads STRICT live providence records past kindergarten, fetches the cited evidence chunks (Option B), builds Shape-B Core content (claim text + tagged citation span), returns `DistillationResult` with `contributing_chunk_indices` pointing to the cited Wikipedia chunks.
|
||||
2. **Wire into `aborist/distill/runner.py`** — register ProvidenceDistiller as a known kind. The existing batched-distill flow handles the per-chunk-proof generation transparently.
|
||||
3. **CLI: `aborist distill --kind providence`** — adds the new kind to the `distill` subcommand's choices. Plumbs the `--kindergarten-seconds` knob from the v1 CLI work.
|
||||
4. **Makefile: `distill-self-providence`** — runs `aborist distill --kind providence` against each shard. Hourly cron candidate.
|
||||
5. **Source-role classifier** — Fact-Cores are tagged `self_reference_source` via the existing URI-prefix path (`aborist://providence/...` from v1 carries through). Cores derived from those records inherit the role. No classifier change needed.
|
||||
6. **Falsification cascade** — when `aborist providence --falsify` flips a record's state, also mark the corresponding Fact-Core's derivation row as stale. New CLI flag or implicit on next ingest pass; tradeoff: explicit is debuggable, implicit is less coordinated.
|
||||
7. **Tests** — unit tests for ProvidenceDistiller (correctly fetches cited chunks, builds Shape-B content, generates valid inclusion proofs), falsification cascade (falsified record → stale Core → rejected citation in new run).
|
||||
8. **Bench validation** — re-run `make bench-qa` after `make ingest-self-providence` AND `make distill-self-providence` have populated some Fact-Cores. Compare to baseline. Questions about aborist itself (currently UNGROUNDED) should ground; questions tangential to past STRICT answers should gain new anchors.
|
||||
|
||||
---
|
||||
|
||||
## 5. What's deliberately NOT in this design
|
||||
|
||||
- **CompositionDistiller** — combining multiple Fact-Cores into a new claim. Reasoning machinery, not infrastructure. Defer until the soft-signal taxonomy (NLI, predicate compatibility) is mature enough that compositions can be sanity-checked.
|
||||
- **Shape-C structured triples** — needs NER + relation extraction. Land Shape B first; promote to Shape C if a fact-graph use case actually needs subject-predicate-object retrieval.
|
||||
- **HYBRID record promotion** — both v1 and v2 gate on STRICT only. HYBRID could become a `self_reference_hybrid_source` role with lower trust, separately gated.
|
||||
- **Cross-shard cascading falsification** — falsifying a record on one shard doesn't auto-falsify a Fact-Core derived from it on another shard. Mesh-sync handles cross-shard coherence eventually; the immediate cascade is per-shard. Acceptable.
|
||||
|
||||
---
|
||||
|
||||
## 6. Bench impact (speculative, disciplined)
|
||||
|
||||
After a few hundred STRICT records have promoted to Fact-Cores:
|
||||
|
||||
- Questions about aborist itself (today's mostly UNGROUNDED) start grounding against Fact-Cores derived from past Q&A about aborist.
|
||||
- Questions tangentially related to past STRICT answers gain anchors that reach back to Wikipedia transparently.
|
||||
- Strict-rate creeps up as the fact-substrate matures; honest-grounded count rises faster.
|
||||
- New failure modes: bad anchors landing inside Fact-Cores. The lazy-anchor sidecar already covers this layer-recursively because Fact-Cores look just like other documents to the verifier.
|
||||
- Latency: same as Wikipedia retrieval. No new path; just more documents indexed.
|
||||
|
||||
---
|
||||
|
||||
## 7. The architectural payoff
|
||||
|
||||
Today the substrate is a one-shot answerer: every query starts cold, retrieves Wikipedia, prompts Hermes, verifies, caches. The cache is a key-value lookup, not a substrate for reasoning.
|
||||
|
||||
After v2 lands, the substrate becomes recursively-deepening: every STRICT answer becomes a Merkle-bound fact in the tree. New questions retrieve old facts as substrate. The fact-graph compounds. Wrong facts get falsified and the cascade reaches the dependent records. Right facts stay grounded and become the foundation for deeper claims.
|
||||
|
||||
That's "tends trees and forests of cross-linked information" — literally. The naming wasn't aspirational; it was load-bearing for the architecture.
|
||||
|
|
@ -1,163 +0,0 @@
|
|||
# Closing the verifier's lazy-anchor semantic gap
|
||||
|
||||
**Date:** 2026-04-30
|
||||
**Scope:** Design proposal for adding a soft semantic-entailment signal alongside the existing lexical claim-lattice verifier in `aborist/qa/verify.py`. Doc-only — no code in this commit. Successor to roadmap item #9 in `docs/qa-modes-bench-2026-04-30.md`.
|
||||
**Audience:** fox + future blackops shifts.
|
||||
**Hard constraint:** the soft signal is a **demote-only sidecar**. It never enters the proof path.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Today's `verify_claim_lattice` and `verify_claim_lattice_json` in `aborist/qa/verify.py` perform six deterministic, lexical checks on every (claim, pointer) pair. Check #5 is the topic of this proposal:
|
||||
|
||||
> Claim's content tokens textually overlap the cited evidence span at coverage ≥ `min_citation_coverage` (default 0.30, lexical, case-insensitive substring).
|
||||
|
||||
This catches the **shape** of a citation mismatch but not its **predicate**. A claim and a span can share enough surface tokens to clear 30% coverage while the span never asserts the claim's relation. The model used training-data knowledge to write the claim, picked the topically-closest pointer the runtime offered, and the verifier said STRICT.
|
||||
|
||||
This is the long-tail "verifier semantic check (soft signal)" item from the bench journey doc:
|
||||
> "did the claim's predicate match the cited span's frame?"
|
||||
|
||||
### 1.1 Concrete failure cases
|
||||
|
||||
**Case A — Great Wall elevation** (surfaced live 2026-04-30 evening).
|
||||
|
||||
Question: *"which side is the ground elevation highest throughout the span of the great wall of china, the north or south?"*
|
||||
Mode: `claim_lattice_pointer`
|
||||
Verdict: `STRICT`, `n_quotes=2`, `n_verified=2`, `lazy_anchor_ratio=0.5`.
|
||||
Claim: *"The ground elevation is higher on the northern side of the Great Wall of China."*
|
||||
Cited span (E13): "...regions have traditionally been referred to as 'Outer China' because they are located beyond the Great Wall of China. ... China is bordered in the north, west and so..."
|
||||
Content tokens in claim ≈ {`ground`, `elevation`, `higher`, `northern`, `side`, `great`, `wall`, `china`}; tokens present in span ≈ {`northern` (stem-flexed), `great`, `wall`, `china`} → 4/8 = 50% coverage, well above the 0.30 floor. **The cited span carries no elevation assertion at all.** The model used training-time topology knowledge of the Wall and grabbed the topically-closest "north + Great Wall" chunk. The verifier had no way to detect that the predicate `is_higher_than(north_side, south_side)` is missing from the span.
|
||||
|
||||
**Case B — JP-dinos / Triceratops & Operation Genesis** (CLAUDE.md retrieval-pipeline & doc F5).
|
||||
|
||||
Question: *"what dinosaurs were in the first jurassic park film?"*
|
||||
Mode: `claim_lattice` (JSON).
|
||||
Several "verified" claims cite spans from `Jurassic Park: Operation Genesis` (a 2003 video game) rather than the 1993 film. The cited span genuinely contains `Triceratops` (the game indexed every dinosaur) and the claim text says "Triceratops appears in Jurassic Park" — coverage is 100%. Lexically perfect; semantically mis-rooted. The dinosaur token's the same; the claim's relation is `appears_in(film=JP1, species=Triceratops)`; the span's relation is `appears_in(video_game=Operation_Genesis, species=Triceratops)`. The current verifier cannot tell.
|
||||
|
||||
**Case C — Boltzmann constant value.**
|
||||
|
||||
Question: *"what is the boltzmann constant?"*
|
||||
Mode: `claim_lattice_pointer`.
|
||||
Inspected one STRICT run; the model emitted *"The Boltzmann constant is approximately 1.380649×10^−23 joules per kelvin"* cited to a chunk that mentions the constant by name and the unit "joule" but **never states the numerical value**. Coverage clears 0.30 because `boltzmann`, `constant`, `joule`, `kelvin` all appear; the value `1.380649×10^−23` does not. Same shape as case A: factually correct claim, partially-grounded citation, but the load-bearing predicate (the *number*) was emitted from training, not the span.
|
||||
|
||||
### 1.2 What these cases share
|
||||
|
||||
Across all three: claim and span talk about the **same topic**, so token overlap is high. The claim's **load-bearing predicate** (a relation, a number, a comparison) is **absent from the span**. This is the canonical NLI "non-entailment" pattern. Lexical coverage is the wrong instrument; entailment is the right one. We need to add an entailment-shaped check **without compromising the determinism, content-addressability, or proof-path purity that v9.8 admissibility depends on.**
|
||||
|
||||
The `lazy_anchor_demote` sidecar in `verify_claim_lattice` already proves the architectural pattern works: a soft signal computed at verify time, surfaced in the verdict for the renderer, and used to **cap** the audit_mode at HYBRID — never to *invent* STRICT. We extend that pattern.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architectural constraints
|
||||
|
||||
Any design must hold every one of these. Violations are not negotiable.
|
||||
|
||||
1. **Soft signals never enter the proof path.** Per CLAUDE.md "Soft hash vs hard hash": SHA-256 is hard; embeddings/TF-IDF/entailment-scores are soft. The new score must NOT thread into `cache_key`, `run_dag_root`, the `audit_events` chain, or the canonical-JSON inputs of `build_run_dag`'s `verify_payload`. Pattern to follow: existing `pointer_id_distribution` / `lazy_anchor_ratio` / `lazy_anchor_demoted` fields in the verdict — surfaced for the renderer, deliberately not folded into `verify_hash`.
|
||||
2. **Determinism: same inputs → same verdict.** No PRNGs leaking through. ONNX/PyTorch with `torch.use_deterministic_algorithms(True)`, fp32, `model.eval()`. Every model load from a pinned weights hash (sha256 of safetensors), with the hash logged but not in the proof path. Output thresholded to a boolean — same shape as `lazy_anchor_demoted: bool`.
|
||||
3. **Hermes is the only allowed external endpoint.** `https://hermes.ai.unturf.com/v1`. No HuggingFace inference API, no OpenAI, no third-party hosted endpoints. Local model files are fine and preferred (CPU-runnable, sub-200M params).
|
||||
4. **Backward compatibility.** Records under the current `governance_policy_hash` keep their verdicts. The new check is gated behind a new policy field (`claim_lattice_semantic_check`) with default `false`; flipping it on bumps the policy hash so new writes go under a fresh cache_key but old cached records still resolve under the old hash. Same migration pattern as the entity-policy and atomic-claim rollouts.
|
||||
5. **Demote-only.** The semantic check may move STRICT → HYBRID. It may NEVER move HYBRID → STRICT, UNGROUNDED → HYBRID, or invent grounding the lexical path didn't find. STRICT remains earned, never inferred.
|
||||
6. **Latency budget.** Current `claim_lattice` mean ~4.6s, `claim_lattice_pointer` ~4.4s. The semantic check must add **< 1s typical, < 3s worst case** for an answer with ≤ 12 claims.
|
||||
7. **Optional dependency.** Same shape as `mwparserfromhell`: an extras group like `aborist[semantic]`. Installs without it set `claim_lattice_semantic_check = False` automatically; verdicts revert to today's behavior.
|
||||
|
||||
---
|
||||
|
||||
## 3. Candidate designs
|
||||
|
||||
Three options, each evaluated against constraints and the §1.1 cases.
|
||||
|
||||
### 3a. Lightweight NLI sidecar (cross-encoder)
|
||||
|
||||
A small NLI cross-encoder (`cross-encoder/nli-MiniLM2-L6-H768` ≈ 80M, `cross-encoder/nli-deberta-v3-small` ≈ 184M, or `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` ≈ 184M) loaded once at import time. Per-claim signature: `score(premise=evidence_span, hypothesis=claim_text) → {entailment, neutral, contradiction}`. Demote STRICT → HYBRID when **all** cited spans for a claim score `entailment_prob < threshold`.
|
||||
|
||||
- **Accuracy on §1.1.** High. NLI cross-encoders trained on MNLI+ANLI+FEVER reliably mark "Outer China beyond the Great Wall" / "north side higher" as **neutral**, and "Triceratops in JP1" / "Triceratops in Operation Genesis" as **neutral or contradiction**. Boltzmann numerical-value entailment is a known weak spot for small NLI models but the directional signal is right ("constant has unit joule per kelvin" entails "exists" but not "equals 1.380649e-23"); demotes correctly often enough.
|
||||
- **Latency.** 80–184M-param cross-encoder on CPU, sequence length capped to ~512 tokens: ~30–80 ms per (claim, span) pair. With max 12 claims × 2 pointers/claim = 24 inferences, worst case ~1.9 s. Inside the < 3s budget. Typical (4–6 claims, 1 pointer each) is < 0.5 s.
|
||||
- **Dependency cost.** New extra: `aborist[semantic]` adds `transformers` + `torch-cpu` + `sentencepiece` ≈ 250 MB install. Weights 70–700 MB; the smaller MiniLM variant fits in ~100 MB. Ship pinned by sha256.
|
||||
- **Determinism.** fp32 with `torch.use_deterministic_algorithms(True)` and `model.eval()` is bit-exact across runs on the same hardware. Cross-hardware drift exists at the LSB; thresholding at 0.5/0.7 absorbs it. Same kind of "deterministic up to thresholding" the lazy-anchor demote already lives with.
|
||||
- **Integration.** Drop-in extension of the verdict-construction tail of `verify_claim_lattice` and `verify_claim_lattice_json`. New helper `_score_semantic_entailment(claim_text, evidence_span) -> (label, prob)` gated on `policy["claim_lattice_semantic_check"]`. New verdict fields: `semantic_entailment_scores` (per-claim, render-layer only), `semantic_demoted: bool`, `semantic_violations`. The bool feeds the same demote pattern as `lazy_anchor_demoted`.
|
||||
|
||||
### 3b. TF-IDF predicate matching
|
||||
|
||||
Pure-Python heuristic: extract a (subject, predicate, object) triple from each claim using hand-rolled rules; same extraction on the cited span's nearest sentence to the claim's spotlight token. Score predicate similarity by cosine over TF-IDF vectors weighted toward verb/relation tokens; demote on score below threshold.
|
||||
|
||||
- **Accuracy on §1.1.** Mixed. Great Wall *probably caught* (claim's predicate is "is_higher_than" — the span has no elevation/comparison verb). Triceratops/Operation Genesis *probably missed* (both sides have "appears" — discriminator is the *subject*: film vs game). Boltzmann numerical *fails fully* (no verb-level distinction; missing element is a number). Catches maybe 1 of 3 named cases.
|
||||
- **Latency.** Negligible.
|
||||
- **Dependency cost.** Zero (`aborist/distill/tfidf.py` already implements pure-Python TF-IDF; reuse).
|
||||
- **Determinism.** Trivially deterministic.
|
||||
- **Integration.** Same hook point. Cheap to implement, but high false-negative rate on cases B & C, and the design pattern doesn't scale — every new failure shape needs new heuristics.
|
||||
|
||||
### 3c. Per-claim re-prompt to Hermes
|
||||
|
||||
Second LLM call per claim: "Does the following span support the following claim? Answer yes or no." `temperature=0`, `max_tokens=4`, single-token response.
|
||||
|
||||
- **Accuracy on §1.1.** Plausibly high. Hermes-3-8B is competent at narrow single-claim entailment. But: the same model that wrote the lazy-anchored claim is being asked to grade it. Self-evaluation bias is documented — confirms its own outputs.
|
||||
- **Latency.** Doubles the LLM-call count. 12 claims = 12 extra round trips. Even with 4-way parallelism, ~1.5–3s, right at the ceiling. Also ties verifier latency to vLLM availability — when JSON-mode 5xx clusters hit (the 19-error morning), the semantic check would have *also* failed.
|
||||
- **Dependency cost.** Zero new code dependencies; operational coupling to Hermes worsens.
|
||||
- **Determinism.** `temperature=0` necessary but not sufficient — vLLM batch-position effects + server-side prefix-cache can change tokenization. Reproducibility across days not guaranteed.
|
||||
- **Integration.** Same hook point. Calls `aborist.qa.client.ChatClient.complete` per claim.
|
||||
|
||||
### 3d. Comparison summary
|
||||
|
||||
| design | catches Wall | catches Tri/OpGenesis | catches Boltzmann | latency added | det. | dep. cost |
|
||||
|--------|--------------|-----------------------|-------------------|---------------|------|-----------|
|
||||
| 3a NLI cross-encoder (MiniLM-class) | yes | yes | weak-yes | 0.5s typ / 1.9s worst | high | +250MB install, +100MB weights |
|
||||
| 3b TF-IDF predicate | maybe | no | no | negligible | full | none |
|
||||
| 3c Re-prompt Hermes | yes | yes | yes | 1.5–8s | medium | none, ops coupling |
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation: 3a (NLI cross-encoder), default off, opt-in by policy
|
||||
|
||||
3a wins on accuracy across all three named failure cases and stays inside the latency budget. 3b's miss rate on JP-dinos and Boltzmann is too high to justify even at zero cost — those are the cases that motivated this work. 3c's self-evaluation bias and operational coupling outweigh its zero-install cost.
|
||||
|
||||
Chosen model: **`cross-encoder/nli-MiniLM2-L6-H768`** (≈80M params, ~100MB weights). Smallest CPU-runnable cross-encoder with serviceable MNLI/ANLI accuracy. Pinned by sha256 of the safetensors file. Threshold default `entailment_prob < 0.50` triggers demote — tuned via §5.
|
||||
|
||||
### 4.1 Implementation plan (high-level, 8 steps)
|
||||
|
||||
1. **Add optional dep.** `aborist[semantic]` extras group in `pyproject.toml` pulling `transformers>=4.40`, `torch>=2.2 --extra-index-url cpu`, `sentencepiece`. Weights distributed out-of-band (pinned-revision download with sha256 verification at first load); cached under `~/.aborist/models/<sha>/`.
|
||||
2. **New module `aborist/qa/semantic_check.py`.** Single public function `score_entailment(premise: str, hypothesis: str) -> tuple[str, float]` returning (label, entail_prob). Module-level lazy-loaded model object. `import semantic_check` is cheap; first call pays the ~2s load. Soft-fails to `(None, None)` if `transformers` is not installed.
|
||||
3. **Policy fields.** Add to `DEFAULT_POLICY` and `DEFAULT_QUERY_POLICY`:
|
||||
- `claim_lattice_semantic_check: bool = False`
|
||||
- `claim_lattice_semantic_threshold: float = 0.50`
|
||||
- `claim_lattice_semantic_model: str = "cross-encoder/nli-MiniLM2-L6-H768"`
|
||||
- `claim_lattice_semantic_model_sha256: str = "<pinned hex>"`
|
||||
These fold into `governance_policy_hash` automatically.
|
||||
4. **Wire into `verify_claim_lattice` and `verify_claim_lattice_json`.** After the lexical six-check loop completes and `audit_mode` is determined, but before `lazy_anchor_demoted` is computed: if policy bool is set, run `score_entailment` over each verified (claim_text, evidence_span) pair. Aggregate per-claim: a claim is "semantically supported" iff at least one of its pointers entails. If `audit_mode == "STRICT"` and ANY claim fails semantic support, demote to `HYBRID` and append a `SEMANTIC_NON_ENTAILMENT` violation.
|
||||
5. **New verdict fields, render-layer only.** `semantic_entailment_scores: list[dict]` (per claim: `{claim_idx, pointer_id, label, prob}`), `semantic_demoted: bool`, `semantic_violations`. None get folded into `build_run_dag`'s `verify_payload` — same architectural choice as `pointer_id_distribution` / `lazy_anchor_ratio`. The `violations` list (which IS in the verify payload) carries only the kind tag (`SEMANTIC_NON_ENTAILMENT`) and the count, not the per-claim scores. Demote stays operator-visible in the audit chain without folding the soft probability values themselves into the hash.
|
||||
6. **Renderer surfaces the smell.** When `semantic_demoted == True`, the human renderer prepends `[non-entailment smell — N of M claims show low entailment]` to the answer prefix, identical pattern to today's lazy-anchor smell prefix.
|
||||
7. **Tests.** Three live fixtures matching §1.1 cases: Great Wall elevation must demote, Boltzmann constant numerical claim must demote, JP-dinos Operation-Genesis citation must demote. Plus a positive-control: "who painted the mona lisa" must NOT demote. Add to `tests/test_qa_quality_live.py` gated on `semantic_check=true`.
|
||||
8. **Bench impact, before/after.** Run `make bench-qa` with `claim_lattice_semantic_check=true` and `=false` on the same question set. Compare strict-rate, grounded count, mean latency. Land the policy default at `false` with the bench numbers in the commit message; defer flipping the default to a separate doc + commit once we've seen the bench delta.
|
||||
|
||||
---
|
||||
|
||||
## 5. Bench impact estimate (speculative but disciplined)
|
||||
|
||||
Sampling current bench data at `bench/qa_results/2026-04-30T21-35-04Z.jsonl`:
|
||||
|
||||
- 75 `claim_lattice_pointer` rows total, ~16 STRICT in the post-stop-sequence run.
|
||||
- 75 `claim_lattice` (JSON) rows total, ~37 STRICT.
|
||||
- Across both lattice modes, an estimated **5–10 STRICT verdicts per 22-question bench (n=3)** look like §1.1-shaped lexical-pass / semantic-fail cases.
|
||||
- Expected demote: **~5–8 of the current ~53 STRICT verdicts move to HYBRID.** That's roughly a **5–10pp drop in strict-rate**, with **zero drop in grounded count**.
|
||||
- Mean latency cost: +0.4s typical (4–6 claims × 1–2 pointers each × ~50ms per scoring call), +1.5–2s on broad-descriptive 12-claim tail.
|
||||
|
||||
This matches the **honest-verdicts-beat-optimistic-ones** principle in CLAUDE.md: each demoted verdict is one false-positive STRICT removed from the audit ledger. The strict-rate goes down; the substrate's claims about itself become more truthful.
|
||||
|
||||
Quote-mode is **out of scope** for this design (it's a separate verifier path with its own verbatim-substring contract). If the quote-mode false-STRICT problem (failure mode F4 in the bench journey) needs addressing later, the same NLI sidecar could be wired into `verify_quotes` post-classification, but that's a separate design.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions for fox
|
||||
|
||||
1. **Model choice.** Default to `cross-encoder/nli-MiniLM2-L6-H768` (~80M, fastest)? Or step up to `cross-encoder/nli-deberta-v3-small` (~184M, more accurate, ~2× slower)? Recommendation: smaller for now; revisit if accuracy disappoints.
|
||||
2. **Threshold tuning policy.** Default `entailment_prob < 0.50` triggers demote. Hardcoded constant (folds into source identity) or `policy[...]` field (folds into governance hash, lets per-deployment tuning bump cache-key cleanly)? Recommendation: policy field.
|
||||
3. **Persistence of the soft signal.** `lazy_anchor_ratio` is render-layer only — never written to `providence_cache`. Should `semantic_entailment_scores` follow same pattern, or should the `semantic_demoted` boolean flag be persisted as a new column on `providence_cache` so an operator querying old records can tell why a HYBRID is HYBRID? Recommendation: bool stays in `violations` (already persisted via `run_dag_blob`), per-claim prob array does NOT persist. Easy migration, no schema bump.
|
||||
4. **Default-on vs default-off.** Plan above is default-off, opt-in via policy. Alternative is default-on, which bumps everyone's `governance_policy_hash` and stales every record on next lookup. Recommendation: default-off until two clean benches under the new path confirm no regression on positive controls.
|
||||
5. **Hardware drift.** If a deployment swaps CPU vendors (Intel BLAS vs AMD vs ARM), float arithmetic differs at the LSB, occasionally crossing the 0.50 threshold for marginal cases. Proof path doesn't care (soft signal), but two operators looking at the same record could see different `semantic_demoted` bits if their machines disagree. Acceptable? Recommendation: yes — soft signals are by definition not bit-stable across machines, and the v9.8 audit chain only commits to inputs that ARE bit-stable. Worth fox's explicit blessing.
|
||||
6. **JP-dinos Operation-Genesis case overlap with retrieval-side fixes.** The JP-dinos case is *also* on the retrieval-side roadmap (extending noisy markers to catch "(NES game)"-style parens). If retrieval-side filtering lands first, the verifier-side semantic check would still catch any *other* topic-aliased citation that retrieval didn't filter. Complementary, not duplicative. Worth doing both?
|
||||
|
||||
---
|
||||
|
||||
## 100-word summary
|
||||
|
||||
The claim-lattice verifier passes any (claim, span) pair with ≥30% lexical token overlap, but topical overlap can mask predicate mismatch. Three live cases — Great Wall elevation, JP-dinos Triceratops/Operation-Genesis, Boltzmann constant value — landed STRICT despite the cited span never asserting the claim's load-bearing relation. Recommended fix: a small NLI cross-encoder (~80M params, CPU-runnable, deterministic at fp32) computing entailment per (claim, span), gated behind a default-off policy field. Soft signal demotes STRICT→HYBRID only — never inverts UNGROUNDED, never enters the proof path, mirrors today's `lazy_anchor_demoted` pattern. Estimated ~5–8 STRICT demotes per 66-run bench, +0.4s typical latency.
|
||||
Loading…
Add table
Add a link
Reference in a new issue