docs(#000058,#000059,#000060): ticketize Dav1dPrometheus protocol-layer report
Three tickets from the 2026-05-20 Dav1dPrometheus "Protocol-Layer AGI" working report (held outside the repo; referenced not committed): - #000058 cache_key_9 verifier-policy mandatory-vs-legible decision + doc reconcile. Records the five-step-#1 correction: verifier fields already fold into governance_policy_hash, so the 9th dim is audit legibility not a correctness gap. - #000059 admission discipline: claim-graveyard burden-shift + self-providence quarantine (guards the existing ingest-self-providence self-confirmation loop). - #000060 H-ABCDEFG same-model substrate-delta harness (jaggedness + curvature); curvature-aware ForkScore folded into #000012 Phase 2, not spawned as a sibling. Reconciles CLAUDE.md cache_key invariant (8-dim -> 8 + optional 9th). Next ID 000058 -> 000061.
This commit is contained in:
parent
c6f8e991f8
commit
d4480cddb7
6 changed files with 385 additions and 5 deletions
|
|
@ -111,6 +111,9 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000060 | H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + curvature) | open · awaiting go/no-go (2026-05-20; from Dav1dPrometheus *Protocol-Layer AGI* working report §26/§50/§82-84). The report's "decisive proof": run the SAME base model substrate-OFF vs substrate-ON over long-horizon/adversarial/non-jagged batteries, report the delta. Two new metrics: jaggedness tensor `J_norm` (§73 — variance across nearby variants, normalized by difficulty) + discrete performance curvature `κ_t` (§5.2, with the honest no-global-convexity bound, Erratum 5). A-vs-C spine (B optional, D=mesh OUT → #000012/#000016). Curvature-aware ForkScore extension folds into **#000012** (NOT a new ticket — reserved `iota`/`kappa` weight slots already exist). Budget: control arms = Hermes/Qwen, never Opus without go; heavy passes on GPU box. Held-out/mechanism-agnostic variants required so ABCDEFG doesn't self-validate. | 2026-05-20 | — |
|
||||
| #000059 | Admission discipline: claim-graveyard query + self-providence quarantine | open · awaiting go/no-go (2026-05-20; Dav1dPrometheus report §11 Priority 2 + §57 "admissible state transition" thesis). Two coupled write-path mechanisms. **(A) GraveyardCheck:** storage already exists (`falsification_state` failed/stale/quarantined records ARE the graveyard); the gap is burden-shifting — a re-asked claim family with a known `failed` history should require stronger evidence (§42.9/§62-63). **(B) Self-providence quarantine:** `make ingest-self-providence` (Makefile:769) deliberately promotes STRICT records into the corpus — the exact self-confirmation loop §70 warns of — and ships with NO guard; detect/lineage-tag self-providence-descended evidence + quarantine for high-impact claims. Both advisory-sidecar-first (run-DAG only, never `providence_cache`/`audit_events`); demote-hooks bench-gated + unwired pending net win (would fold `governance_policy_hash`). Soft signals never enter the hard proof path. | 2026-05-20 | — |
|
||||
| #000058 | `cache_key_9` verifier-policy: mandatory-vs-legible decision + doc reconciliation | open · awaiting go/no-go (2026-05-20; Dav1dPrometheus report §2 Erratum 1 / §11 Priority 1 "mandatory cache_key_9"). **Five-step #1 correction:** the report's *correctness* premise is already false in arborist — verifier fields are a subset of the policy dict and so already fold into `governance_policy_hash` (`keys.py:269-275`); a verifier-rule change ALREADY changes the cache_key today. The explicit 9th `verifier_policy_hash` buys **audit legibility**, not correctness — so "mandatory" would stale every prior record for zero correctness gain. Decision: A leave-as-is (8-dim default, 9th optional) + doc reconcile [recommended] · B default-write 9-dim · C flag-staged bench-gated default-write — never a hard mandatory flip. Doc reconcile (CLAUDE.md/concepts "8-dim" → "8 + optional legible 9th") is the do-regardless. | 2026-05-20 | — |
|
||||
| #000057 | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). **Sharpened by fox 2026-05-19 (§4b/§5):** the review's strongest point — every measurement so far is *within* Arborist, no control arm. v1 (smallest proof) = single-shot **Hermes-solo vs Arborist** on a fabrication-bait fixture; headline = fabrication-vs-honest-abstention delta (NOT raw accuracy — common-knowledge would falsely show null via parametric memory). **Judge correction (fox):** "no LLM-as-judge" is a *proof-path* rule (LLM judge inside Arborist's verifier = the circularity it kills); the experiment's judge is **external science** outside *both* arms → use a **strong SOTA model**, blinded + reference-grounded + transcript-logged. That lifts the atomic-only limit (judge handles essay grounding too — #000049's exclusion was right for the proof path, wrong for external science); deterministic gold-key demoted to a cheap atomic cross-check. Recursive-drift loop = v2, gated on v1. Still open (fox calls): fixture population, **which SOTA judge** (API cost + Voyeur egress note), spend. Scaffold; no build past gate-1. One ticket, not ten. | 2026-05-19 | — |
|
||||
| #000056 | Operation Sandwich — cross-language grounding via query+display MT | **implemented & landed 2026-05-17 · default-OFF** (fox: "call it operation sandwich, create a new ticket and finish it"). Mechanism + bright line **verified live end-to-end** (real Hermes + real opus-mt: es query → English answer+verifier → es display; `answer_text` English, `display_answer` Spanish additive, `question_hash`/`verifier_policy_hash` invariant; 6 tests + full suite 2477 passed, 0 regressions). **Fan-out measured (§9, n=1, 75 q):** EN baseline 85% → es+sandwich 71% = **−14pp cost**; transitions PRESERVED 31 / DOWNGRADE 16 / LOST 17 / N/A 11. A deterministic round-trip predictor was tried and **refuted** (12/17 LOST round-tripped CLEAN; another instance of the codified CLAUDE.md bench-maxing lesson — not re-added). LOST taxonomy from the artifact: ≈7 entity-translate (`Boltzmann`→"perntzmann", `Tarsus`→"Tarso"), ≈3 broad-enum, ≈several n=1 noise. **Lever built+validated:** `arborist/qa/mt/entity_mask.py` mask/restore (real opus-mt: `who is Paul of Tarsus?` "Pablo de Tarso"→**"Paul of Tarsus"**); default-ON within the default-OFF sandwich. Caveat: bench is lowercased so cap-detector lift is a **lower bound** (corpus-title anchor = v2). Also: fan-out caught + fixed an 88%-engine-error concurrency defect (per-call model load → memoised singleton + lazy per-pair); French + Russian breads added (manifest, `crosslang_source_lang`). **Lift measured 2026-05-17 (comparator corrected, fox):** the true baseline is the pre-ticket ≈0% (raw es query → song-title noise, UNGROUNDED, 10.4s) — NOT native English. Against that: **the sandwich is a large net win (≈0% → 71% es grounded, proof core never corrupted); the −14pp vs English is the cost of a new capability, not a regression — calling it a "fail" was a comparator error.** Firmed 2026-05-18 (n=1, 0 engine-err): EN 85% · es-nomask 71% · fr-nomask **61%** (NOT 47% — that was the mask artifact; honest fr correction). The genuine negative is the **entity-mask lever**: net-negative in *both* languages (es 71→65 borderline, fr 61→47 = −14pp unambiguous; isolated Paul-of-Tarsus win didn't replicate — 3rd bench-maxing-lesson instance), now **default-OFF** (`crosslang_entity_mask=False`); no-mask sandwich is the keeper. Recommendation flipped: **worth continuing (minus the mask)**, not park. Remaining: n=3, fr no-mask, corpus-title anchoring (only untried lowercase-capable detector). CLAUDE.md updated with the durable cross-lang *convention* only. Tasks #14–#17. Phase 1 of the #000001 §7 family; new ticket clears don't-proliferate (fox-directed + distinct Dav1d audience + architectural inflection: a model dependency `[mt]` + a presentation-translation layer — anticipated by #000001 §7's "split the `[mt]` model-distribution work like `[nli]`/vecpack"). **Sandwich:** translate query es→en (retrieval-side, == `--retrieval-keywords`, binds into `retrieval_plan_hash`, NOT `question_hash`) → English answer through the **byte-for-byte untouched verifier** → translate the verified English `answer_text` en→es into a NEW `display_answer` field, banner-labelled, zero grounding (the `_render_audit_label` render-projection pattern). Engine: local `[mt]` extra, Helsinki-NLP `opus-mt-es-en`/`-en-es`, Apache-2.0, hash-pinned, off-repo `~/.arborist/models/mt/`, optional dep, graceful-degrade — mirrors `[nli]`/`ShadowNLI` (#000049) + vecpack (#000051) verbatim; never Hermes-3-8B; not an external API (reproducibility + zero egress + es↔en is the best-resourced pair). Default OFF (`crosslang_translate_enabled`, gated under Phase-0 `crosslang_guard_enabled`); `--crosslang-translate` / `XLANG_MT=1`. Hash invariants (corrected 2026-05-17 — `governance_policy_hash` is sha256 of the *whole* policy, keys.py:182): `question_hash` + `verifier_policy_hash` untouched (user question preserved, verifier byte-identical); `governance_policy_hash` moves like every policy flag → correct cache partitioning by config (not a leak); MT engine identity binds into `RetrievalPlan.mt_*` (run-DAG), not a policy hash. | 2026-05-17 | — |
|
||||
| #000055 | Windows quickstart without `make` (`tasks.py` + `make.bat`) | **in progress** — opened 2026-05-16 (fox: "bat files or some shit … avoid needing makefile for windows … we will test the quickstart on windows"). Pure-stdlib `tasks.py` runner mirroring the **quickstart subset** of the Makefile (bootstrap / fetch-cur / ingest-cur-attached / distill ×2 / query / inspect / falsify / burn / bootstrap-crawler / crawl-ingest / stats / verify / search / clean) + a ~10-line `make.bat` shim so `make <target>` works in Windows cmd and `.\make.bat <target>` in PowerShell. Audited the artifact (not the docs): the 2003 dump is opened via stdlib `bz2` (no external `bzip2`); only `fetch-cur` used `curl` (→ stdlib `urllib`); bash `for…&wait` → `subprocess.Popen` fan-out; `arborist` console-script lands at `.venv\Scripts\arborist.exe`. Net: quickstart needs only **Python 3.10+ + sqlite3** — the repo's existing ethos, now true on native Windows. Same `KEY=VALUE` make-style args so documented commands translate 1:1 (one doc form). Makefile untouched, still canonical on POSIX ("keep it as an option"). Found + fixed a README/Makefile discrepancy: README claimed `[dev,html]` bootstrap extras, Makefile installs `.[dev]` — artifact wins. Drift-pinned by `tests/test_tasks_runner.py`. | 2026-05-16 | — |
|
||||
|
|
@ -171,4 +174,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000058`
|
||||
`000061`
|
||||
|
|
|
|||
|
|
@ -591,3 +591,34 @@ protocol must account for:
|
|||
legit cross-sentence summary; #000048 §5), so closing those needs
|
||||
an attribution-aware verifier (#000049, its own ticket if
|
||||
wanted) — an optional follow-up, not a #000012 blocker.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 (proposed) — curvature-aware ForkScore extension (folded from the Dav1dPrometheus report, 2026-05-20)
|
||||
|
||||
The Dav1dPrometheus *Protocol-Layer AGI* working report (2026-05-20,
|
||||
held outside the repo per fox; §11 Priority 3 / §7.3) proposes
|
||||
extending `ForkScore` with truth-maintenance deltas. **Folded here,
|
||||
not spawned as a sibling** (don't-proliferate): it is a weight-set
|
||||
extension of this in-progress ticket, and `arborist/substrate/weights.py`
|
||||
already reserves the slots (`iota` SecurityRisk, `kappa` Complexity).
|
||||
|
||||
Proposed extension (hard rejects unchanged — hard regression /
|
||||
NEG_INF_REGRESSION / rollback-missing / audit-break all still reject):
|
||||
|
||||
```
|
||||
ExtendedForkScore = ForkScore
|
||||
+ a·ΔHABCDEFG + b·κ_HABCDEFG + c·ΔFalsifierCoverage
|
||||
+ d·ΔJaggednessReduction + e·ΔPoisonSurvivalReduction
|
||||
− f·LatencyPenalty − g·SecurityRisk
|
||||
```
|
||||
|
||||
Inputs ΔHABCDEFG / κ_HABCDEFG / ΔJaggednessReduction come from the
|
||||
**#000060** H-ABCDEFG same-model substrate-delta harness — this phase
|
||||
is gated on that harness producing the metrics. ΔPoisonSurvivalReduction
|
||||
pairs with the **#000059** admission-discipline signals. The honest
|
||||
bound (report §5.3, Erratum 5): performance is bounded in [0,1], so no
|
||||
global-convexity claim — target positive slope + non-destructive
|
||||
curvature in the unsaturated regime, route to new constraint surfaces
|
||||
when curvature decays. Status: proposed, awaiting go/no-go + the
|
||||
#000060 metric outputs.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,105 @@
|
|||
# Ticket #000058 — `cache_key_9` verifier-policy dimension: mandatory-vs-legible decision + doc reconciliation
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-20
|
||||
**Scope:** Decide whether the 9-dim `cache_key` (with explicit
|
||||
`verifier_policy_hash`) becomes the default write form, and reconcile
|
||||
the "8-dim" language in CLAUDE.md / concepts docs with the
|
||||
`keys.py` reality (8-dim legacy + optional 9th).
|
||||
**Audience:** fox + a Dav1d de-novo review of the cache-identity
|
||||
invariant + maintainers of `arborist/qa/keys.py`.
|
||||
**Hard constraint:** do NOT silently stale every prior providence
|
||||
record without a measured, fox-gated reason. The audit chain and the
|
||||
schema column stay unchanged either way.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin
|
||||
|
||||
Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the
|
||||
Final Bet — Master Whitepaper Edition* (2026-05-20, Asia/Kuala_Lumpur;
|
||||
non-public working report, held outside the repo per fox 2026-05-20 —
|
||||
private-forecast / public-spine split). Its **§2 Erratum 1** and
|
||||
**§11 Priority 1** call for "mandatory `cache_key_9`":
|
||||
|
||||
> No high-warrant cache lookup may omit `verifier_policy_hash`. A claim
|
||||
> verified under old verifier rules should not silently serve under new
|
||||
> verifier rules.
|
||||
|
||||
This ticket is the isolated decision fox split out from the
|
||||
admission-discipline cluster (#000059) because flipping it mandatory
|
||||
would invalidate every cached record — a clean, separable call.
|
||||
|
||||
## 2. The requirement is already half-dumb (five-step #1)
|
||||
|
||||
**Make the requirement less dumb first.** Checked against the artifact
|
||||
(`arborist/qa/keys.py:259-307`), the report's stated *correctness*
|
||||
motivation is **already satisfied** in arborist:
|
||||
|
||||
- The verifier-relevant fields (`_VERIFIER_POLICY_FIELDS`, `keys.py:195`)
|
||||
are a **subset of the policy dict**. They therefore already fold into
|
||||
`governance_policy_hash` (`keys.py:182`), which IS one of the 8
|
||||
legacy dimensions.
|
||||
- The docstring is explicit (`keys.py:269-275`): *"bumping a verifier
|
||||
rule bumps BOTH dimensions … Bumping a non-verifier field (e.g.
|
||||
temperature) bumps ONLY governance_policy_hash."*
|
||||
|
||||
So a verifier-rule change **already** changes the cache_key today, via
|
||||
`governance_policy_hash`. A claim verified under old verifier rules
|
||||
**cannot** silently serve under new rules. The report's Erratum 1
|
||||
premise does not hold for arborist.
|
||||
|
||||
What the explicit 9th dimension actually buys is **audit legibility**:
|
||||
answering *"did the verifier rules change, specifically?"* with a
|
||||
single hash diff instead of scanning the whole policy. That is real,
|
||||
but it is not the correctness fix the report claims.
|
||||
|
||||
## 3. The decision
|
||||
|
||||
Given §2, "mandatory" is the wrong frame. The genuine choices:
|
||||
|
||||
- **Option A — leave as-is (legacy 8-dim default, 9th optional).**
|
||||
Zero cache churn. Legibility available to callers that opt in.
|
||||
- **Option B — make 9-dim the default *write* form** (call-sites pass
|
||||
`verifier_policy_hash_value`). One-time stale of all prior records on
|
||||
next lookup. Buys: every new record is verifier-legible by
|
||||
construction. Cost: full re-warm of the providence cache.
|
||||
- **Option C — Option B but bench-gated + flag-staged** (the
|
||||
#000008 / #000011 / #000049 / #000056 rollout discipline): land the
|
||||
default-write behind a flag, default-OFF, measure re-warm cost on a
|
||||
real shard, fox-flip later.
|
||||
|
||||
**Recommendation: Option A + doc reconciliation now**, unless fox wants
|
||||
the explicit verifier-legibility as a standing audit affordance — in
|
||||
which case **Option C**, never a hard mandatory flip. Correctness does
|
||||
not justify the cache invalidation; only legibility does, and legibility
|
||||
does not require staling history.
|
||||
|
||||
## 4. Doc reconciliation (do regardless of A/B/C)
|
||||
|
||||
The artifact divergence the report correctly flagged:
|
||||
|
||||
- `CLAUDE.md` schema-invariants: *"v9.8 8-dim cache_key"* — states 8,
|
||||
silent on the optional 9th.
|
||||
- `docs/_source/concepts.rst` (concepts page): lists the 8 dimensions,
|
||||
*"changing any … invalidates prior records."*
|
||||
- `arborist/qa/keys.py`: 8-dim legacy **+ optional 9th**
|
||||
`verifier_policy_hash`, with the both-dimensions overlap documented.
|
||||
|
||||
Reconcile to one truth: **8 canonical dimensions; an optional explicit
|
||||
9th (`verifier_policy_hash`) for verifier-rule legibility, whose content
|
||||
is already covered by `governance_policy_hash`.** Wrap any numeric claim
|
||||
in AUTOCOUNT per the doc-drift discipline if a count is asserted.
|
||||
|
||||
## 5. Scope boundaries
|
||||
|
||||
- No new schema column. No audit-chain change.
|
||||
- No hard mandatory flip without a measured re-warm cost + explicit go.
|
||||
- This ticket does NOT touch the graveyard / self-providence work
|
||||
(#000059) or the H-ABCDEFG harness (#000060), though all three trace
|
||||
to the same report's "admissible state transition" thesis.
|
||||
|
||||
## 6. Done when
|
||||
|
||||
Docs reconciled; fox has chosen A / B / C; if B/C, the default-write
|
||||
behaviour lands flag-staged with a measured re-warm cost recorded here.
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
# Ticket #000059 — Admission discipline: claim-graveyard query + self-providence quarantine
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-20
|
||||
**Scope:** Two coupled write-path mechanisms that decide what may
|
||||
*re-enter* durable state: (1) a graveyard-check that raises proof
|
||||
burden when a previously-failed claim is re-asked; (2) a
|
||||
self-providence guard that detects and quarantines model-output
|
||||
masquerading as independent evidence.
|
||||
**Audience:** fox + a Dav1d de-novo review of the
|
||||
"accepted-state-transition" admission boundary + maintainers of
|
||||
`arborist/qa/runner.py`, `arborist/store.py`, the `ingest-self`
|
||||
family.
|
||||
**Hard constraint:** verifier stays binary; these are admission /
|
||||
burden sidecars, not new verifier output fields. No write to
|
||||
`providence_cache` semantic columns beyond the existing
|
||||
`falsification_state` machine. Cores never evict.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin
|
||||
|
||||
Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the
|
||||
Final Bet* (2026-05-20 working report; held outside the repo per fox).
|
||||
This is the report's **§11 Priority 2** ("claim graveyard +
|
||||
self-providence quarantine") and the strongest concrete arborist gaps
|
||||
under its central thesis (**§57**): *mature cognition is admissible
|
||||
state transition — what is allowed to enter durable state — not output
|
||||
generation.*
|
||||
|
||||
## 2. Why these two together (clears don't-proliferate)
|
||||
|
||||
Both answer the same question on the **write / re-ask path**: *should
|
||||
this be admitted?* They share the same audit seam and the same
|
||||
`falsification_state` machine, and the report groups them as one
|
||||
priority. Splitting them would fragment one admission-discipline review.
|
||||
They are a distinct Dav1d-reviewable audience from the *measurement*
|
||||
work (#000060) and the *cache-identity* call (#000058).
|
||||
|
||||
## 3. Sub-item A — Graveyard-check (burden-shift on re-ask)
|
||||
|
||||
**What exists.** `falsification_state ∈ {live, failed, stale,
|
||||
quarantined}` already preserves rejected claims in `providence_cache`
|
||||
(audit-preserving `make falsify`, not delete). Cache lookups filter to
|
||||
`state='live'`. So the *graveyard storage* already exists — failed /
|
||||
stale / quarantined records are the graveyard.
|
||||
|
||||
**The gap.** There is no **GraveyardCheck**: when a new question
|
||||
canonicalizes to a claim family that previously `failed`, the runner
|
||||
does not raise the proof burden. The report's point (§42.9, §62-63):
|
||||
errors repeat; a claim with a known failure history should require
|
||||
*stronger* evidence, not start fresh.
|
||||
|
||||
**Sketch (read-only sidecar first — five-step #5, automate last):**
|
||||
- On `ask()`, after canonicalizing the question/claim, query the
|
||||
shard(s) for prior `failed`/`quarantined` records in the same
|
||||
canonical/predicate/source-root family.
|
||||
- Emit an advisory `graveyard_status` + `required_proof_burden` on the
|
||||
run-DAG (NOT into `providence_cache` / `audit_events`), mirroring the
|
||||
existing `arborist inspect` sidecar discipline.
|
||||
- A demote-only verifier hook (raise the warrant threshold for a
|
||||
graveyard hit) is possible but, like the coherence sidecar, stays
|
||||
**deliberately unwired** until a bench shows it nets positive — wiring
|
||||
it would fold into `governance_policy_hash`.
|
||||
|
||||
## 4. Sub-item B — Self-providence quarantine
|
||||
|
||||
**What exists — and why it's a live risk.** `make
|
||||
ingest-self-providence` (Makefile:769) deliberately promotes STRICT
|
||||
live providence records back into the document corpus. That is the
|
||||
exact recursive self-confirmation loop the report warns about (§70):
|
||||
|
||||
```
|
||||
model output → promoted to corpus → retrieved later as evidence
|
||||
→ supports related model output → recursive self-confirmation
|
||||
```
|
||||
|
||||
Arborist ships the mechanism that *creates* the risk and **no guard
|
||||
against it**. That asymmetry is the strongest single motivation in this
|
||||
ticket — it is not a hypothetical.
|
||||
|
||||
**The gap → guard.** Detect when a retrieved evidence chunk's
|
||||
`source_root` descends from a prior-model-output root (i.e. was ingested
|
||||
via the self-providence path) with no independent corroborating source,
|
||||
and lower its warrant / quarantine it for high-impact claims.
|
||||
|
||||
**Sketch:**
|
||||
- Tag self-providence-ingested documents at ingest time with a
|
||||
provenance marker (an edge / source-role: `self_providence_edges`, or
|
||||
reuse the existing edge/`supersedes` machinery) so the lineage is
|
||||
queryable without scanning text.
|
||||
- At retrieval/verify time, compute `SelfProvenanceRisk` = high when a
|
||||
cited evidence root descends from a self-providence root AND no
|
||||
independent source root supports the same claim.
|
||||
- Response: lower warrant; require an independent source; quarantine if
|
||||
used for a high-impact claim. Advisory-first (sidecar), demote-hook
|
||||
bench-gated like §3.
|
||||
|
||||
**Warrant adjustment (report §70), as a *soft* ranking signal only —
|
||||
never in the hard proof path:**
|
||||
`WarrantAdjusted = WarrantBase − λ·SelfProvenanceDepth + μ·IndependentSourceDiversity`.
|
||||
|
||||
## 5. Scope boundaries
|
||||
|
||||
- Advisory sidecars first; any demote-hook is a separate bench-gated
|
||||
flag that folds into `governance_policy_hash`.
|
||||
- No new semantic `providence_cache` columns; reuse
|
||||
`falsification_state` + edges + run-DAG advisory fields.
|
||||
- Soft signals (similarity, depth, diversity) never enter the hard
|
||||
hash / proof path (CLAUDE.md soft-vs-hard invariant).
|
||||
- Burn/delete stays scratch-only; default is falsify / quarantine /
|
||||
stale (report §8 fail-safe hierarchy = arborist's existing one).
|
||||
|
||||
## 6. Done when
|
||||
|
||||
GraveyardCheck emits an advisory burden signal on the run-DAG for
|
||||
re-asked failed-claim families; self-providence-ingested documents are
|
||||
lineage-tagged and a `SelfProvenanceRisk` sidecar flags
|
||||
self-confirming evidence; both behind read-only sidecars with
|
||||
bench-gated demote-hooks left unwired pending a measured net win.
|
||||
Tests + `make chain-check-shards` clean (0 per shard).
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
# Ticket #000060 — H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + performance curvature)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-20
|
||||
**Scope:** The report's "decisive proof" — a harness that runs the
|
||||
SAME base model with and without the arborist substrate over
|
||||
long-horizon / adversarial / non-jagged batteries, and reports the
|
||||
delta. Produces two new metrics — a jaggedness tensor and discrete
|
||||
performance curvature — as first-class outputs.
|
||||
**Audience:** fox + a Dav1d de-novo review of the empirical bet + bench
|
||||
maintainers (`bench/`, `docs/benchmarks.md`).
|
||||
**Hard constraint:** deterministic where possible; gate numbers come
|
||||
from bench-qa pipeline output, never contrived fixtures (CLAUDE.md
|
||||
bench-maxing rule); n=3, 5pp signal floor. Mined fixtures complement,
|
||||
never replace, the curated adversarial set.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin
|
||||
|
||||
Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the
|
||||
Final Bet* (2026-05-20 working report; held outside the repo per fox).
|
||||
Across **§26, §50, §82-84** the report's single most-repeated claim is
|
||||
that the architecture is *theoretically strong but not empirically
|
||||
settled*, and the decisive next step is:
|
||||
|
||||
> same model · same compute · same tasks · with and without substrate
|
||||
> · measure H-ABCDEFG performance.
|
||||
|
||||
That is a measurement gap, not a strategy gap — squarely in arborist's
|
||||
remit and the cheapest high-value thing to build.
|
||||
|
||||
## 2. What exists vs the gap
|
||||
|
||||
**Exists:** the benchmark surface (5S/5T/5F/5R batteries, deterministic
|
||||
fixtures, no LLM-as-judge); `bench/mine_questions.py` +
|
||||
`bench/recall_at_k.py` (deterministic retrieval recall@k); `bench-qa`
|
||||
QA-quality sweep; the cost/quality matrix work (#000057).
|
||||
|
||||
**Gap:** none of these run the **same base model in a substrate-on /
|
||||
substrate-off A/B** and report the delta. The report's systems:
|
||||
|
||||
```
|
||||
A: base model + standard agent scaffold
|
||||
B: base model + lab-style memory/tools/evals
|
||||
C: base model + Arborist / PROMETHEUS-Σ substrate
|
||||
D: base model + open proof mesh (out of scope here — #000012/#000016)
|
||||
```
|
||||
|
||||
For arborist the tractable, honest cut is **A vs C** (and optionally B):
|
||||
toggle the substrate, hold the model + tasks fixed, measure.
|
||||
|
||||
## 3. The two new metrics (report §73, §5)
|
||||
|
||||
- **Jaggedness tensor (§73).** For a task family with semantically
|
||||
nearby variants, `J = variance of performance across variants`,
|
||||
normalized by expected difficulty (`J_norm`). A system with high
|
||||
benchmark score but high jaggedness fails the Hassabis non-jagged
|
||||
bar. Testable prediction the harness checks: *same base model +
|
||||
substrate should reduce `J_norm` vs the model alone.* This rides the
|
||||
existing `mine_questions` / perturbation tooling — generate nearby
|
||||
variants, measure variance.
|
||||
- **Performance curvature (§5.2).** Discrete second difference
|
||||
`κ_t = P_t − 2·P_{t-1} + P_{t-2}` over upgrade cycles, with the
|
||||
honest bound correction (§5.3, Erratum 5): performance is bounded in
|
||||
`[0,1]`, so **no global-convexity claim** — target positive slope +
|
||||
non-destructive curvature in the unsaturated regime, route to new
|
||||
constraint surfaces when curvature decays.
|
||||
|
||||
## 4. Coupled fold (NOT a new ticket): curvature-aware ForkScore → #000012
|
||||
|
||||
The report's **§11 Priority 3 / §7.3** wants ForkScore extended with
|
||||
ΔHABCDEFG, κ, jaggedness-delta, falsifier-coverage-delta. That is an
|
||||
**extension of an in-progress ticket, not a new one** — `weights.py`
|
||||
already reserves the slots (`iota` SecurityRisk, `kappa` Complexity).
|
||||
Per don't-proliferate, this folds into **#000012** (Selection &
|
||||
consensus / ForkScore) as a new phase, consuming this harness's metric
|
||||
outputs. Cross-link both ways; do not spawn a sibling.
|
||||
|
||||
Suggested extension (recorded for #000012, hard rejects unchanged —
|
||||
hard regression / NEG_INF_REGRESSION / rollback-missing / audit-break
|
||||
all still reject):
|
||||
|
||||
```
|
||||
ExtendedForkScore = ForkScore
|
||||
+ a·ΔHABCDEFG + b·κ_HABCDEFG + c·ΔFalsifierCoverage
|
||||
+ d·ΔJaggednessReduction + e·ΔPoisonSurvivalReduction
|
||||
− f·LatencyPenalty − g·SecurityRisk
|
||||
```
|
||||
|
||||
## 5. Scope boundaries
|
||||
|
||||
- A-vs-C (substrate off/on) is the spine; B (lab-style memory/tools) is
|
||||
optional; D (proof mesh) is explicitly OUT (consensus/mesh =
|
||||
#000012/#000016).
|
||||
- Budget discipline: control-arm LLM calls route to Hermes-3-8B +
|
||||
Qwen, never Opus, without an explicit fox go (CLAUDE.md budget rule).
|
||||
Heavy off-device passes run on the GPU box, never in arborist's
|
||||
python+sqlite3 core.
|
||||
- Deterministic retrieval-recall arms (no LLM) fan out in parallel;
|
||||
only same-fixture LLM A/B that toggles state needs serialization.
|
||||
- ABCDEFG self-validation caveat (report §27.2, Erratum 4 / Anomaly 4):
|
||||
include external held-out / mechanism-agnostic variants so the
|
||||
harness does not reward the substrate by construction.
|
||||
|
||||
## 6. Done when
|
||||
|
||||
A `make` target runs the same base model substrate-off vs substrate-on
|
||||
over the batteries and reports per-battery delta + `J_norm` + curvature,
|
||||
with gate numbers from bench-qa pipeline output (n≥3); the jaggedness
|
||||
and curvature outputs are consumable by the #000012 ForkScore
|
||||
extension. `docs/benchmarks.md` gains the new harness + metric schema.
|
||||
Loading…
Add table
Add a link
Reference in a new issue