diff --git a/CLAUDE.md b/CLAUDE.md index 9e6166a..6d1d21d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -162,10 +162,18 @@ chain audit. Chain breaks are the loudest possible signal. ## Schema invariants (do not break) -- **v9.8 8-dim cache_key**: `source_root | question_hash | - model_profile_hash | conversation_hash | governance_policy_hash | - schema_version | canonicalization_version | chunking_version`. - Bumping any one invalidates prior records on lookup. +- **v9.8 8-dim cache_key** (+ optional legible 9th): `source_root | + question_hash | model_profile_hash | conversation_hash | + governance_policy_hash | schema_version | canonicalization_version | + chunking_version`. Bumping any one invalidates prior records on + lookup. `cache_key()` also accepts an **optional 9th dimension**, + `verifier_policy_hash` (`keys.py`), for *audit legibility* — it does + NOT add correctness coverage, because the verifier-policy fields are + a subset of the policy dict and so already fold into + `governance_policy_hash` (a verifier-rule change already changes the + cache_key today; the 9th just makes "did the *verifier* rules + change?" answerable from one hash diff). 8-dim is the default write + form; the 9-dim form is opt-in. Mandatory-vs-legible is #000058. - **`falsification_state ∈ {live, failed, stale, quarantined}`**. Cache lookups filter on `state='live'`. Drift → `stale`. - **Audit chain**: every state-changing op writes one row in diff --git a/docs/TICKETS.md b/docs/TICKETS.md index ad46b14..13ff6c0 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -111,6 +111,9 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| +| #000060 | H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + curvature) | open · awaiting go/no-go (2026-05-20; from Dav1dPrometheus *Protocol-Layer AGI* working report §26/§50/§82-84). The report's "decisive proof": run the SAME base model substrate-OFF vs substrate-ON over long-horizon/adversarial/non-jagged batteries, report the delta. Two new metrics: jaggedness tensor `J_norm` (§73 — variance across nearby variants, normalized by difficulty) + discrete performance curvature `κ_t` (§5.2, with the honest no-global-convexity bound, Erratum 5). A-vs-C spine (B optional, D=mesh OUT → #000012/#000016). Curvature-aware ForkScore extension folds into **#000012** (NOT a new ticket — reserved `iota`/`kappa` weight slots already exist). Budget: control arms = Hermes/Qwen, never Opus without go; heavy passes on GPU box. Held-out/mechanism-agnostic variants required so ABCDEFG doesn't self-validate. | 2026-05-20 | — | +| #000059 | Admission discipline: claim-graveyard query + self-providence quarantine | open · awaiting go/no-go (2026-05-20; Dav1dPrometheus report §11 Priority 2 + §57 "admissible state transition" thesis). Two coupled write-path mechanisms. **(A) GraveyardCheck:** storage already exists (`falsification_state` failed/stale/quarantined records ARE the graveyard); the gap is burden-shifting — a re-asked claim family with a known `failed` history should require stronger evidence (§42.9/§62-63). **(B) Self-providence quarantine:** `make ingest-self-providence` (Makefile:769) deliberately promotes STRICT records into the corpus — the exact self-confirmation loop §70 warns of — and ships with NO guard; detect/lineage-tag self-providence-descended evidence + quarantine for high-impact claims. Both advisory-sidecar-first (run-DAG only, never `providence_cache`/`audit_events`); demote-hooks bench-gated + unwired pending net win (would fold `governance_policy_hash`). Soft signals never enter the hard proof path. | 2026-05-20 | — | +| #000058 | `cache_key_9` verifier-policy: mandatory-vs-legible decision + doc reconciliation | open · awaiting go/no-go (2026-05-20; Dav1dPrometheus report §2 Erratum 1 / §11 Priority 1 "mandatory cache_key_9"). **Five-step #1 correction:** the report's *correctness* premise is already false in arborist — verifier fields are a subset of the policy dict and so already fold into `governance_policy_hash` (`keys.py:269-275`); a verifier-rule change ALREADY changes the cache_key today. The explicit 9th `verifier_policy_hash` buys **audit legibility**, not correctness — so "mandatory" would stale every prior record for zero correctness gain. Decision: A leave-as-is (8-dim default, 9th optional) + doc reconcile [recommended] · B default-write 9-dim · C flag-staged bench-gated default-write — never a hard mandatory flip. Doc reconcile (CLAUDE.md/concepts "8-dim" → "8 + optional legible 9th") is the do-regardless. | 2026-05-20 | — | | #000057 | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). **Sharpened by fox 2026-05-19 (§4b/§5):** the review's strongest point — every measurement so far is *within* Arborist, no control arm. v1 (smallest proof) = single-shot **Hermes-solo vs Arborist** on a fabrication-bait fixture; headline = fabrication-vs-honest-abstention delta (NOT raw accuracy — common-knowledge would falsely show null via parametric memory). **Judge correction (fox):** "no LLM-as-judge" is a *proof-path* rule (LLM judge inside Arborist's verifier = the circularity it kills); the experiment's judge is **external science** outside *both* arms → use a **strong SOTA model**, blinded + reference-grounded + transcript-logged. That lifts the atomic-only limit (judge handles essay grounding too — #000049's exclusion was right for the proof path, wrong for external science); deterministic gold-key demoted to a cheap atomic cross-check. Recursive-drift loop = v2, gated on v1. Still open (fox calls): fixture population, **which SOTA judge** (API cost + Voyeur egress note), spend. Scaffold; no build past gate-1. One ticket, not ten. | 2026-05-19 | — | | #000056 | Operation Sandwich — cross-language grounding via query+display MT | **implemented & landed 2026-05-17 · default-OFF** (fox: "call it operation sandwich, create a new ticket and finish it"). Mechanism + bright line **verified live end-to-end** (real Hermes + real opus-mt: es query → English answer+verifier → es display; `answer_text` English, `display_answer` Spanish additive, `question_hash`/`verifier_policy_hash` invariant; 6 tests + full suite 2477 passed, 0 regressions). **Fan-out measured (§9, n=1, 75 q):** EN baseline 85% → es+sandwich 71% = **−14pp cost**; transitions PRESERVED 31 / DOWNGRADE 16 / LOST 17 / N/A 11. A deterministic round-trip predictor was tried and **refuted** (12/17 LOST round-tripped CLEAN; another instance of the codified CLAUDE.md bench-maxing lesson — not re-added). LOST taxonomy from the artifact: ≈7 entity-translate (`Boltzmann`→"perntzmann", `Tarsus`→"Tarso"), ≈3 broad-enum, ≈several n=1 noise. **Lever built+validated:** `arborist/qa/mt/entity_mask.py` mask/restore (real opus-mt: `who is Paul of Tarsus?` "Pablo de Tarso"→**"Paul of Tarsus"**); default-ON within the default-OFF sandwich. Caveat: bench is lowercased so cap-detector lift is a **lower bound** (corpus-title anchor = v2). Also: fan-out caught + fixed an 88%-engine-error concurrency defect (per-call model load → memoised singleton + lazy per-pair); French + Russian breads added (manifest, `crosslang_source_lang`). **Lift measured 2026-05-17 (comparator corrected, fox):** the true baseline is the pre-ticket ≈0% (raw es query → song-title noise, UNGROUNDED, 10.4s) — NOT native English. Against that: **the sandwich is a large net win (≈0% → 71% es grounded, proof core never corrupted); the −14pp vs English is the cost of a new capability, not a regression — calling it a "fail" was a comparator error.** Firmed 2026-05-18 (n=1, 0 engine-err): EN 85% · es-nomask 71% · fr-nomask **61%** (NOT 47% — that was the mask artifact; honest fr correction). The genuine negative is the **entity-mask lever**: net-negative in *both* languages (es 71→65 borderline, fr 61→47 = −14pp unambiguous; isolated Paul-of-Tarsus win didn't replicate — 3rd bench-maxing-lesson instance), now **default-OFF** (`crosslang_entity_mask=False`); no-mask sandwich is the keeper. Recommendation flipped: **worth continuing (minus the mask)**, not park. Remaining: n=3, fr no-mask, corpus-title anchoring (only untried lowercase-capable detector). CLAUDE.md updated with the durable cross-lang *convention* only. Tasks #14–#17. Phase 1 of the #000001 §7 family; new ticket clears don't-proliferate (fox-directed + distinct Dav1d audience + architectural inflection: a model dependency `[mt]` + a presentation-translation layer — anticipated by #000001 §7's "split the `[mt]` model-distribution work like `[nli]`/vecpack"). **Sandwich:** translate query es→en (retrieval-side, == `--retrieval-keywords`, binds into `retrieval_plan_hash`, NOT `question_hash`) → English answer through the **byte-for-byte untouched verifier** → translate the verified English `answer_text` en→es into a NEW `display_answer` field, banner-labelled, zero grounding (the `_render_audit_label` render-projection pattern). Engine: local `[mt]` extra, Helsinki-NLP `opus-mt-es-en`/`-en-es`, Apache-2.0, hash-pinned, off-repo `~/.arborist/models/mt/`, optional dep, graceful-degrade — mirrors `[nli]`/`ShadowNLI` (#000049) + vecpack (#000051) verbatim; never Hermes-3-8B; not an external API (reproducibility + zero egress + es↔en is the best-resourced pair). Default OFF (`crosslang_translate_enabled`, gated under Phase-0 `crosslang_guard_enabled`); `--crosslang-translate` / `XLANG_MT=1`. Hash invariants (corrected 2026-05-17 — `governance_policy_hash` is sha256 of the *whole* policy, keys.py:182): `question_hash` + `verifier_policy_hash` untouched (user question preserved, verifier byte-identical); `governance_policy_hash` moves like every policy flag → correct cache partitioning by config (not a leak); MT engine identity binds into `RetrievalPlan.mt_*` (run-DAG), not a policy hash. | 2026-05-17 | — | | #000055 | Windows quickstart without `make` (`tasks.py` + `make.bat`) | **in progress** — opened 2026-05-16 (fox: "bat files or some shit … avoid needing makefile for windows … we will test the quickstart on windows"). Pure-stdlib `tasks.py` runner mirroring the **quickstart subset** of the Makefile (bootstrap / fetch-cur / ingest-cur-attached / distill ×2 / query / inspect / falsify / burn / bootstrap-crawler / crawl-ingest / stats / verify / search / clean) + a ~10-line `make.bat` shim so `make ` works in Windows cmd and `.\make.bat ` in PowerShell. Audited the artifact (not the docs): the 2003 dump is opened via stdlib `bz2` (no external `bzip2`); only `fetch-cur` used `curl` (→ stdlib `urllib`); bash `for…&wait` → `subprocess.Popen` fan-out; `arborist` console-script lands at `.venv\Scripts\arborist.exe`. Net: quickstart needs only **Python 3.10+ + sqlite3** — the repo's existing ethos, now true on native Windows. Same `KEY=VALUE` make-style args so documented commands translate 1:1 (one doc form). Makefile untouched, still canonical on POSIX ("keep it as an option"). Found + fixed a README/Makefile discrepancy: README claimed `[dev,html]` bootstrap extras, Makefile installs `.[dev]` — artifact wins. Drift-pinned by `tests/test_tasks_runner.py`. | 2026-05-16 | — | @@ -171,4 +174,4 @@ Newest first. Update on every open/close. ## Next ID -`000058` +`000061` diff --git a/docs/tickets/ticket-000012-selection-consensus-protocol.md b/docs/tickets/ticket-000012-selection-consensus-protocol.md index 5af1636..6de35b7 100644 --- a/docs/tickets/ticket-000012-selection-consensus-protocol.md +++ b/docs/tickets/ticket-000012-selection-consensus-protocol.md @@ -591,3 +591,34 @@ protocol must account for: legit cross-sentence summary; #000048 §5), so closing those needs an attribution-aware verifier (#000049, its own ticket if wanted) — an optional follow-up, not a #000012 blocker. + +--- + +## Phase 2 (proposed) — curvature-aware ForkScore extension (folded from the Dav1dPrometheus report, 2026-05-20) + +The Dav1dPrometheus *Protocol-Layer AGI* working report (2026-05-20, +held outside the repo per fox; §11 Priority 3 / §7.3) proposes +extending `ForkScore` with truth-maintenance deltas. **Folded here, +not spawned as a sibling** (don't-proliferate): it is a weight-set +extension of this in-progress ticket, and `arborist/substrate/weights.py` +already reserves the slots (`iota` SecurityRisk, `kappa` Complexity). + +Proposed extension (hard rejects unchanged — hard regression / +NEG_INF_REGRESSION / rollback-missing / audit-break all still reject): + +``` +ExtendedForkScore = ForkScore + + a·ΔHABCDEFG + b·κ_HABCDEFG + c·ΔFalsifierCoverage + + d·ΔJaggednessReduction + e·ΔPoisonSurvivalReduction + − f·LatencyPenalty − g·SecurityRisk +``` + +Inputs ΔHABCDEFG / κ_HABCDEFG / ΔJaggednessReduction come from the +**#000060** H-ABCDEFG same-model substrate-delta harness — this phase +is gated on that harness producing the metrics. ΔPoisonSurvivalReduction +pairs with the **#000059** admission-discipline signals. The honest +bound (report §5.3, Erratum 5): performance is bounded in [0,1], so no +global-convexity claim — target positive slope + non-destructive +curvature in the unsaturated regime, route to new constraint surfaces +when curvature decays. Status: proposed, awaiting go/no-go + the +#000060 metric outputs. diff --git a/docs/tickets/ticket-000058-cache-key-9-verifier-policy-legibility.md b/docs/tickets/ticket-000058-cache-key-9-verifier-policy-legibility.md new file mode 100644 index 0000000..3073930 --- /dev/null +++ b/docs/tickets/ticket-000058-cache-key-9-verifier-policy-legibility.md @@ -0,0 +1,105 @@ +# Ticket #000058 — `cache_key_9` verifier-policy dimension: mandatory-vs-legible decision + doc reconciliation + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-20 +**Scope:** Decide whether the 9-dim `cache_key` (with explicit +`verifier_policy_hash`) becomes the default write form, and reconcile +the "8-dim" language in CLAUDE.md / concepts docs with the +`keys.py` reality (8-dim legacy + optional 9th). +**Audience:** fox + a Dav1d de-novo review of the cache-identity +invariant + maintainers of `arborist/qa/keys.py`. +**Hard constraint:** do NOT silently stale every prior providence +record without a measured, fox-gated reason. The audit chain and the +schema column stay unchanged either way. + +--- + +## 1. Origin + +Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the +Final Bet — Master Whitepaper Edition* (2026-05-20, Asia/Kuala_Lumpur; +non-public working report, held outside the repo per fox 2026-05-20 — +private-forecast / public-spine split). Its **§2 Erratum 1** and +**§11 Priority 1** call for "mandatory `cache_key_9`": + +> No high-warrant cache lookup may omit `verifier_policy_hash`. A claim +> verified under old verifier rules should not silently serve under new +> verifier rules. + +This ticket is the isolated decision fox split out from the +admission-discipline cluster (#000059) because flipping it mandatory +would invalidate every cached record — a clean, separable call. + +## 2. The requirement is already half-dumb (five-step #1) + +**Make the requirement less dumb first.** Checked against the artifact +(`arborist/qa/keys.py:259-307`), the report's stated *correctness* +motivation is **already satisfied** in arborist: + +- The verifier-relevant fields (`_VERIFIER_POLICY_FIELDS`, `keys.py:195`) + are a **subset of the policy dict**. They therefore already fold into + `governance_policy_hash` (`keys.py:182`), which IS one of the 8 + legacy dimensions. +- The docstring is explicit (`keys.py:269-275`): *"bumping a verifier + rule bumps BOTH dimensions … Bumping a non-verifier field (e.g. + temperature) bumps ONLY governance_policy_hash."* + +So a verifier-rule change **already** changes the cache_key today, via +`governance_policy_hash`. A claim verified under old verifier rules +**cannot** silently serve under new rules. The report's Erratum 1 +premise does not hold for arborist. + +What the explicit 9th dimension actually buys is **audit legibility**: +answering *"did the verifier rules change, specifically?"* with a +single hash diff instead of scanning the whole policy. That is real, +but it is not the correctness fix the report claims. + +## 3. The decision + +Given §2, "mandatory" is the wrong frame. The genuine choices: + +- **Option A — leave as-is (legacy 8-dim default, 9th optional).** + Zero cache churn. Legibility available to callers that opt in. +- **Option B — make 9-dim the default *write* form** (call-sites pass + `verifier_policy_hash_value`). One-time stale of all prior records on + next lookup. Buys: every new record is verifier-legible by + construction. Cost: full re-warm of the providence cache. +- **Option C — Option B but bench-gated + flag-staged** (the + #000008 / #000011 / #000049 / #000056 rollout discipline): land the + default-write behind a flag, default-OFF, measure re-warm cost on a + real shard, fox-flip later. + +**Recommendation: Option A + doc reconciliation now**, unless fox wants +the explicit verifier-legibility as a standing audit affordance — in +which case **Option C**, never a hard mandatory flip. Correctness does +not justify the cache invalidation; only legibility does, and legibility +does not require staling history. + +## 4. Doc reconciliation (do regardless of A/B/C) + +The artifact divergence the report correctly flagged: + +- `CLAUDE.md` schema-invariants: *"v9.8 8-dim cache_key"* — states 8, + silent on the optional 9th. +- `docs/_source/concepts.rst` (concepts page): lists the 8 dimensions, + *"changing any … invalidates prior records."* +- `arborist/qa/keys.py`: 8-dim legacy **+ optional 9th** + `verifier_policy_hash`, with the both-dimensions overlap documented. + +Reconcile to one truth: **8 canonical dimensions; an optional explicit +9th (`verifier_policy_hash`) for verifier-rule legibility, whose content +is already covered by `governance_policy_hash`.** Wrap any numeric claim +in AUTOCOUNT per the doc-drift discipline if a count is asserted. + +## 5. Scope boundaries + +- No new schema column. No audit-chain change. +- No hard mandatory flip without a measured re-warm cost + explicit go. +- This ticket does NOT touch the graveyard / self-providence work + (#000059) or the H-ABCDEFG harness (#000060), though all three trace + to the same report's "admissible state transition" thesis. + +## 6. Done when + +Docs reconciled; fox has chosen A / B / C; if B/C, the default-write +behaviour lands flag-staged with a measured re-warm cost recorded here. diff --git a/docs/tickets/ticket-000059-admission-discipline-graveyard-self-providence.md b/docs/tickets/ticket-000059-admission-discipline-graveyard-self-providence.md new file mode 100644 index 0000000..9ca6713 --- /dev/null +++ b/docs/tickets/ticket-000059-admission-discipline-graveyard-self-providence.md @@ -0,0 +1,121 @@ +# Ticket #000059 — Admission discipline: claim-graveyard query + self-providence quarantine + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-20 +**Scope:** Two coupled write-path mechanisms that decide what may +*re-enter* durable state: (1) a graveyard-check that raises proof +burden when a previously-failed claim is re-asked; (2) a +self-providence guard that detects and quarantines model-output +masquerading as independent evidence. +**Audience:** fox + a Dav1d de-novo review of the +"accepted-state-transition" admission boundary + maintainers of +`arborist/qa/runner.py`, `arborist/store.py`, the `ingest-self` +family. +**Hard constraint:** verifier stays binary; these are admission / +burden sidecars, not new verifier output fields. No write to +`providence_cache` semantic columns beyond the existing +`falsification_state` machine. Cores never evict. + +--- + +## 1. Origin + +Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the +Final Bet* (2026-05-20 working report; held outside the repo per fox). +This is the report's **§11 Priority 2** ("claim graveyard + +self-providence quarantine") and the strongest concrete arborist gaps +under its central thesis (**§57**): *mature cognition is admissible +state transition — what is allowed to enter durable state — not output +generation.* + +## 2. Why these two together (clears don't-proliferate) + +Both answer the same question on the **write / re-ask path**: *should +this be admitted?* They share the same audit seam and the same +`falsification_state` machine, and the report groups them as one +priority. Splitting them would fragment one admission-discipline review. +They are a distinct Dav1d-reviewable audience from the *measurement* +work (#000060) and the *cache-identity* call (#000058). + +## 3. Sub-item A — Graveyard-check (burden-shift on re-ask) + +**What exists.** `falsification_state ∈ {live, failed, stale, +quarantined}` already preserves rejected claims in `providence_cache` +(audit-preserving `make falsify`, not delete). Cache lookups filter to +`state='live'`. So the *graveyard storage* already exists — failed / +stale / quarantined records are the graveyard. + +**The gap.** There is no **GraveyardCheck**: when a new question +canonicalizes to a claim family that previously `failed`, the runner +does not raise the proof burden. The report's point (§42.9, §62-63): +errors repeat; a claim with a known failure history should require +*stronger* evidence, not start fresh. + +**Sketch (read-only sidecar first — five-step #5, automate last):** +- On `ask()`, after canonicalizing the question/claim, query the + shard(s) for prior `failed`/`quarantined` records in the same + canonical/predicate/source-root family. +- Emit an advisory `graveyard_status` + `required_proof_burden` on the + run-DAG (NOT into `providence_cache` / `audit_events`), mirroring the + existing `arborist inspect` sidecar discipline. +- A demote-only verifier hook (raise the warrant threshold for a + graveyard hit) is possible but, like the coherence sidecar, stays + **deliberately unwired** until a bench shows it nets positive — wiring + it would fold into `governance_policy_hash`. + +## 4. Sub-item B — Self-providence quarantine + +**What exists — and why it's a live risk.** `make +ingest-self-providence` (Makefile:769) deliberately promotes STRICT +live providence records back into the document corpus. That is the +exact recursive self-confirmation loop the report warns about (§70): + +``` +model output → promoted to corpus → retrieved later as evidence +→ supports related model output → recursive self-confirmation +``` + +Arborist ships the mechanism that *creates* the risk and **no guard +against it**. That asymmetry is the strongest single motivation in this +ticket — it is not a hypothetical. + +**The gap → guard.** Detect when a retrieved evidence chunk's +`source_root` descends from a prior-model-output root (i.e. was ingested +via the self-providence path) with no independent corroborating source, +and lower its warrant / quarantine it for high-impact claims. + +**Sketch:** +- Tag self-providence-ingested documents at ingest time with a + provenance marker (an edge / source-role: `self_providence_edges`, or + reuse the existing edge/`supersedes` machinery) so the lineage is + queryable without scanning text. +- At retrieval/verify time, compute `SelfProvenanceRisk` = high when a + cited evidence root descends from a self-providence root AND no + independent source root supports the same claim. +- Response: lower warrant; require an independent source; quarantine if + used for a high-impact claim. Advisory-first (sidecar), demote-hook + bench-gated like §3. + +**Warrant adjustment (report §70), as a *soft* ranking signal only — +never in the hard proof path:** +`WarrantAdjusted = WarrantBase − λ·SelfProvenanceDepth + μ·IndependentSourceDiversity`. + +## 5. Scope boundaries + +- Advisory sidecars first; any demote-hook is a separate bench-gated + flag that folds into `governance_policy_hash`. +- No new semantic `providence_cache` columns; reuse + `falsification_state` + edges + run-DAG advisory fields. +- Soft signals (similarity, depth, diversity) never enter the hard + hash / proof path (CLAUDE.md soft-vs-hard invariant). +- Burn/delete stays scratch-only; default is falsify / quarantine / + stale (report §8 fail-safe hierarchy = arborist's existing one). + +## 6. Done when + +GraveyardCheck emits an advisory burden signal on the run-DAG for +re-asked failed-claim families; self-providence-ingested documents are +lineage-tagged and a `SelfProvenanceRisk` sidecar flags +self-confirming evidence; both behind read-only sidecars with +bench-gated demote-hooks left unwired pending a measured net win. +Tests + `make chain-check-shards` clean (0 per shard). diff --git a/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md b/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md new file mode 100644 index 0000000..e699fc9 --- /dev/null +++ b/docs/tickets/ticket-000060-habcdefg-same-model-substrate-delta-harness.md @@ -0,0 +1,112 @@ +# Ticket #000060 — H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + performance curvature) + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-20 +**Scope:** The report's "decisive proof" — a harness that runs the +SAME base model with and without the arborist substrate over +long-horizon / adversarial / non-jagged batteries, and reports the +delta. Produces two new metrics — a jaggedness tensor and discrete +performance curvature — as first-class outputs. +**Audience:** fox + a Dav1d de-novo review of the empirical bet + bench +maintainers (`bench/`, `docs/benchmarks.md`). +**Hard constraint:** deterministic where possible; gate numbers come +from bench-qa pipeline output, never contrived fixtures (CLAUDE.md +bench-maxing rule); n=3, 5pp signal floor. Mined fixtures complement, +never replace, the curated adversarial set. + +--- + +## 1. Origin + +Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the +Final Bet* (2026-05-20 working report; held outside the repo per fox). +Across **§26, §50, §82-84** the report's single most-repeated claim is +that the architecture is *theoretically strong but not empirically +settled*, and the decisive next step is: + +> same model · same compute · same tasks · with and without substrate +> · measure H-ABCDEFG performance. + +That is a measurement gap, not a strategy gap — squarely in arborist's +remit and the cheapest high-value thing to build. + +## 2. What exists vs the gap + +**Exists:** the benchmark surface (5S/5T/5F/5R batteries, deterministic +fixtures, no LLM-as-judge); `bench/mine_questions.py` + +`bench/recall_at_k.py` (deterministic retrieval recall@k); `bench-qa` +QA-quality sweep; the cost/quality matrix work (#000057). + +**Gap:** none of these run the **same base model in a substrate-on / +substrate-off A/B** and report the delta. The report's systems: + +``` +A: base model + standard agent scaffold +B: base model + lab-style memory/tools/evals +C: base model + Arborist / PROMETHEUS-Σ substrate +D: base model + open proof mesh (out of scope here — #000012/#000016) +``` + +For arborist the tractable, honest cut is **A vs C** (and optionally B): +toggle the substrate, hold the model + tasks fixed, measure. + +## 3. The two new metrics (report §73, §5) + +- **Jaggedness tensor (§73).** For a task family with semantically + nearby variants, `J = variance of performance across variants`, + normalized by expected difficulty (`J_norm`). A system with high + benchmark score but high jaggedness fails the Hassabis non-jagged + bar. Testable prediction the harness checks: *same base model + + substrate should reduce `J_norm` vs the model alone.* This rides the + existing `mine_questions` / perturbation tooling — generate nearby + variants, measure variance. +- **Performance curvature (§5.2).** Discrete second difference + `κ_t = P_t − 2·P_{t-1} + P_{t-2}` over upgrade cycles, with the + honest bound correction (§5.3, Erratum 5): performance is bounded in + `[0,1]`, so **no global-convexity claim** — target positive slope + + non-destructive curvature in the unsaturated regime, route to new + constraint surfaces when curvature decays. + +## 4. Coupled fold (NOT a new ticket): curvature-aware ForkScore → #000012 + +The report's **§11 Priority 3 / §7.3** wants ForkScore extended with +ΔHABCDEFG, κ, jaggedness-delta, falsifier-coverage-delta. That is an +**extension of an in-progress ticket, not a new one** — `weights.py` +already reserves the slots (`iota` SecurityRisk, `kappa` Complexity). +Per don't-proliferate, this folds into **#000012** (Selection & +consensus / ForkScore) as a new phase, consuming this harness's metric +outputs. Cross-link both ways; do not spawn a sibling. + +Suggested extension (recorded for #000012, hard rejects unchanged — +hard regression / NEG_INF_REGRESSION / rollback-missing / audit-break +all still reject): + +``` +ExtendedForkScore = ForkScore + + a·ΔHABCDEFG + b·κ_HABCDEFG + c·ΔFalsifierCoverage + + d·ΔJaggednessReduction + e·ΔPoisonSurvivalReduction + − f·LatencyPenalty − g·SecurityRisk +``` + +## 5. Scope boundaries + +- A-vs-C (substrate off/on) is the spine; B (lab-style memory/tools) is + optional; D (proof mesh) is explicitly OUT (consensus/mesh = + #000012/#000016). +- Budget discipline: control-arm LLM calls route to Hermes-3-8B + + Qwen, never Opus, without an explicit fox go (CLAUDE.md budget rule). + Heavy off-device passes run on the GPU box, never in arborist's + python+sqlite3 core. +- Deterministic retrieval-recall arms (no LLM) fan out in parallel; + only same-fixture LLM A/B that toggles state needs serialization. +- ABCDEFG self-validation caveat (report §27.2, Erratum 4 / Anomaly 4): + include external held-out / mechanism-agnostic variants so the + harness does not reward the substrate by construction. + +## 6. Done when + +A `make` target runs the same base model substrate-off vs substrate-on +over the batteries and reports per-battery delta + `J_norm` + curvature, +with gate numbers from bench-qa pipeline output (n≥3); the jaggedness +and curvature outputs are consumable by the #000012 ForkScore +extension. `docs/benchmarks.md` gains the new harness + metric schema.