diff --git a/docs/TICKETS.md b/docs/TICKETS.md index d9f60b5..cbdfbd8 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -111,7 +111,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| -| #000057 | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). Scaffold only until fox rules scope (task + metric + spend). One ticket, not ten. | 2026-05-19 | — | +| #000057 | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). **Sharpened by fox 2026-05-19 (§4b/§5):** the review's strongest point — every measurement so far is *within* Arborist, no control arm. v1 (smallest proof) reordered to single-shot **Hermes-solo vs Arborist** on a fabrication-bait atomic-fact fixture, judged by an **external deterministic gold-fact key** (no Arborist verifier = no circularity; no LLM-as-judge; paraphrase-invariant for atomic facts); headline = fabrication-vs-honest-abstention delta (not raw accuracy — common-knowledge would falsely show null). Recursive-drift loop = v2, only after v1 shows a delta. Atomic-fact only (essay-grounding = #000049's unsolved NLI; OUT). Still open (fox calls): fixture population, gold-key shapes, spend. Scaffold; no build past gate-1 until ruled. One ticket, not ten. | 2026-05-19 | — | | #000056 | Operation Sandwich — cross-language grounding via query+display MT | **implemented & landed 2026-05-17 · default-OFF** (fox: "call it operation sandwich, create a new ticket and finish it"). Mechanism + bright line **verified live end-to-end** (real Hermes + real opus-mt: es query → English answer+verifier → es display; `answer_text` English, `display_answer` Spanish additive, `question_hash`/`verifier_policy_hash` invariant; 6 tests + full suite 2477 passed, 0 regressions). **Fan-out measured (§9, n=1, 75 q):** EN baseline 85% → es+sandwich 71% = **−14pp cost**; transitions PRESERVED 31 / DOWNGRADE 16 / LOST 17 / N/A 11. A deterministic round-trip predictor was tried and **refuted** (12/17 LOST round-tripped CLEAN; another instance of the codified CLAUDE.md bench-maxing lesson — not re-added). LOST taxonomy from the artifact: ≈7 entity-translate (`Boltzmann`→"perntzmann", `Tarsus`→"Tarso"), ≈3 broad-enum, ≈several n=1 noise. **Lever built+validated:** `arborist/qa/mt/entity_mask.py` mask/restore (real opus-mt: `who is Paul of Tarsus?` "Pablo de Tarso"→**"Paul of Tarsus"**); default-ON within the default-OFF sandwich. Caveat: bench is lowercased so cap-detector lift is a **lower bound** (corpus-title anchor = v2). Also: fan-out caught + fixed an 88%-engine-error concurrency defect (per-call model load → memoised singleton + lazy per-pair); French + Russian breads added (manifest, `crosslang_source_lang`). **Lift measured 2026-05-17 (comparator corrected, fox):** the true baseline is the pre-ticket ≈0% (raw es query → song-title noise, UNGROUNDED, 10.4s) — NOT native English. Against that: **the sandwich is a large net win (≈0% → 71% es grounded, proof core never corrupted); the −14pp vs English is the cost of a new capability, not a regression — calling it a "fail" was a comparator error.** Firmed 2026-05-18 (n=1, 0 engine-err): EN 85% · es-nomask 71% · fr-nomask **61%** (NOT 47% — that was the mask artifact; honest fr correction). The genuine negative is the **entity-mask lever**: net-negative in *both* languages (es 71→65 borderline, fr 61→47 = −14pp unambiguous; isolated Paul-of-Tarsus win didn't replicate — 3rd bench-maxing-lesson instance), now **default-OFF** (`crosslang_entity_mask=False`); no-mask sandwich is the keeper. Recommendation flipped: **worth continuing (minus the mask)**, not park. Remaining: n=3, fr no-mask, corpus-title anchoring (only untried lowercase-capable detector). CLAUDE.md updated with the durable cross-lang *convention* only. Tasks #14–#17. Phase 1 of the #000001 §7 family; new ticket clears don't-proliferate (fox-directed + distinct Dav1d audience + architectural inflection: a model dependency `[mt]` + a presentation-translation layer — anticipated by #000001 §7's "split the `[mt]` model-distribution work like `[nli]`/vecpack"). **Sandwich:** translate query es→en (retrieval-side, == `--retrieval-keywords`, binds into `retrieval_plan_hash`, NOT `question_hash`) → English answer through the **byte-for-byte untouched verifier** → translate the verified English `answer_text` en→es into a NEW `display_answer` field, banner-labelled, zero grounding (the `_render_audit_label` render-projection pattern). Engine: local `[mt]` extra, Helsinki-NLP `opus-mt-es-en`/`-en-es`, Apache-2.0, hash-pinned, off-repo `~/.arborist/models/mt/`, optional dep, graceful-degrade — mirrors `[nli]`/`ShadowNLI` (#000049) + vecpack (#000051) verbatim; never Hermes-3-8B; not an external API (reproducibility + zero egress + es↔en is the best-resourced pair). Default OFF (`crosslang_translate_enabled`, gated under Phase-0 `crosslang_guard_enabled`); `--crosslang-translate` / `XLANG_MT=1`. Hash invariants (corrected 2026-05-17 — `governance_policy_hash` is sha256 of the *whole* policy, keys.py:182): `question_hash` + `verifier_policy_hash` untouched (user question preserved, verifier byte-identical); `governance_policy_hash` moves like every policy flag → correct cache partitioning by config (not a leak); MT engine identity binds into `RetrievalPlan.mt_*` (run-DAG), not a policy hash. | 2026-05-17 | — | | #000055 | Windows quickstart without `make` (`tasks.py` + `make.bat`) | **in progress** — opened 2026-05-16 (fox: "bat files or some shit … avoid needing makefile for windows … we will test the quickstart on windows"). Pure-stdlib `tasks.py` runner mirroring the **quickstart subset** of the Makefile (bootstrap / fetch-cur / ingest-cur-attached / distill ×2 / query / inspect / falsify / burn / bootstrap-crawler / crawl-ingest / stats / verify / search / clean) + a ~10-line `make.bat` shim so `make ` works in Windows cmd and `.\make.bat ` in PowerShell. Audited the artifact (not the docs): the 2003 dump is opened via stdlib `bz2` (no external `bzip2`); only `fetch-cur` used `curl` (→ stdlib `urllib`); bash `for…&wait` → `subprocess.Popen` fan-out; `arborist` console-script lands at `.venv\Scripts\arborist.exe`. Net: quickstart needs only **Python 3.10+ + sqlite3** — the repo's existing ethos, now true on native Windows. Same `KEY=VALUE` make-style args so documented commands translate 1:1 (one doc form). Makefile untouched, still canonical on POSIX ("keep it as an option"). Found + fixed a README/Makefile discrepancy: README claimed `[dev,html]` bootstrap extras, Makefile installs `.[dev]` — artifact wins. Drift-pinned by `tests/test_tasks_runner.py`. | 2026-05-16 | — | | #000054 | Acronym-parens concept extractor (closes the abbreviation→expansion retrieval gap) | **in progress** — Phase 1 (extractor + 481K edges) landed `58027e9`; Phase 2 (consumer-side surfacing — `synonym_expand` rank-and-truncate over the per-token cap, FTS5-`bm25` ordering in `_search_titles`, expanded `accept_tokens` in title-search + core-keyword + title-rerank, `synonym_expand_strict()` for the multiplicative title-purity rerank to exclude noisy `link_reciprocity` edges, tightened extractor regex to `[A-Z]{3,6}` purging 2-letter homonym edges) landed `ce855db`. **End-to-end verified:** `what is a CPU?` → Central processing unit at #1; `what is a GPU?` → Graphics processing unit at #1 EVIDENCE-WARRANTED 1/1; Mount Kilimanjaro / Soviet Union queries unchanged. **bench-qa n=3 limit=5** (2026-05-13T14:24Z): 30/45 STRICT (67%), zero regressions on basics (mona lisa / capital of france / new london bridge each 9/9 STRICT). 2026-05-13 — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in `EXTRACTORS` (`evidence_kind="acronym_parens"`). Scans each doc's lead chunk for ` (ACRO)` where the all-caps acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the lowercased acronym and each ≥3-char content token of the phrase. Conservative (strict 1:1 initials, function words filtered, repeated definitions deduped per doc). Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect* — not an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML/blogs/textbooks benefit equally. Retrieval-side only — never proof-path. 8 new tests; full suite green. Closes #000050 §2a's CPU/GPU fixture rows *upstream* of vec; the Orwell-shape conceptual-allusion row remains the genuine #000050 justification. Operational follow-up (not code): `arborist concepts derive --extractor acronym_parens` on each shard. | 2026-05-13 | — | diff --git a/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md b/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md index a1d1ee4..854fb7a 100644 --- a/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md +++ b/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md @@ -80,10 +80,71 @@ experiment), distinct audience, not a sub-tweak of an existing ticket. Exactly **one** ticket — the rest of the review is explicitly refused above. -## 5. Decision required (go/no-go, fox) +## 4b. Control-arm + external judge (fox ruling 2026-05-19) -Scaffold only until fox rules on **scope**: (a) which single task -(stale-source-invalidation vs contradiction-repair), (b) the exact -deterministic drift metric, (c) whether the minimal A/B is worth the -spend now vs parked. No design past §2 gate 1 until that ruling — -building the experiment before the metric is the precedent error. +The sharpest point in the review (§4.1) and fox's correction: +**every measurement so far is *within* Arborist; there has been no +control arm.** A treatment-vs-control experiment needs (a) the +control = **Hermes-solo** (no Arborist retrieval / claim-lattice / +provenance — pure parametric answer), and (b) a judge **outside +Arborist** (scoring with Arborist's own verifier = treatment grading +itself = invalid). + +Three constraints, one resolving design: + +- circularity → judge must be external to Arborist; +- LLM-as-judge is forbidden (CLAUDE.md; #000049 showed a model-judge + is itself only 0.24–0.48 reliable on real haystacks); +- a lexical/substring judge over-punishes paraphrase **and** is + Arborist's own machinery (circular again). + +**Resolution — deterministic gold-fact key.** Mine questions whose +target article yields an **atomic gold fact** (year / name / place). +The judge is a normalization-tolerant gold-fact match (number↔word, +date formats, curated name aliases): external, fully deterministic, +no LLM, paraphrase-invariant *for atomic facts*. Same discipline as +`recall_at_k` — ground-truth-carrying, no LLM-as-judge — extended +from "did the right article retrieve" to "did the answer state the +right fact". Both arms (Hermes-solo, Arborist) scored identically. + +**Metric — fabrication vs honest abstention, not raw accuracy.** +A common-knowledge fixture would falsely show "Arborist adds +nothing" (parametric memory already knows "capital of France"). The +claim Arborist actually makes is: on corpus-specific / stale / +false-premise / fabrication-bait questions, Hermes-solo +*confidently fabricates* while Arborist *fails closed (UNGROUNDED)*. +The headline delta = Hermes-solo confidently-wrong rate vs Arborist +honest-abstention rate, deterministically computable from the gold +key (gold absent AND a competing wrong fact asserted = fabrication). + +**Scope limit (pinned):** atomic-fact factoids only. Essay-grounding +judgment ("describe DNA structure" — no single gold token) requires +entailment/NLI = #000049's *unsolved* problem; explicitly OUT of v1. +Optional documented extension: a middle arm (Hermes + naive RAG, no +Arborist verification) isolates the verification-layer delta vs the +retrieval delta — not v1. + +## 5. Decision status & the smallest-proof decomposition + +fox's control-arm framing (§4b) reorders the experiment honestly: +the **single-shot treatment-vs-control** is *more foundational and +smaller* than the recursive-drift loop. So: + +- **v1 (the smallest irreversible proof):** single-shot **Hermes-solo + vs Arborist** on a fabrication-bait atomic-fact fixture, judged by + the external deterministic gold-fact key (§4b); headline = the + fabrication-vs-honest-abstention delta. This *is* the "force the + architecture to beat the baseline" result; no recursion needed yet. +- **v2 (the §1 claim):** the recursive-drift loop (witness ON/OFF, N + iterations) — only after v1 establishes a measurable single-shot + delta. Building v2 before v1 measures is the precedent error. + +**Resolved by fox (2026-05-19):** control = Hermes-solo; judge = +external, deterministic, paraphrase-tolerant (gold-fact key). +**Still open (genuine fox calls, not autonomous):** (a) the +fabrication-bait fixture population — corpus-specific / stale / +false-premise / known-fabrication-bait, *not* common-knowledge where +parametric memory wins anyway (the resolving-fixture lesson: +prevalence ≠ what isolates the claim); (b) gold-key construction +(which atomic-fact shapes, alias handling); (c) spend now vs parked. +No design past §2 gate-1 / §4b until (a)–(c) ruled.