docs(#000057): correct judge design — strong SOTA external judge (fox), not deterministic-only

I over-applied a proof-path rule. "No LLM-as-judge" guards Arborist's
*verifier* (an LLM in the proof path = the circularity Arborist
kills; #000049's poor reliability is a proof-path-model constraint).
The control experiment's judge is external science — outside BOTH
arms, scoring post-hoc, touching neither system's internals — so it
cannot break the experiment. fox: use a strong SOTA model
(stronger than both systems under test), blinded + reference-
grounded + transcript-logged. This lifts the wrongly-imposed
atomic-fact-only limit (judge handles essay grounding; #000049's
exclusion was right for the proof path, wrong for external science);
deterministic gold-key demoted to a cheap atomic cross-check.
Still fox calls: fixture population, which SOTA judge (API
cost/egress), spend. Scaffold; no build past gate-1.

docs/TICKETS.md

@@ -111,7 +111,7 @@ Newest first. Update on every open/close.
 
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
-| #000057  | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). **Sharpened by fox 2026-05-19 (§4b/§5):** the review's strongest point — every measurement so far is *within* Arborist, no control arm. v1 (smallest proof) reordered to single-shot **Hermes-solo vs Arborist** on a fabrication-bait atomic-fact fixture, judged by an **external deterministic gold-fact key** (no Arborist verifier = no circularity; no LLM-as-judge; paraphrase-invariant for atomic facts); headline = fabrication-vs-honest-abstention delta (not raw accuracy — common-knowledge would falsely show null). Recursive-drift loop = v2, only after v1 shows a delta. Atomic-fact only (essay-grounding = #000049's unsolved NLI; OUT). Still open (fox calls): fixture population, gold-key shapes, spend. Scaffold; no build past gate-1 until ruled. One ticket, not ten. | 2026-05-19 | — |
+| #000057  | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). **Sharpened by fox 2026-05-19 (§4b/§5):** the review's strongest point — every measurement so far is *within* Arborist, no control arm. v1 (smallest proof) = single-shot **Hermes-solo vs Arborist** on a fabrication-bait fixture; headline = fabrication-vs-honest-abstention delta (NOT raw accuracy — common-knowledge would falsely show null via parametric memory). **Judge correction (fox):** "no LLM-as-judge" is a *proof-path* rule (LLM judge inside Arborist's verifier = the circularity it kills); the experiment's judge is **external science** outside *both* arms → use a **strong SOTA model**, blinded + reference-grounded + transcript-logged. That lifts the atomic-only limit (judge handles essay grounding too — #000049's exclusion was right for the proof path, wrong for external science); deterministic gold-key demoted to a cheap atomic cross-check. Recursive-drift loop = v2, gated on v1. Still open (fox calls): fixture population, **which SOTA judge** (API cost + Voyeur egress note), spend. Scaffold; no build past gate-1. One ticket, not ten. | 2026-05-19 | — |
 | #000056  | Operation Sandwich — cross-language grounding via query+display MT | **implemented & landed 2026-05-17 · default-OFF** (fox: "call it operation sandwich, create a new ticket and finish it"). Mechanism + bright line **verified live end-to-end** (real Hermes + real opus-mt: es query → English answer+verifier → es display; `answer_text` English, `display_answer` Spanish additive, `question_hash`/`verifier_policy_hash` invariant; 6 tests + full suite 2477 passed, 0 regressions). **Fan-out measured (§9, n=1, 75 q):** EN baseline 85% → es+sandwich 71% = **−14pp cost**; transitions PRESERVED 31 / DOWNGRADE 16 / LOST 17 / N/A 11. A deterministic round-trip predictor was tried and **refuted** (12/17 LOST round-tripped CLEAN; another instance of the codified CLAUDE.md bench-maxing lesson — not re-added). LOST taxonomy from the artifact: ≈7 entity-translate (`Boltzmann`→"perntzmann", `Tarsus`→"Tarso"), ≈3 broad-enum, ≈several n=1 noise. **Lever built+validated:** `arborist/qa/mt/entity_mask.py` mask/restore (real opus-mt: `who is Paul of Tarsus?` "Pablo de Tarso"→**"Paul of Tarsus"**); default-ON within the default-OFF sandwich. Caveat: bench is lowercased so cap-detector lift is a **lower bound** (corpus-title anchor = v2). Also: fan-out caught + fixed an 88%-engine-error concurrency defect (per-call model load → memoised singleton + lazy per-pair); French + Russian breads added (manifest, `crosslang_source_lang`). **Lift measured 2026-05-17 (comparator corrected, fox):** the true baseline is the pre-ticket ≈0% (raw es query → song-title noise, UNGROUNDED, 10.4s) — NOT native English. Against that: **the sandwich is a large net win (≈0% → 71% es grounded, proof core never corrupted); the −14pp vs English is the cost of a new capability, not a regression — calling it a "fail" was a comparator error.** Firmed 2026-05-18 (n=1, 0 engine-err): EN 85% · es-nomask 71% · fr-nomask **61%** (NOT 47% — that was the mask artifact; honest fr correction). The genuine negative is the **entity-mask lever**: net-negative in *both* languages (es 71→65 borderline, fr 61→47 = −14pp unambiguous; isolated Paul-of-Tarsus win didn't replicate — 3rd bench-maxing-lesson instance), now **default-OFF** (`crosslang_entity_mask=False`); no-mask sandwich is the keeper. Recommendation flipped: **worth continuing (minus the mask)**, not park. Remaining: n=3, fr no-mask, corpus-title anchoring (only untried lowercase-capable detector). CLAUDE.md updated with the durable cross-lang *convention* only. Tasks #14–#17. Phase 1 of the #000001 §7 family; new ticket clears don't-proliferate (fox-directed + distinct Dav1d audience + architectural inflection: a model dependency `[mt]` + a presentation-translation layer — anticipated by #000001 §7's "split the `[mt]` model-distribution work like `[nli]`/vecpack"). **Sandwich:** translate query es→en (retrieval-side, == `--retrieval-keywords`, binds into `retrieval_plan_hash`, NOT `question_hash`) → English answer through the **byte-for-byte untouched verifier** → translate the verified English `answer_text` en→es into a NEW `display_answer` field, banner-labelled, zero grounding (the `_render_audit_label` render-projection pattern). Engine: local `[mt]` extra, Helsinki-NLP `opus-mt-es-en`/`-en-es`, Apache-2.0, hash-pinned, off-repo `~/.arborist/models/mt/`, optional dep, graceful-degrade — mirrors `[nli]`/`ShadowNLI` (#000049) + vecpack (#000051) verbatim; never Hermes-3-8B; not an external API (reproducibility + zero egress + es↔en is the best-resourced pair). Default OFF (`crosslang_translate_enabled`, gated under Phase-0 `crosslang_guard_enabled`); `--crosslang-translate` / `XLANG_MT=1`. Hash invariants (corrected 2026-05-17 — `governance_policy_hash` is sha256 of the *whole* policy, keys.py:182): `question_hash` + `verifier_policy_hash` untouched (user question preserved, verifier byte-identical); `governance_policy_hash` moves like every policy flag → correct cache partitioning by config (not a leak); MT engine identity binds into `RetrievalPlan.mt_*` (run-DAG), not a policy hash. | 2026-05-17 | — |
 | #000055  | Windows quickstart without `make` (`tasks.py` + `make.bat`) | **in progress** — opened 2026-05-16 (fox: "bat files or some shit … avoid needing makefile for windows … we will test the quickstart on windows"). Pure-stdlib `tasks.py` runner mirroring the **quickstart subset** of the Makefile (bootstrap / fetch-cur / ingest-cur-attached / distill ×2 / query / inspect / falsify / burn / bootstrap-crawler / crawl-ingest / stats / verify / search / clean) + a ~10-line `make.bat` shim so `make <target>` works in Windows cmd and `.\make.bat <target>` in PowerShell. Audited the artifact (not the docs): the 2003 dump is opened via stdlib `bz2` (no external `bzip2`); only `fetch-cur` used `curl` (→ stdlib `urllib`); bash `for…&wait` → `subprocess.Popen` fan-out; `arborist` console-script lands at `.venv\Scripts\arborist.exe`. Net: quickstart needs only **Python 3.10+ + sqlite3** — the repo's existing ethos, now true on native Windows. Same `KEY=VALUE` make-style args so documented commands translate 1:1 (one doc form). Makefile untouched, still canonical on POSIX ("keep it as an option"). Found + fixed a README/Makefile discrepancy: README claimed `[dev,html]` bootstrap extras, Makefile installs `.[dev]` — artifact wins. Drift-pinned by `tests/test_tasks_runner.py`. | 2026-05-16 | — |
 | #000054  | Acronym-parens concept extractor (closes the abbreviation→expansion retrieval gap) | **in progress** — Phase 1 (extractor + 481K edges) landed `58027e9`; Phase 2 (consumer-side surfacing — `synonym_expand` rank-and-truncate over the per-token cap, FTS5-`bm25` ordering in `_search_titles`, expanded `accept_tokens` in title-search + core-keyword + title-rerank, `synonym_expand_strict()` for the multiplicative title-purity rerank to exclude noisy `link_reciprocity` edges, tightened extractor regex to `[A-Z]{3,6}` purging 2-letter homonym edges) landed `ce855db`. **End-to-end verified:** `what is a CPU?` → Central processing unit at #1; `what is a GPU?` → Graphics processing unit at #1 EVIDENCE-WARRANTED 1/1; Mount Kilimanjaro / Soviet Union queries unchanged. **bench-qa n=3 limit=5** (2026-05-13T14:24Z): 30/45 STRICT (67%), zero regressions on basics (mona lisa / capital of france / new london bridge each 9/9 STRICT). 2026-05-13 — `arborist/concepts/extract.py:acronym_parens_synonym` lands as a new corpus-agnostic extractor in `EXTRACTORS` (`evidence_kind="acronym_parens"`). Scans each doc's lead chunk for `<Multi-Word Phrase> (ACRO)` where the all-caps acronym's letters match the content-word initials of the phrase in order; emits bidirectional synonym edges between the lowercased acronym and each ≥3-char content token of the phrase. Conservative (strict 1:1 initials, function words filtered, repeated definitions deduped per doc). Closes the *retrieval-side* abbreviation gap (`CPU↔central processing unit`, `GPU↔graphics processing unit`, `RAM↔random access memory`, `FBI↔federal bureau of investigation`, `WHO↔world health organization`, …) that `link_reciprocity_synonym` can't reach because the relation lives in body text, not the wiki link graph (Wikipedia represents abbreviation→expansion as a *redirect* — not an edge). Per-shard like all `concept_relations` data; corpus-agnostic so HTML/blogs/textbooks benefit equally. Retrieval-side only — never proof-path. 8 new tests; full suite green. Closes #000050 §2a's CPU/GPU fixture rows *upstream* of vec; the Orwell-shape conceptual-allusion row remains the genuine #000050 justification. Operational follow-up (not code): `arborist concepts derive --extractor acronym_parens` on each shard. | 2026-05-13 | —         |

docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md

@@ -90,22 +90,37 @@ provenance — pure parametric answer), and (b) a judge **outside
 Arborist** (scoring with Arborist's own verifier = treatment grading
 itself = invalid).
 
-Three constraints, one resolving design:
+**CORRECTION (fox 2026-05-19, supersedes the earlier draft):** the
+"no LLM-as-judge" rule is a **proof-path** rule (an LLM judge
+*inside* Arborist's verifier is the fabrication/circularity risk
+Arborist exists to kill; #000049's 0.24–0.48 reliability is about a
+model *in the proof path*, cheap/pinned/shadow-constrained). The
+control experiment's judge is **external science** — outside *both*
+arms (Arborist and Hermes-solo), scoring outputs post-hoc. It does
+NOT touch either system's internals, so it cannot "break the
+experiment." Importing the proof-path constraint here was dogma.
 
-- circularity → judge must be external to Arborist;
-- LLM-as-judge is forbidden (CLAUDE.md; #000049 showed a model-judge
-  is itself only 0.24–0.48 reliable on real haystacks);
-- a lexical/substring judge over-punishes paraphrase **and** is
-  Arborist's own machinery (circular again).
+**Resolution — a strong SOTA model as the external judge.** A judge
+**stronger than both systems under test** (frontier-class grading
+Hermes-8B-class output) is standard, sound ML-eval. This also
+removes the atomic-fact-only limit: it judges paraphrase **and**
+non-atomic grounding ("describe DNA structure") — the #000049
+exclusion was right for the proof path, wrong for external science.
 
-**Resolution — deterministic gold-fact key.** Mine questions whose
-target article yields an **atomic gold fact** (year / name / place).
-The judge is a normalization-tolerant gold-fact match (number↔word,
-date formats, curated name aliases): external, fully deterministic,
-no LLM, paraphrase-invariant *for atomic facts*. Same discipline as
-`recall_at_k` — ground-truth-carrying, no LLM-as-judge — extended
-from "did the right article retrieve" to "did the answer state the
-right fact". Both arms (Hermes-solo, Arborist) scored identically.
+Science hygiene (kept — methodology, not dogma):
+- **blinded / order-randomised** — judge unaware which answer is
+  Arborist vs Hermes-solo (kills pro-treatment bias);
+- **reference-grounded** — judge scores "supported by *this source
+  text*" (source supplied), NOT vs its own parametric knowledge
+  (else judge cutoff/hallucination contaminates; essential for the
+  fabrication metric);
+- **the experiment is itself audited** — log judge model+version,
+  prompt, per-item verdict; the science measuring Arborist's
+  auditability must itself be replayable;
+- **deterministic gold-key demoted to a cheap cross-check** on the
+  atomic-fact subset: judge↔key disagreement = a judge-quality
+  flag (belt-and-suspenders, not the primary instrument).
+Both arms scored by the identical blinded reference-grounded judge.
 
 **Metric — fabrication vs honest abstention, not raw accuracy.**
 A common-knowledge fixture would falsely show "Arborist adds
@@ -117,10 +132,11 @@ The headline delta = Hermes-solo confidently-wrong rate vs Arborist
 honest-abstention rate, deterministically computable from the gold
 key (gold absent AND a competing wrong fact asserted = fabrication).
 
-**Scope limit (pinned):** atomic-fact factoids only. Essay-grounding
-judgment ("describe DNA structure" — no single gold token) requires
-entailment/NLI = #000049's *unsolved* problem; explicitly OUT of v1.
-Optional documented extension: a middle arm (Hermes + naive RAG, no
+**Scope (corrected):** the atomic-fact-only limit is **lifted** — a
+strong external judge handles non-atomic/essay grounding too. The
+v1 fixture is still chosen for *claim-isolating power* (next §),
+not because the judge can't do prose. Optional documented
+extension: a middle arm (Hermes + naive RAG, no
 Arborist verification) isolates the verification-layer delta vs the
 retrieval delta — not v1.
 
@@ -139,12 +155,17 @@ smaller* than the recursive-drift loop. So:
   iterations) — only after v1 establishes a measurable single-shot
   delta. Building v2 before v1 measures is the precedent error.
 
-**Resolved by fox (2026-05-19):** control = Hermes-solo; judge =
-external, deterministic, paraphrase-tolerant (gold-fact key).
+**Resolved by fox (2026-05-19):** control = Hermes-solo; judge = a
+**strong SOTA model, external to the experiment**, blinded +
+reference-grounded (§4b); atomic-only limit lifted; gold-key kept
+only as a cheap cross-check.
 **Still open (genuine fox calls, not autonomous):** (a) the
 fabrication-bait fixture population — corpus-specific / stale /
 false-premise / known-fabrication-bait, *not* common-knowledge where
 parametric memory wins anyway (the resolving-fixture lesson:
-prevalence ≠ what isolates the claim); (b) gold-key construction
-(which atomic-fact shapes, alias handling); (c) spend now vs parked.
-No design past §2 gate-1 / §4b until (a)–(c) ruled.
+prevalence ≠ what isolates the claim); (b) **which SOTA judge model**
+— and the spend/egress note: an external frontier judge is
+experiment-time API cost + Operation-Voyeur egress (low-sensitivity
+2003-Wikipedia QA, but a fox call, and the judge transcript must be
+logged for replay); (c) spend now vs parked. No design past §2
+gate-1 / §4b until (a)–(c) ruled.