russell@unturf.com
2c98fc964e
Three workstreams, full suite 2482 passed, experimental paths default-OFF. #000055 — Windows quickstart without make tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes; README Windows section rewritten. Quickstart needs only Python 3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart subset; drift-pinned by tests/test_tasks_runner.py. #000001 §7 Phase 0 — deterministic cross-language guard arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an es function-word stoppack. Fail-closed to UNGROUNDED before retrieval/LLM (mirrors the quantifier reject-DAG) when no content token survives, else strips es stopwords from the retrieval query only. English path byte-identical by construction. Default OFF (crosslang_guard_enabled). Measured: the anarcocapitalismo field case 10.4s -> 1.6s. #000056 — Operation Sandwich (cross-language grounding) arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised singleton (fixes the 88%-engine-error concurrency defect), manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich = translate query in (retrieval + LLM prompt) -> English answer -> UNTOUCHED verifier grounds English-vs-English -> translate the verified answer out as display-only (banner-labelled, zero grounding). question_hash + verifier_policy_hash invariant; MT engine identity binds into RetrievalPlan, not governance. CLI --crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask default OFF (measured net-negative at bench scale). Fan-out bench (bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support baseline; the round-trip predictor was tried and refuted; the entity-mask lever failed at scale (corpus-title anchoring untried). CLAUDE.md: cross-language bright-line convention + module map. Pre-existing modified diagram files are intentionally excluded.
86 lines
3.2 KiB
Python
86 lines
3.2 KiB
Python
#!/usr/bin/env python3
|
||
"""The real metric: per-question English-baseline → Spanish+sandwich
|
||
transition. Absolute es grounding % is uninterpretable on an
|
||
adversarial bench; what the sandwich *costs* is the transition.
|
||
|
||
usage: es_delta.py <en_baseline.jsonl> <es_sandwich.jsonl>
|
||
|
||
Joins on the en↔es map. Buckets each question:
|
||
PRESERVED en grounded (S/H) & es grounded
|
||
DOWNGRADE en STRICT & es HYBRID (partial cost)
|
||
LOST en grounded & es UNGROUNDED (the real cost)
|
||
N/A en UNGROUNDED (sandwich not at fault)
|
||
GAINED en UNGROUNDED & es grounded (noise/curio)
|
||
Then cross-tabs LOST vs the round-trip bucket — does drift predict
|
||
the *loss* transition (even though it didn't predict absolute %)?
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import sys
|
||
from collections import Counter
|
||
from pathlib import Path
|
||
|
||
ROOT = Path(__file__).resolve().parents[1]
|
||
RES = ROOT / "bench" / "qa_results"
|
||
GROUNDED = {"STRICT", "HYBRID"}
|
||
|
||
|
||
def load(p):
|
||
return [json.loads(x) for x in Path(p).read_text().splitlines() if x.strip()]
|
||
|
||
|
||
def main() -> int:
|
||
en_rows = load(sys.argv[1])
|
||
es_rows = load(sys.argv[2])
|
||
pairs = json.loads((ROOT / "bench" / "qa_questions_es_map.json").read_text())
|
||
rt = {r["es"]: r["bucket"] for r in
|
||
json.loads((RES / "es_roundtrip.json").read_text())}
|
||
en_am = {r["question"]: r.get("audit_mode") for r in en_rows}
|
||
es_am = {r["question"]: r.get("audit_mode") for r in es_rows}
|
||
|
||
cls = Counter()
|
||
lost_by_bucket = Counter()
|
||
lost_list, downgrade_list = [], []
|
||
for p in pairs:
|
||
en, es = p["en"], p["es"]
|
||
a, b = en_am.get(en), es_am.get(es)
|
||
if a is None or b is None:
|
||
cls["MISSING"] += 1
|
||
continue
|
||
if a not in GROUNDED:
|
||
cls["N/A (en ungrounded)"] += 1
|
||
if b in GROUNDED:
|
||
cls[" └ of which GAINED"] += 1
|
||
continue
|
||
if b not in GROUNDED:
|
||
cls["LOST"] += 1
|
||
lost_by_bucket[rt.get(es, "?")] += 1
|
||
lost_list.append((en, es, a, b))
|
||
elif a == "STRICT" and b == "HYBRID":
|
||
cls["DOWNGRADE (S→H)"] += 1
|
||
downgrade_list.append((en, es))
|
||
else:
|
||
cls["PRESERVED"] += 1
|
||
|
||
n = len(pairs)
|
||
en_g = sum(1 for p in pairs if en_am.get(p["en"]) in GROUNDED)
|
||
es_g = sum(1 for p in pairs if es_am.get(p["es"]) in GROUNDED)
|
||
print(f"n={n} EN-baseline grounded={en_g} ({en_g/n:.0%}) "
|
||
f"ES+sandwich grounded={es_g} ({es_g/n:.0%}) "
|
||
f"net Δ={es_g-en_g:+d}")
|
||
print("\ntransition (only en-grounded questions can be 'LOST'):")
|
||
for k, v in cls.most_common():
|
||
print(f" {k:<24} {v}")
|
||
print("\nLOST × round-trip bucket (does drift predict the LOSS?):")
|
||
for bk in ("CLEAN", "DRIFT", "COLLAPSE", "?"):
|
||
if lost_by_bucket.get(bk):
|
||
print(f" {bk:<9} {lost_by_bucket[bk]}")
|
||
print("\n-- LOST questions (en grounded, es+sandwich UNGROUNDED) --")
|
||
for en, es, a, b in lost_list:
|
||
print(f" [{a}→{b}] en={en}\n es={es} rt={rt.get(es)}")
|
||
return 0
|
||
|
||
|
||
if __name__ == "__main__":
|
||
raise SystemExit(main())
|