russell@unturf.com
2c98fc964e
Three workstreams, full suite 2482 passed, experimental paths default-OFF. #000055 — Windows quickstart without make tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes; README Windows section rewritten. Quickstart needs only Python 3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart subset; drift-pinned by tests/test_tasks_runner.py. #000001 §7 Phase 0 — deterministic cross-language guard arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an es function-word stoppack. Fail-closed to UNGROUNDED before retrieval/LLM (mirrors the quantifier reject-DAG) when no content token survives, else strips es stopwords from the retrieval query only. English path byte-identical by construction. Default OFF (crosslang_guard_enabled). Measured: the anarcocapitalismo field case 10.4s -> 1.6s. #000056 — Operation Sandwich (cross-language grounding) arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised singleton (fixes the 88%-engine-error concurrency defect), manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich = translate query in (retrieval + LLM prompt) -> English answer -> UNTOUCHED verifier grounds English-vs-English -> translate the verified answer out as display-only (banner-labelled, zero grounding). question_hash + verifier_policy_hash invariant; MT engine identity binds into RetrievalPlan, not governance. CLI --crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask default OFF (measured net-negative at bench scale). Fan-out bench (bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support baseline; the round-trip predictor was tried and refuted; the entity-mask lever failed at scale (corpus-title anchoring untried). CLAUDE.md: cross-language bright-line convention + module map. Pre-existing modified diagram files are intentionally excluded.
71 lines
2.8 KiB
Python
71 lines
2.8 KiB
Python
#!/usr/bin/env python3
|
|
"""Round-trip drift analysis for #000056 (the pattern, deterministic).
|
|
|
|
For each bench pair: en --[opus-mt-en-es]--> es --[opus-mt-es-en]--> en'
|
|
The es→en leg is the SANDWICH'S ACTUAL edge-IN. If en' preserves the
|
|
content nouns of en, the sandwich feeds FTS5 the right terms and
|
|
grounding tracks the English baseline. If a content noun is lost
|
|
(Hamlet→"village", "New London"→"new London"), retrieval can't find
|
|
the article no matter how good Hermes is — the failure is upstream of
|
|
grounding, in named-entity-preserving MT.
|
|
|
|
Buckets by content-token recall of en' vs en (stopword-stripped):
|
|
CLEAN >= 0.80 COLLAPSE < 0.40 DRIFT otherwise
|
|
No Hermes, no network — offline + reproducible.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(__file__).resolve().parents[1]
|
|
sys.path.insert(0, str(ROOT))
|
|
from arborist.qa.mt import OpusMTTranslator # noqa: E402
|
|
|
|
_W = re.compile(r"[A-Za-z][A-Za-z0-9]*")
|
|
_STOP = set("the a an is are was were be of to in on at for with by from as "
|
|
"and or what who where when why how which this that who whom did "
|
|
"do does has have had can could would will who're were name named "
|
|
"between play wrote write written who's".split())
|
|
|
|
|
|
def toks(s: str) -> set[str]:
|
|
return {w.lower() for w in _W.findall(s) if w.lower() not in _STOP and len(w) > 1}
|
|
|
|
|
|
def main() -> int:
|
|
pairs = json.loads((ROOT / "bench" / "qa_questions_es_map.json").read_text())
|
|
tr = OpusMTTranslator()
|
|
rows = []
|
|
for p in pairs:
|
|
en, es = p["en"], p["es"]
|
|
back = tr.translate(es, "es", "en") # the sandwich's edge-IN
|
|
a, b = toks(en), toks(back)
|
|
recall = round(len(a & b) / len(a), 2) if a else 1.0
|
|
bucket = "CLEAN" if recall >= 0.80 else "COLLAPSE" if recall < 0.40 else "DRIFT"
|
|
lost = sorted(a - b)
|
|
rows.append({"en": en, "es": es, "back": back, "recall": recall,
|
|
"bucket": bucket, "lost_tokens": lost})
|
|
out = ROOT / "bench" / "qa_results" / "es_roundtrip.json"
|
|
out.write_text(json.dumps(rows, ensure_ascii=False, indent=2) + "\n")
|
|
from collections import Counter
|
|
c = Counter(r["bucket"] for r in rows)
|
|
n = len(rows)
|
|
print(f"n={n} CLEAN={c['CLEAN']} ({c['CLEAN']/n:.0%}) "
|
|
f"DRIFT={c['DRIFT']} ({c['DRIFT']/n:.0%}) "
|
|
f"COLLAPSE={c['COLLAPSE']} ({c['COLLAPSE']/n:.0%})")
|
|
print("\n-- COLLAPSE (named entity / key noun lost on round-trip) --")
|
|
for r in rows:
|
|
if r["bucket"] == "COLLAPSE":
|
|
print(f" en : {r['en']}\n back: {r['back']} lost={r['lost_tokens']}")
|
|
print("\n-- a few CLEAN --")
|
|
for r in [x for x in rows if x["bucket"] == "CLEAN"][:6]:
|
|
print(f" {r['en']} ==~ {r['back']}")
|
|
print(f"\nwrote {out}")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|