feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF

Three workstreams, full suite 2482 passed, experimental paths default-OFF.

#000055 — Windows quickstart without make
  tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
  README Windows section rewritten. Quickstart needs only Python
  3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
  subset; drift-pinned by tests/test_tasks_runner.py.

#000001 §7 Phase 0 — deterministic cross-language guard
  arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
  es function-word stoppack. Fail-closed to UNGROUNDED before
  retrieval/LLM (mirrors the quantifier reject-DAG) when no content
  token survives, else strips es stopwords from the retrieval query
  only. English path byte-identical by construction. Default OFF
  (crosslang_guard_enabled). Measured: the anarcocapitalismo field
  case 10.4s -> 1.6s.

#000056 — Operation Sandwich (cross-language grounding)
  arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
  singleton (fixes the 88%-engine-error concurrency defect),
  manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
  translate query in (retrieval + LLM prompt) -> English answer ->
  UNTOUCHED verifier grounds English-vs-English -> translate the
  verified answer out as display-only (banner-labelled, zero
  grounding). question_hash + verifier_policy_hash invariant; MT
  engine identity binds into RetrievalPlan, not governance. CLI
  --crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
  default OFF (measured net-negative at bench scale). Fan-out bench
  (bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
  baseline; the round-trip predictor was tried and refuted; the
  entity-mask lever failed at scale (corpus-title anchoring untried).

CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.

2026-05-18 12:12:23 -04:00

5.2 KiB

Raw Blame History

Ticket #000055 — Windows quickstart without `make` (`tasks.py` + `make.bat`)

Status: in progress — opened 2026-05-16, implementation in same commit. Opened: 2026-05-16 Asked by: fox ("we need bat files or some shit so we could avoid needing makefile for windows (but keep it as an option) windows uses py py3 for python3 etc we will test the quickstart on windows so fix it up"). Scope: One new pure-stdlib runner tasks.py at repo root + one small make.bat Windows shim + README Windows/Quickstart rewrite + one smoke/drift test. No change to arborist/, the schema, the Makefile recipes, or any audit/proof path. The Makefile stays the canonical Unix entry point and is left byte-for-byte unchanged ("keep it as an option"). Audience: fox + anyone running the quickstart on native Windows + maintainers who touch the quickstart and must keep two thin entry points in sync.

1. The gap, from the field

README.md Setup §Windows currently says native Windows is not supported — WSL2 only — because the Makefile uses bash idioms (for i in $(seq …); do … & done; wait, sed, backgrounding) and GNU-make conditionals, and the docs assume make, curl, bzip2.

fox wants the quickstart runnable on native Windows (cmd / PowerShell), no make, no WSL. Windows ships the py launcher (py -3), not python3.

2. What actually blocks Windows (and what does not)

Audited the quickstart targets against the artifact, not the docs:

Decompression: the 2003 dump is opened with Python's stdlib bz2 (arborist/sources/wikipedia.py:288). No external bzip2 needed — the README's "needs … bzip2" line is stale for the quickstart path.
Download: the only curl use in the quickstart is fetch-cur. Replaceable with stdlib urllib.request → no curl dependency.
Sharding/distill loops: bash for … & wait. Replaceable with subprocess.Popen fan-out + wait.
CLI: arborist is a console script (arborist.cli:main, pyproject.toml:120). On a Windows venv it lands at .venv\Scripts\arborist.exe; on POSIX .venv/bin/arborist.
Net: after this, the quickstart needs only Python 3.10+ and sqlite3 (sqlite3 ships with CPython) — exactly the repo's "fresh checkout needs only python3.12 + venv + sqlite3" ethos. No make, no bzip2, no curl, no bash.

3. Design — why one runner, not N `.bat` files

Five-step algorithm, step 2 (delete the part): hand-porting ~20 bash recipes into brittle per-target .bat scripts would (a) duplicate logic 1:1 with the Makefile in a second fragile dialect, (b) proliferate files, (c) drift fast (DRY-in-context). Instead:

tasks.py — one pure-stdlib runner implementing the quickstart subset only. Cross-platform (also runs on POSIX as python3 tasks.py …, but the Makefile remains the documented Unix path). Detects venv layout (Scripts vs bin) and the Python launcher (py -3 on Windows, python3 on POSIX; ARBORIST_PYTHON overrides). Accepts the same KEY=VALUE token style as make (query Q="…" JSON=1, inspect KEY=… JSON=1, crawl-ingest URL=… DEPTH=2) so the documented commands translate 1:1 — one doc form, both platforms.
make.bat — a ~10-line dispatcher so make <target> keeps working in Windows cmd (cmd searches the current dir for make.bat) and .\make.bat <target> in PowerShell. It only resolves a Python (py -3, fallback python) and forwards argv to tasks.py. No logic in the .bat.

This keeps the Windows surface to two files, one of them Python, mirrors the existing "Makefile is build glue, logic is Python" precedent, and gives Windows the exact make-style UX.

tasks.py is the quickstart subset, not the full 1170-line Makefile. Bench/π*/textbook/docs targets stay Makefile-only (Unix/WSL2). The runner's help enumerates exactly the supported targets so the boundary is self-documenting.

4. Targets covered

bootstrap fetch-cur ingest-cur-attached distill-shards-parallel distill-shards-tfidf-parallel query query-dry inspect falsify burn burn-kindergarten bootstrap-crawler crawl-ingest recrawl-check stats stats-shards verify search test help clean clean-db clean-data — the full README Quickstart (Wikipedia + crawl paths), "After the answer", and Setup blocks.

5. Discrepancy found (reported to fox)

README.md Setup §Bootstrap claims bootstrap installs the [dev,html] extras; Makefile:52 installs .[dev] only. Artifact wins (CLAUDE.md "artifact over instruction"): tasks.py bootstrap installs .[dev] to match the Makefile. README corrected in the same commit.

6. Drift control

tests/test_tasks_runner.py: importable on the host platform; help lists exactly the documented quickstart targets (a removed or renamed target fails the test); venv-layout + launcher detection verified for both os.name == "nt" and "posix" via monkeypatch. Two thin entry points are a known DRY cost — the test pins the target set so divergence from the documented quickstart is loud.

7. Out of scope

Full Makefile parity on Windows; bench/π*/NLI/textbook/docs targets; changing the Unix workflow (Makefile untouched, still canonical on POSIX).

5.2 KiB Raw Blame History

Ticket #000055 — Windows quickstart without make (tasks.py + make.bat)