engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000026 Phase 2: real-shard baseline + lazy concept loading

Browse source 
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.

## Lazy concept_relations loading

Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.

Refactored arborist/concepts/query.py:

- New _load_neighbors_for(shards_dir, tokens) — targeted
  WHERE token IN (...) OR target IN (...) query. Returns just the
  direct synonym neighborhood for the given tokens (~300 rows for
  a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
  ~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
  expansion exceeds max_total (the cap). Most queries never reach
  the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
  don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
  loaders. Eager _load_indices / _get_indices kept for any
  back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.

All 14 concept tests pass unchanged — the contract is preserved.

Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):

  metric           pre-fix   post-Phase-1   post-lazy-concepts
  wall_ms          14,500    13,400         9,300       (-36%)
  search_ms         9,900    10,600         5,000       (-49%)
  SQLite executes  10,623     3,687         3,708         ~same
  synonym_expand     2,966     2,840           ~0       (lazy hit)

Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).

## Real-shard baseline artifact (Phase 2)

bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:

- bench/results/real-shard-baseline.json (durable; commit_sha,
  shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)

Question set in bench/fixtures/real-shard-baseline-v1.jsonl:

  virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
  (rivalry path), and two canonical-projection cases (math + logic
  preflight short-circuit).

First baseline run (commit e78814c plus this fan-out, BURN=1):

  audit_mode      n   notes
  STRICT          4   (virt-back, France, Mac OS X, Linux)
  HYBRID          2   (Microsoft founder, AMD/Intel)
  CANONICAL       2   (0.1+0.2, A IMPL B; <1 ms each)

  wall median   4.1 s   (range 0.6 ms – 7.2 s)
  primary used  4 / 8

`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).

Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.

`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.

## Test status

Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
This commit is contained in:
russell@unturf.com 2026-05-08 13:10:58 -04:00
parent e78814ca29
commit 8b1de208c2
No known key found for this signature in database
9 changed files with 806 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

9
bench/fixtures/real-shard-baseline-v1.jsonl Normal file
View file

@ -0,0 +1,9 @@
{"_meta":{"name":"real-shard-baseline-v1","notes":"Question set for #000026 Phase 2. Stable across runs so deltas are interpretable. Mix of wiki-primary, crawl-primary, and canonical-projection (math/logic) cases. Authored 2026-05-08."}}
{"id":"baseline-001","question":"who wrote virt-back?","expected_routing":"crawl_primary","note":"reviewer's first real-world success case; cited evidence is copyright footer (warrant-quality finding)"}
{"id":"baseline-002","question":"What is the capital of France?","expected_routing":"wiki_primary","note":"trivial wiki retrieval; sanity check"}
{"id":"baseline-003","question":"What is Mac OS X?","expected_routing":"wiki_primary","note":"named-entity retrieval through 2003 dump"}
{"id":"baseline-004","question":"What programming language was the Linux kernel written in?","expected_routing":"wiki_primary","note":"specific factual claim; well-grounded"}
{"id":"baseline-005","question":"Who founded Microsoft?","expected_routing":"wiki_primary","note":"named-entity authorship"}
{"id":"baseline-006","question":"What is the relationship between AMD and Intel?","expected_routing":"wiki_primary","note":"rivalry-pair tokens; exercises rivalry_excluded path"}
{"id":"baseline-007","question":"0.1 + 0.2","expected_routing":"canonical_projection","note":"arithmetic@v1 short-circuit; should bypass RAG entirely"}
{"id":"baseline-008","question":"A IMPL B","expected_routing":"canonical_projection","note":"logic-kernel@v1 short-circuit; should bypass RAG entirely"}

199
bench/results/real-shard-baseline.json Normal file
View file

@ -0,0 +1,199 @@
{
"schema_version": "real-shard-baseline-v1",
"timestamp_utc": "2026-05-08T17:09:43+00:00",
"commit_sha": "e78814ca29ed4114572f2410b2615733d4b715f6",
"git_dirty": true,
"shards_dir": "/home/fox/.arborist/shards",
"shards_fingerprint": "d93bed2c48c7ac47b22286ac77508d5519fec2f3685d470966642eef4285cb5e",
"model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
"endpoint": "https://hermes.ai.unturf.com/v1",
"burn": true,
"fixtures_path": "bench/fixtures/real-shard-baseline-v1.jsonl",
"summary": {
"questions": 8,
"wall_ms_median": 4140.9,
"wall_ms_max": 7212.4,
"wall_ms_min": 0.6,
"by_audit_mode": {
"STRICT": 4,
"HYBRID": 2,
"CANONICAL_PROJECTION": 2
},
"by_status": {
"cache_miss_then_written": 6,
"canonical_projection": 2
},
"primary_source_used_count": 4,
"canonical_projection_count": 2
},
"queries": [
{
"question": "who wrote virt-back?",
"status": "cache_miss_then_written",
"audit_mode": "STRICT",
"verifier_method": "quote",
"n_verified": 1,
"n_quotes": 1,
"pi_star_ref": null,
"lookup_path": "miss",
"wall_ms": 6309.6,
"total_ms": 6297.3,
"search_ms": 5002.6,
"llm_ms": 1213.1,
"verify_ms": null,
"canonical_preflight_ms": null,
"sources_count": 8,
"primary_source_used": true,
"answer_text": "Based on the provided sources, the author of virt-back is Russell Ballestrini. The source states:\n\n\"Over the weekend I wrote virt-back, a backup utility for QEMU, KVM, XEN, or Virtualbox guests.\"",
"expected_routing": "crawl_primary",
"note": "reviewer's first real-world success case; cited evidence is copyright footer (warrant-quality finding)"
},
{
"question": "What is the capital of France?",
"status": "cache_miss_then_written",
"audit_mode": "STRICT",
"verifier_method": "quote",
"n_verified": 1,
"n_quotes": 1,
"pi_star_ref": null,
"lookup_path": "miss",
"wall_ms": 3867.2,
"total_ms": 3866.5,
"search_ms": 2663.1,
"llm_ms": 1123.0,
"verify_ms": null,
"canonical_preflight_ms": null,
"sources_count": 5,
"primary_source_used": true,
"answer_text": "The capital of France is Paris. This is stated in the source: \"With the liberation of Paris in 1944, Charles de Gaulle established the Provisional Government of the French Republic, restoring Paris as the French capital.\"",
"expected_routing": "wiki_primary",
"note": "trivial wiki retrieval; sanity check"
},
{
"question": "What is Mac OS X?",
"status": "cache_miss_then_written",
"audit_mode": "STRICT",
"verifier_method": "quote",
"n_verified": 1,
"n_quotes": 1,
"pi_star_ref": null,
"lookup_path": "miss",
"wall_ms": 7212.4,
"total_ms": 7211.4,
"search_ms": 1485.5,
"llm_ms": 5644.9,
"verify_ms": null,
"canonical_preflight_ms": null,
"sources_count": 4,
"primary_source_used": true,
"answer_text": "Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, Mac OS X has been included with all new Macintosh computer systems. It is the successor to Mac OS 9, ",
"expected_routing": "wiki_primary",
"note": "named-entity retrieval through 2003 dump"
},
{
"question": "What programming language was the Linux kernel written in?",
"status": "cache_miss_then_written",
"audit_mode": "STRICT",
"verifier_method": "quote",
"n_verified": 1,
"n_quotes": 1,
"pi_star_ref": null,
"lookup_path": "miss",
"wall_ms": 4140.9,
"total_ms": 4140.0,
"search_ms": 2088.9,
"llm_ms": 1989.7,
"verify_ms": null,
"canonical_preflight_ms": null,
"sources_count": 8,
"primary_source_used": false,
"answer_text": "The Linux kernel was written in the C programming language. From the source:\n\"The Linux kernel is an operating system kernel used by the Linux family of Unix-like operating systems. It is one of the most prominent examples of free and open ",
"expected_routing": "wiki_primary",
"note": "specific factual claim; well-grounded"
},
{
"question": "Who founded Microsoft?",
"status": "cache_miss_then_written",
"audit_mode": "HYBRID",
"verifier_method": "entity",
"n_verified": 4,
"n_quotes": 4,
"pi_star_ref": null,
"lookup_path": "miss",
"wall_ms": 2248.7,
"total_ms": 2247.6,
"search_ms": 1134.6,
"llm_ms": 1070.8,
"verify_ms": null,
"canonical_preflight_ms": null,
"sources_count": 5,
"primary_source_used": true,
"answer_text": "Based on the provided sources, I don't have enough information to determine who founded Microsoft. The sources mention Microsoft Research, Microsoft TechNet, Microsoft Hardware, and Microsoft Dynamics NAV, but do not contain any verbatim qu",
"expected_routing": "wiki_primary",
"note": "named-entity authorship"
},
{
"question": "What is the relationship between AMD and Intel?",
"status": "cache_miss_then_written",
"audit_mode": "HYBRID",
"verifier_method": "paraphrase",
"n_verified": 4,
"n_quotes": 5,
"pi_star_ref": null,
"lookup_path": "miss",
"wall_ms": 4327.0,
"total_ms": 4326.3,
"search_ms": 1636.2,
"llm_ms": 2623.3,
"verify_ms": null,
"canonical_preflight_ms": null,
"sources_count": 8,
"primary_source_used": false,
"answer_text": "The relationship between AMD and Intel is that of competitors in the microprocessor market. AMD filed an antitrust lawsuit against Intel in June 2005, alleging that Intel engaged in unfair competition by offering rebates to Japanese PC manu",
"expected_routing": "wiki_primary",
"note": "rivalry-pair tokens; exercises rivalry_excluded path"
},
{
"question": "0.1 + 0.2",
"status": "canonical_projection",
"audit_mode": "CANONICAL_PROJECTION",
"verifier_method": "canonical_projection",
"n_verified": 1,
"n_quotes": 1,
"pi_star_ref": "arithmetic@v1",
"lookup_path": "preflight_canonical",
"wall_ms": 0.9,
"total_ms": 0.3,
"search_ms": null,
"llm_ms": null,
"verify_ms": null,
"canonical_preflight_ms": 0.3,
"sources_count": 0,
"primary_source_used": false,
"answer_text": "3/10",
"expected_routing": "canonical_projection",
"note": "arithmetic@v1 short-circuit; should bypass RAG entirely"
},
{
"question": "A IMPL B",
"status": "canonical_projection",
"audit_mode": "CANONICAL_PROJECTION",
"verifier_method": "canonical_projection",
"n_verified": 1,
"n_quotes": 1,
"pi_star_ref": "logic-kernel@v1",
"lookup_path": "preflight_canonical",
"wall_ms": 0.6,
"total_ms": 0.1,
"search_ms": null,
"llm_ms": null,
"verify_ms": null,
"canonical_preflight_ms": 0.1,
"sources_count": 0,
"primary_source_used": false,
"answer_text": "(NOT A OR B)",
"expected_routing": "canonical_projection",
"note": "logic-kernel@v1 short-circuit; should bypass RAG entirely"
}
]
}

30
bench/results/real-shard-baseline.md Normal file
View file

@ -0,0 +1,30 @@
# Real-shard baseline — 2026-05-08T17:09:43+00:00
**Commit:** `e78814ca29ed` (dirty)
**Shards fingerprint:** `d93bed2c48c7ac47…`
**Shards directory:** `/home/fox/.arborist/shards`
## Summary
- Questions: **8**
- Wall median: **4140.9 ms**
- Wall range: 0.6 – 7212.4 ms
- Primary source used: 4/8
- Canonical-projection short-circuits: 2
**Audit-mode distribution:** `CANONICAL_PROJECTION` 2, `HYBRID` 2, `STRICT` 4
**Status distribution:** `cache_miss_then_written` 6, `canonical_projection` 2
## Per-question
| # | Question | Audit | Status | Wall (ms) | Method |
|---|----------|-------|--------|-----------|--------|
| 1 | who wrote virt-back? | STRICT | cache_miss_then_written | 6309.6 | quote |
| 2 | What is the capital of France? | STRICT | cache_miss_then_written | 3867.2 | quote |
| 3 | What is Mac OS X? | STRICT | cache_miss_then_written | 7212.4 | quote |
| 4 | What programming language was the Linux kernel written in? | STRICT | cache_miss_then_written | 4140.9 | quote |
| 5 | Who founded Microsoft? | HYBRID | cache_miss_then_written | 2248.7 | entity |
| 6 | What is the relationship between AMD and Intel? | HYBRID | cache_miss_then_written | 4327.0 | paraphrase |
| 7 | 0.1 + 0.2 | CANONICAL_PROJECTION | canonical_projection | 0.9 | canonical_projection |
| 8 | A IMPL B | CANONICAL_PROJECTION | canonical_projection | 0.6 | canonical_projection |

0
bench/scripts/__init__.py Normal file
View file

335
bench/scripts/real_shard_baseline.py Normal file
View file

@ -0,0 +1,335 @@
"""Real-shard workload baseline (#000026 Phase 2).
Runs a fixed question set against the operator's real shard
directory and emits one JSON artifact + a short markdown summary.
Usage:
python -m bench.scripts.real_shard_baseline \
--shards-dir ~/.arborist/shards \
--fixtures bench/fixtures/real-shard-baseline-v1.jsonl \
--out bench/results/real-shard-baseline.json
Or via the Makefile:
make bench-real-shard
Outputs:
bench/results/real-shard-baseline.json one row per question
bench/results/real-shard-baseline.md markdown summary
Captured per question:
- question, audit_mode, verifier_method, n_verified, n_quotes
- wall_ms, search_ms, llm_ms, canonical_ms (preflight short-circuit)
- primary_source_used, sources_count
- status (canonical_projection | cache_hit | cache_miss_then_written
| no_sources | broad_quantifier_rejected | ...)
This is observation, not enforcement. Baselines NEVER gate CI per the
ticket's hard constraint. The artifact is for: (a) confirming a fix
moved the needle, (b) seeding ForkScore comparisons, (c) noting
warrant-quality findings worth tickets.
"""
from __future__ import annotations
import argparse
import datetime as _dt
import hashlib
import json
import os
import subprocess
import sys
import time
from pathlib import Path
def _shard_fingerprint(shards_dir: Path) -> str:
"""SHA-256 over (path, size, mtime_ns) for every *.db in shards_dir.
Stable across re-opens; changes when any shard mutates."""
if not shards_dir.is_dir():
return ""
parts = []
for p in sorted(shards_dir.glob("*.db")):
st = p.stat()
parts.append(f"{p.name}\t{st.st_size}\t{st.st_mtime_ns}")
return hashlib.sha256("\n".join(parts).encode("utf-8")).hexdigest()
def _commit_sha() -> str:
try:
return subprocess.check_output(
["git", "rev-parse", "HEAD"], cwd=Path(__file__).parent
).decode().strip()
except Exception:
return ""
def _git_dirty() -> bool:
try:
out = subprocess.check_output(
["git", "status", "--porcelain"], cwd=Path(__file__).parent
).decode()
return bool(out.strip())
except Exception:
return False
def _read_fixtures(path: Path) -> list[dict]:
rows = []
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
obj = json.loads(line)
if "_meta" in obj:
continue
rows.append(obj)
return rows
def _run_one(
*,
question: str,
shards_dir: Path,
qa_db: Path,
chat_client,
model_id: str,
burn: bool,
) -> dict:
"""Execute one query through the real query() entry point and
extract a small, stable summary dict."""
from arborist.qa.query import query as _query
from arborist.qa.client import StubClient # noqa: F401 — type hint use
t0 = time.monotonic()
result = _query(
question=question,
qa_db=qa_db,
chat_client=chat_client,
model_id=model_id,
shards_dir=shards_dir,
burn_existing=burn,
)
elapsed_ms = round((time.monotonic() - t0) * 1000, 1)
timings = result.get("timings") or {}
sources = result.get("sources") or []
primary_used = any(
s.get("source_role") in (
"primary", "primary_answer", "primary_answer_source"
) and s.get("used") is not False
for s in sources
)
return {
"question": question,
"status": result.get("status"),
"audit_mode": result.get("audit_mode"),
"verifier_method": result.get("verifier_method"),
"n_verified": result.get("n_verified"),
"n_quotes": result.get("n_quotes"),
"pi_star_ref": result.get("pi_star_ref"),
"lookup_path": result.get("lookup_path"),
"wall_ms": elapsed_ms,
"total_ms": timings.get("total_ms"),
"search_ms": timings.get("search_ms"),
"llm_ms": timings.get("llm_ms"),
"verify_ms": timings.get("verify_ms"),
"canonical_preflight_ms": timings.get("canonical_preflight_ms"),
"sources_count": len(sources),
"primary_source_used": primary_used,
"answer_text": (result.get("answer_text") or "")[:240],
}
def _summarize(rows: list[dict]) -> dict:
"""Aggregate the per-question rows into headline metrics."""
n = len(rows)
walls = [r["wall_ms"] for r in rows if isinstance(r.get("wall_ms"), (int, float))]
walls_sorted = sorted(walls)
median = walls_sorted[len(walls_sorted) // 2] if walls_sorted else None
by_audit: dict[str, int] = {}
by_status: dict[str, int] = {}
for r in rows:
a = r.get("audit_mode") or "—"
s = r.get("status") or "—"
by_audit[a] = by_audit.get(a, 0) + 1
by_status[s] = by_status.get(s, 0) + 1
return {
"questions": n,
"wall_ms_median": median,
"wall_ms_max": max(walls) if walls else None,
"wall_ms_min": min(walls) if walls else None,
"by_audit_mode": by_audit,
"by_status": by_status,
"primary_source_used_count": sum(
1 for r in rows if r.get("primary_source_used")
),
"canonical_projection_count": sum(
1 for r in rows if r.get("status") == "canonical_projection"
),
}
def _markdown(rows: list[dict], summary: dict, meta: dict) -> str:
lines = [
"# Real-shard baseline — " + meta["timestamp_utc"],
"",
f"**Commit:** `{meta['commit_sha'][:12]}`"
+ (" (dirty)" if meta.get("git_dirty") else ""),
f"**Shards fingerprint:** `{meta['shards_fingerprint'][:16]}…`",
f"**Shards directory:** `{meta['shards_dir']}`",
"",
"## Summary",
"",
f"- Questions: **{summary['questions']}**",
f"- Wall median: **{summary['wall_ms_median']} ms**",
f"- Wall range: {summary['wall_ms_min']} – {summary['wall_ms_max']} ms",
f"- Primary source used: {summary['primary_source_used_count']}/"
f"{summary['questions']}",
f"- Canonical-projection short-circuits: "
f"{summary['canonical_projection_count']}",
"",
"**Audit-mode distribution:** "
+ ", ".join(f"`{k}` {v}" for k, v in sorted(summary["by_audit_mode"].items())),
"",
"**Status distribution:** "
+ ", ".join(f"`{k}` {v}" for k, v in sorted(summary["by_status"].items())),
"",
"## Per-question",
"",
"| # | Question | Audit | Status | Wall (ms) | Method |",
"|---|----------|-------|--------|-----------|--------|",
]
for i, r in enumerate(rows, 1):
q = r["question"]
if len(q) > 60:
q = q[:57] + "…"
lines.append(
f"| {i} | {q} | "
f"{r.get('audit_mode') or '—'} | "
f"{r.get('status') or '—'} | "
f"{r.get('wall_ms') or '—'} | "
f"{r.get('verifier_method') or r.get('pi_star_ref') or '—'} |"
)
return "\n".join(lines) + "\n"
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
p.add_argument(
"--shards-dir", type=Path, required=True,
help="path to the shard directory (e.g. ~/.arborist/shards)",
)
p.add_argument(
"--fixtures", type=Path,
default=Path("bench/fixtures/real-shard-baseline-v1.jsonl"),
help="JSONL question set (default: real-shard-baseline-v1)",
)
p.add_argument(
"--out", type=Path,
default=Path("bench/results/real-shard-baseline.json"),
help="JSON output path",
)
p.add_argument(
"--md-out", type=Path,
default=Path("bench/results/real-shard-baseline.md"),
help="markdown summary path",
)
p.add_argument(
"--endpoint", default=os.environ.get(
"ARBORIST_LLM_ENDPOINT", "https://hermes.ai.unturf.com/v1"
),
help="LLM endpoint URL",
)
p.add_argument(
"--model", default=os.environ.get(
"ARBORIST_LLM_MODEL",
"adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
),
help="model id (default $ARBORIST_LLM_MODEL or hermes-3)",
)
p.add_argument(
"--burn", action="store_true",
help="force fresh inference per question (skip cache)",
)
args = p.parse_args(argv)
if not args.shards_dir.is_dir():
print(f"error: shards-dir not found: {args.shards_dir}", file=sys.stderr)
return 2
if not args.fixtures.is_file():
print(f"error: fixtures not found: {args.fixtures}", file=sys.stderr)
return 2
# Defer LLM client construction until the args are known.
from arborist.qa.client import OpenAICompatibleClient
qa_db = args.shards_dir / "qa.db"
chat_client = OpenAICompatibleClient(base_url=args.endpoint)
fixtures = _read_fixtures(args.fixtures)
print(
f"running {len(fixtures)} questions against {args.shards_dir}",
file=sys.stderr,
)
rows = []
for fx in fixtures:
question = fx["question"]
print(f" → {question[:60]}", file=sys.stderr)
try:
row = _run_one(
question=question,
shards_dir=args.shards_dir,
qa_db=qa_db,
chat_client=chat_client,
model_id=args.model,
burn=args.burn,
)
except Exception as exc: # one bad question shouldn't kill the run
row = {
"question": question,
"status": "error",
"error": str(exc),
}
# Fold any operator notes from the fixture
if "expected_routing" in fx:
row["expected_routing"] = fx["expected_routing"]
if "note" in fx:
row["note"] = fx["note"]
rows.append(row)
summary = _summarize(rows)
meta = {
"schema_version": "real-shard-baseline-v1",
"timestamp_utc": _dt.datetime.now(_dt.timezone.utc).isoformat(
timespec="seconds"
),
"commit_sha": _commit_sha(),
"git_dirty": _git_dirty(),
"shards_dir": str(args.shards_dir),
"shards_fingerprint": _shard_fingerprint(args.shards_dir),
"model": args.model,
"endpoint": args.endpoint,
"burn": args.burn,
"fixtures_path": str(args.fixtures),
}
artifact = {**meta, "summary": summary, "queries": rows}
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(json.dumps(artifact, indent=2, ensure_ascii=False))
args.md_out.write_text(_markdown(rows, summary, meta))
print(f"wrote {args.out}", file=sys.stderr)
print(f"wrote {args.md_out}", file=sys.stderr)
print(json.dumps(summary, indent=2))
return 0
if __name__ == "__main__":
sys.exit(main())