diff --git a/README.md b/README.md index 5c0bd09..944aeaf 100644 --- a/README.md +++ b/README.md @@ -482,21 +482,26 @@ For diagrams of how these modules wire together, see `docs/diagrams/` (rendered Source papers (read first if confused): -- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). §13.8 covers the verifier; §13.9 covers claim-lattice / CTI; §13.4.11 covers the corpus-derived concept layer. +- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). §6 Aborist architecture, §7 layered verifier, §8 claim-lattice / CTI, §9 four-rung ladder, §13 decisions & constraints. - `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5). - `docs/concept-relations-design.md` — synonym & rivalry layer architecture + 1.6% storage tradeoff rationale. - `docs/mesh.md` + `docs/mesh-deploy.md` — protocol contract + two-host runbook for federation. -## Tests +## Tests & bench ``` -make test # 641+ tests, default suite (stdlib + pytest, no network) -make test-crawler # opt-in: tests for the verbatim crawler lift -make bench # ETL throughput across configs (serial / shared-WAL / attached) +make test # 641+ tests, default suite, ~10s with pytest -n auto +make test-crawler # opt-in: tests for the verbatim crawler lift +make test-live # gated: live QA quality fixtures against Hermes (~1 min) +make bench # ETL throughput across configs (serial / shared-WAL / attached) +make bench-qa # full QA-quality sweep, sample-shuffled, 71q × 3m × 3n at c=4 +make bench-qa-smoke # 5-question smoke fixture for prompt-iteration loops (~1-3 min) ``` The default suite never hits the network. The crawler suite is gated behind `make bootstrap-crawler` (installs the `[crawler]` extras). +`make bench-qa` writes JSONL + markdown into `bench/qa_results/.{jsonl,md}` (gitignored); design-log entries live in `docs/qa-modes-bench-.md`. The bench is stop/start-able via `--resume ` (same `--seed` required for shuffled-task-order alignment). `BENCH_QA_CONCURRENCY=N` Makefile variable overrides the default `4`. See `docs/bench-maxing.md` for the full speed playbook. + ## License License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists. diff --git a/docs/bench-maxing.md b/docs/bench-maxing.md index 9a8e6bc..cc01b61 100644 --- a/docs/bench-maxing.md +++ b/docs/bench-maxing.md @@ -107,6 +107,94 @@ during the JSON-mode hardening journey: - **Doc captures the journey, not just the end state.** `docs/qa-modes-bench-2026-04-30.md` shows the day's progression (baseline → retry → trim-and-verify → pointer-IDs → stop-sequence - → self-heal). Each row is a named failure + fix + bench delta. - Future-you (or a new agent) can read the doc and see WHY each - commit landed, not just WHAT. + → self-heal). `docs/qa-modes-bench-2026-05-02.md` continues the + journey through Rule 8 promotion, anchor-class warrant + generalization, per-mode context budgets, and the bench-harness + upgrades named below. Each row is a named failure + fix + bench + delta. Future-you (or a new agent) can read the docs and see WHY + each commit landed, not just WHAT. + +## Bench harness — the speed playbook + +Patterns added 2026-05-02. The bench is now the inner-loop tool; +treat it like a CI gate, not a once-a-week ritual. + +- **Sample-level shuffled scheduling, not cell-grouped.** + Cell-grouped scheduling (run all n samples of one question/mode + back-to-back before the next cell) confounds two things: + (1) it correlates the n samples — vLLM's continuous batcher fills + with same-cache_key requests, KV-cache locality and queue load + become shared signal across "samples 1-3"; (2) it starves the + batcher of diverse work, so per-call latency stretches and + aggregate throughput drops. Sample-level scheduling — every + `(q, mode, sample_idx)` is its own task, shuffled with a + deterministic seed — gives true i.i.d. variance for n>=3 and + delivers the batcher uncorrelated requests. Per-cell `Lock` dict + serializes the rare adjacent-same-cell case so burn-then-write + on the shared `cache_key` never races. Measured on 2026-05-02: + +58% throughput at the same concurrency. + +- **Persistent HTTP client.** The OpenAI-compatible client used to + build a fresh `httpx.Client` per `chat_completion` call, paying + a TLS handshake on every request. Move it to `__init__` so + HTTP/1.1 keep-alive holds the TCP+TLS connection across calls + from the same client instance. httpx's pool is thread-safe; one + client serves all worker threads. Saved 1-2 min on a 426-call + bench, more on longer sweeps. + +- **Concurrency tuning, empirically.** vLLM has a sweet spot — + more concurrent requests fill the batch better, but only up to + the point where per-call latency growth outpaces parallelism + gain. On `hermes.ai.unturf.com` the sweep result (15-task smoke + fixture, 2026-05-02): + + c=3 102s 8.8 tasks/min ← peak + c=4 106s 8.5 tasks/min ← within 4%; chosen for runs + c=5 119s 7.6 tasks/min ← 12% slower + c=6 185s 4.9 tasks/min ← 45% slower (saturation) + + Run a small smoke at multiple concurrency levels before + committing to a long full-bench. The peak shifts when vLLM is + upgraded, when the corpus changes the average context size, or + when other tenants load the endpoint differently. + +- **Per-mode `max_context_chars` from bench feedback.** The + bench's "recommended context budget" table (per-mode strict-rate + by context-size bucket) is the substrate's voice telling the + policy where each mode peaks. As of the 2026-05-02 bench: + quote 8-16KB, pointer 16-32KB, JSON 32-64KB. Those defaults + live in `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]` and + fold into `governance_policy_hash` so any retuning cleanly + partitions the cache namespace. Tighten the cap when bench + shows degradation past a smaller bucket; loosen it when JSON + starts paging in 64-128KB. The harness reports the table on + every run — let it drive the policy, not the other way around. + +- **`--seed` for reproducibility.** Same `--seed` reproduces the + shuffle. Useful when a regression bench needs to be re-run + against the exact same task order; useful when the `--resume` + path needs the post-resume order to align with the pre-kill + order. Default `--seed 0`. + +- **`--resume ` — stop/start-able.** A 60-90 minute + full bench is annoying to re-run from zero when something + interrupts (Ctrl-C, kernel panic, network outage). The bench + reads the existing JSONL, skips already-completed + `(q, mode, sample_idx)` triples, appends fresh rows to the same + file, and re-renders the markdown summary from the union. Same + `--seed` required so the remaining tasks come out in the + original order. Stops can now be cheap. + +- **Smoke fixture for inner-loop.** `bench/qa_questions_smoke.txt` + carries 5 questions covering all anchor classes — date, place, + relation, entity-list, why-cause — each chosen because pointer + mode currently fails 100% on them and JSON aces 100%. Run via + `make bench-qa-smoke` (~1-3 min wall-clock). The smoke is the + inner-loop tool for prompt iteration; the full 71-question + bench stays the scoreboard. The bench-as-build-gate doctrine: + smoke before full. + +- **`pytest -n auto` for the test suite.** `pytest-xdist` lands + the test suite in 10s instead of 36s on this hardware (3.6× + speedup). Wired via `make test`. Cycle-time enabler for + prompt-iteration loops where the test suite gates the bench.