engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/bench
History
russell@unturf.com 329337ac11
#000072: Path A v3 wire-up bench v5 — identical to baseline, no improvement 
Bench-v5 (5 themes × 3 questions × 2 paths, Hermes-3-8B) AFTER the
fold-stack lift + apply_title_boost wire-up to use _title_query_tokens
shows the SAME 12 regressions as the pre-wire-up baseline:

  - 5 wrong-primary picks (Dr Who, Albert/Ahmed/Alaric the third/first,
    Casa Batlló error)
  - 7 STRICT→HYBRID demotes on correct primaries (Spider-Man, Ampère,
    Dr Syn, Dr V64, Finnish Defence Forces, Hopewell Centre, labor
    economics)

Diagnosis: apply_title_boost only reranks docs ALREADY in the
candidate set. body-only retrieval (default with multi_route=False)
never surfaces "Doctor (Doctor Who)" so no fold-aware rerank can
promote it. Legacy surfaces it via the title route. FTS5 porter
stemmer handles plurals but NOT Dr→Doctor or third→III, so the
SqliteShardCorpus.fts_title method also doesn't help here without
the folds applied at retrieval-token-gen time, not rank time.

Updated ticket with the honest assessment: 5 Path A stages across
v1/v2/v3 have now proven legacy query() doesn't decompose into a
library of helpers. Three forward paths offered for fox to decide:
  A. leave default at providence, accept fold regressions (env
     escape hatch ARBORIST_LEGACY_QUERY=1 already shipped)
  B. flip default back to legacy, treat providence as
     infrastructure for cloud-query/corpus-query only
  C. keep both alive long-term — separate query2 command
2026-05-31 20:06:39 -04:00
..
batteries #000025 §10.11 + §10.13 + §10.14 — close the 5F battery 2026-05-11 07:41:37 -04:00
cloud_vs_local_results qa/corpus: shared title-boost (extras penalty) lifted into run_query 2026-05-31 08:43:24 -04:00
fixtures #000052 §3.2.2 step 3 part B: real-context MIS-CITE fixtures (n=20) — bge-large catches 100% of mis-cite at strictly 0% real-STRICT FP (the Zionist failure mode is fully covered with margin) 2026-05-13 15:27:58 -04:00
legacy_vs_providence_results #000072: Path A v3 wire-up bench v5 — identical to baseline, no improvement 2026-05-31 20:06:39 -04:00
results #000067 phase 2: 3rd "fts" pack kind for skip-rebuild hydrate 2026-05-26 19:13:01 -04:00
scripts feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale 2026-05-19 12:34:04 -04:00
slim_fts_parity_results bench: chunk_fetch_speed (apsw vs blob GET) + archive session bench results 2026-05-31 15:44:55 -04:00
three_way_results bench: chunk_fetch_speed (apsw vs blob GET) + archive session bench results 2026-05-31 15:44:55 -04:00
analyze_judge_disagreement.py feat(#000057): reconcile code judge against Opus — 4 calibrated rules 2026-05-19 18:14:38 -04:00
chunk_fetch_speed.py bench: chunk_fetch_speed (apsw vs blob GET) + archive session bench results 2026-05-31 15:44:55 -04:00
cloud_vs_local.py sidecar+bucket: extras penalty + raw-score merge for sidecar-only paths 2026-05-30 21:21:33 -04:00
cold_pack_roundtrip.py bench: cold-pack producer/consumer roundtrip recorder (#000061 + #46) 2026-05-26 16:19:54 -04:00
control_ab.py fix(#000057): control_ab header prints actual model/answer_mode/judge 2026-05-21 12:37:24 -04:00
control_sweep.py feat: STOCK V.1 two-mode config family + wire treatment arms to the pin 2026-05-21 10:15:26 -04:00
cross_model_selfplay.py bench: teacher-model-judge end-to-end — cross-judge with chat_template_kwargs reasoning disable 2026-05-31 12:43:18 -04:00
emergent_log.jsonl #000006 — +30 emergent cycles (2026-05-12); verifier-ladder health re-confirmed 2026-05-12 11:28:57 -04:00
es_delta.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
es_join_patterns.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
es_roundtrip_analysis.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
extract_audit_events.py #000065 step 1: routing helper + meta field + pre-migration snapshot 2026-05-26 13:13:44 -04:00
jaggedness.py feat(#000060): bench/jaggedness.py — deterministic retrieval jaggedness instrument 2026-05-21 08:38:07 -04:00
judge.py feat(#000057): fail-closed Opus judge gate — ARBORIST_JUDGE_ENABLE=1 to run 2026-05-19 17:26:06 -04:00
judge_code.py docs+code: ground §12 judge pipeline in the actual judge_code.py 2026-05-22 19:17:28 -04:00
legacy_vs_providence_bench.py #000072: bench-driven diagnosis — port fold-variants stack first 2026-05-31 19:07:58 -04:00
load_monitor.py feat: bench/load_monitor.py — stdlib request-load monitor for single-slot endpoints 2026-05-21 09:20:53 -04:00
make_es_questions.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
make_lang_questions.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
mine_questions.py fix(#000057): solo arm never ran — chat_completion(model=) was omitted 2026-05-19 11:49:30 -04:00
pre_migration_snapshot.py #000065 step 1: routing helper + meta field + pre-migration snapshot 2026-05-26 13:13:44 -04:00
prometheus_sigma_trigger_probe.py #000012 Phase 1c follow-through: wire #000037 §12 Trigger 1 probe to fork_score_branches 2026-05-11 06:56:09 -04:00
qa_questions.txt user_payload_layout: opt-in policy knob for question placement 2026-05-27 10:13:59 -04:00
qa_questions_accent.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_accent_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_amp.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_amp_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_brit.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_brit_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_canonical_witness_npower.txt three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c 2026-05-10 07:46:35 -04:00
qa_questions_es.txt feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_es_map.json feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_fr.txt feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_fr_map.json feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_honorific.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_honorific_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_hyphen.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_hyphen_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_metacog_subset.txt qa(#000011 + 4 more): SOFT_PREFLIGHT_HINT impl + 5-task fan-out 2026-05-03 23:00:56 -04:00
qa_questions_numeral.txt fix(#000057): restore numeral fixture clobbered by b573c59 (40 entries, not 5) 2026-05-19 11:12:12 -04:00
qa_questions_numeral_map.json fix(#000057): restore numeral fixture clobbered by b573c59 (40 entries, not 5) 2026-05-19 11:12:12 -04:00
qa_questions_progressive_and.txt bench: progressive-AND fixture + 2026-05-09 A/B baseline report 2026-05-10 06:35:18 -04:00
qa_questions_quantifier_baseline.txt bench(#000008): harness extension — FC rate, violation kinds, raw brackets 2026-05-02 18:35:08 -04:00
qa_questions_quantifier_subset.txt ticket(#000008): §12 dry-run bench findings + --policy harness flag 2026-05-03 08:39:20 -04:00
qa_questions_smoke.txt speed: pytest-xdist, bench smoke, concurrency default; UTF surrogate fix 2026-05-02 09:29:40 -04:00
qa_questions_stale.txt feat(#000057): parallel + incremental + huge-N control sweep 2026-05-19 13:00:45 -04:00
qa_questions_stale_map.json feat(#000057): parallel + incremental + huge-N control sweep 2026-05-19 13:00:45 -04:00
qa_questions_warrant_chain_aggressive.txt bench: aggressive warrant fixture confirms Phase 3 is rescue-only, not default-path 2026-05-10 09:53:21 -04:00
qa_questions_warrant_chain_paraphrase.txt bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus 2026-05-10 10:06:23 -04:00
qa_questions_warrant_chain_probe.txt bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture 2026-05-10 09:37:27 -04:00
qa_sweep.py #000068 Phase 2+3: bench + opt-in demote flag for missed-answer guard 2026-05-27 10:40:35 -04:00
recall_at_k.py feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
run.sh aborist/arborist 2026-05-07 09:31:49 -04:00
score_with_code_judge.py feat(#000057): bench/score_with_code_judge.py — code-judge rescore tool 2026-05-19 17:46:52 -04:00
shard_count_sweep.py #000065: pin M=4 + bench script + SQLite-alternative decision tree 2026-05-26 12:40:48 -04:00
slim_fts_parity_bench.py wallet/fts-sidecar: real FTS5 in the cloud sidecar; delete custom BM25 2026-05-31 10:43:03 -04:00
spatial_anchor_validation.py #000070: spatial-anchor pi*_w_object ticket + pre-review bench 2026-05-31 10:16:02 -04:00
spatial_anchor_validation_results.md #000070: spatial-anchor pi*_w_object ticket + pre-review bench 2026-05-31 10:16:02 -04:00
stock_v1.py feat: STOCK V.1 two-mode config family + wire treatment arms to the pin 2026-05-21 10:15:26 -04:00
teacher_judge.py bench: teacher-model-judge end-to-end — cross-judge with chat_template_kwargs reasoning disable 2026-05-31 12:43:18 -04:00
three_way_bench.py qa/corpus: shared title-boost (extras penalty) lifted into run_query 2026-05-31 08:43:24 -04:00
watt_bench.py fix(#000057): real token usage + cost per input/output separately 2026-05-21 11:41:01 -04:00
watt_calibrate.py feat(#000057): bench/watt_calibrate.py — separate prefill vs decode energy 2026-05-21 11:48:36 -04:00
watt_probe.py fix(#000057): measure power STATES, not a duty-cycle blend; guarantee cache miss 2026-05-21 10:58:56 -04:00