engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/bench/results
History
russell@unturf.com 9d0015b4d4
feat(#000060): bench/jaggedness.py — deterministic retrieval jaggedness instrument 
v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.

Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).

make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
2026-05-21 08:38:07 -04:00
..
5f-threshold-calibration-2026-05-11.md #000047 — close: delta_aggregator knob on ForkScore (Option D) 2026-05-11 08:27:38 -04:00
baseline-falsification-hard.json #000048 step 2.1 — verify_quotes entity salient-token-disagreement gate 2026-05-11 13:57:45 -04:00
citation-aliases-batch-2026-05-10.md citation-aliases batch: 18 -> 35 / 92 records resolve (38%) 2026-05-10 06:58:27 -04:00
claim-pack-retrieval-lift-2026-05-09.md bench/results: claim-pack retrieval lift measurement (2026-05-09) 2026-05-09 13:38:04 -04:00
full-bench-qa-2026-05-10.md bench/full-bench-qa: no regression after substrate+Phase 3 sprint (675 cells) 2026-05-10 13:42:12 -04:00
full-warrant-resolution-2026-05-10.md docs: refresh stale alias counts (residual 40/54 → 74) 2026-05-10 12:27:23 -04:00
grinstead-snell-and-cascade-tuning-2026-05-10.md Grinstead-Snell ingest + cascade tuning batch: 68 -> 77/92 (84%) 2026-05-10 08:03:11 -04:00
hilbert-pillar-iv-fully-resolved-2026-05-09.md ticket #000042: 4 geometry aliases resolve all 18 Hilbert pillar IV records 2026-05-09 20:18:58 -04:00
jaggedness_2026-05-21T01-56-17Z.json feat(#000060): bench/jaggedness.py — deterministic retrieval jaggedness instrument 2026-05-21 08:38:07 -04:00
lambda-and-pm-additions-2026-05-10.md PLFA + SF-LF + PM Vol 1 ingest: 62 -> 68/92 (74%) — pillar IX nearly closed 2026-05-10 07:45:40 -04:00
medium-confidence-aliases-and-peano-2026-05-10.md medium-conf aliases + Peano ingest + title backfill: 35 -> 54/92 (59%) 2026-05-10 07:14:44 -04:00
nli-shadow-grid-n1-fixtures-vs-realstrict.json #000049 §7 #23: grid sweep flips the verdict — the lexical-candidate NLI veto PASSES the gate 2026-05-12 18:12:36 -04:00
nli-shadow-grid-n1-synth-vs-realstrict.json #000049 §7 #23: grid sweep flips the verdict — the lexical-candidate NLI veto PASSES the gate 2026-05-12 18:12:36 -04:00
nli-shadow-grid-n1-synth-vs-synth.json #000049 §7 #23: grid sweep flips the verdict — the lexical-candidate NLI veto PASSES the gate 2026-05-12 18:12:36 -04:00
nli-shadow-grid-n3-synth-vs-realstrict.json #000049 §7 #24: n=3 confirmation — §7 #23 was a small-FP-sample artifact; the no-stone-unturned mega-grid found the config that survives 2026-05-12 18:38:43 -04:00
nli-shadow-grid-n5-synth-vs-realstrict.json #000049 §7 #25: n=5 confirmation (444 STRICT cells) — verdict settles; the large models are the robust ones, MiniLM-cost-pick overturned 2026-05-12 19:42:55 -04:00
nli-shadow-grid-pooled808-synth-vs-realstrict.json #000049 §7 #26: mining for real recombinations + the pooled-808-STRICT check — verdict settles on margin@0.995 2026-05-12 21:11:35 -04:00
nli-shadow-grid-realctx25-vs-pooled808.json #000049 §7 #27: recall-side realism check — synthetic 28/28 was a 4x over-estimate; bart-large-mnli/k=12/max/θc=0.999 is the new fp=0 leader on real haystacks at 0.48 recall (vs §7 #26's claimed 28/28 → 0.24 on real) 2026-05-13 06:35:36 -04:00
nli-shadow-sweep-benchqa-n1.json #000049 §7 #22: speedup (batch + cuda auto-detect + ONNX-int8 export) + the gate-item-4 verdict at proper n 2026-05-12 17:21:32 -04:00
nli-shadow-sweep-smoke-live.json #000049 §7 #21: candidate-clause restriction in ShadowNLI.check — helps, doesn't close it 2026-05-12 14:34:44 -04:00
nli-shadow-sweep.json #000049 §7 #21: candidate-clause restriction in ShadowNLI.check — helps, doesn't close it 2026-05-12 14:34:44 -04:00
nli-strict-fires-mining-pooled.jsonl #000049 §7 #26: mining for real recombinations + the pooled-808-STRICT check — verdict settles on margin@0.995 2026-05-12 21:11:35 -04:00
phase3-live-validation-2026-05-10.md Phase 3 live validation: 6/7 probes reach EVIDENCE-WARRANTED 2026-05-10 09:46:10 -04:00
phase3-warrant-chain-A-B-2026-05-10.md bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture 2026-05-10 09:37:27 -04:00
phase3-warrant-chain-A-B-aggressive-2026-05-10.md bench: aggressive warrant fixture confirms Phase 3 is rescue-only, not default-path 2026-05-10 09:53:21 -04:00
phase3-warrant-chain-paraphrase-investigation-2026-05-10.md bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus 2026-05-10 10:06:23 -04:00
progressive-and-2026-05-09.md bench: progressive-AND fixture + 2026-05-09 A/B baseline report 2026-05-10 06:35:18 -04:00
prometheus-sigma-sweep-dryrun-2026-05-10.md #000037 §22: dry-run swap to sweep_weights() + §15.4 added 2026-05-10 19:22:03 -04:00
prometheus-sigma-triggers-2026-05-10-npower.md three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c 2026-05-10 07:46:35 -04:00
prometheus-sigma-triggers-2026-05-10.md #000037 §12 Trigger 2 fires empirically — Phase 1 gate satisfied 2026-05-10 17:55:19 -04:00
prometheus-sigma-triggers-2026-05-11.md #000012 Phase 1c follow-through: wire #000037 §12 Trigger 1 probe to fork_score_branches 2026-05-11 06:56:09 -04:00
real-shard-baseline.json modified: .gitignore 2026-05-08 16:38:09 -04:00
real-shard-baseline.md modified: .gitignore 2026-05-08 16:38:09 -04:00
relevance-aboutness-candidatebench-round2.json #000052 §3.2.1 round-2: extend size spectrum to 7 models (30MB-560MB, ~20×) — real model science from a wider sweep 2026-05-13 10:05:56 -04:00
relevance-aboutness-candidatebench.json #000052 §3.2.1: candidate-bench landed (4 models × 26 pairs on the 4090 — proper bench, not the 6-pair smoke) 2026-05-13 09:57:26 -04:00
relevance-shadow-sweep-pooled808-cleaned.json #000052 §3.2.2 refinement: claim-lattice metadata cleaning — STRICT FP drops universally (-0.5 to -5.6 pts), L-2 down to 1.5% 2026-05-13 15:01:50 -04:00
relevance-shadow-sweep-pooled808.json #000052 §3.2.2 step 2: relevance shadow sweep on 808 pooled bench-qa STRICT — universal walk-back, runtime veto NOT VIABLE on this design 2026-05-13 14:17:38 -04:00
textbook-base-knowledge-upgrade-2026-05-09.md textbook ingest: 4 base-knowledge additions for pillars I/II/III/IX 2026-05-09 20:38:32 -04:00
three-more-textbooks-2026-05-10.md 3 more textbooks (Russell PoM + Laplace + Dedekind): 54 -> 62/92 (67%) 2026-05-10 07:31:40 -04:00
warrant-chain-render-2026-05-09.md bench/results: warrant-chain render tail measurement (2026-05-09) 2026-05-09 18:00:36 -04:00
witness-sweep.json fan-out: witness automation · #000018 analysis · #000033 pillar VII (Claude) 2026-05-09 14:36:39 -04:00