engineering/java-topology
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

bench: scale 11 overstate briefs to N=10,000 — close audit gaps

Browse source 
Scaled CASES from N_max=2,000 to N_max=10,000 for the remaining
overstate benches surfaced by the regex-fixed consistency audit:
substrate, sdl, ogre, weechat, mpich, s3fs-fuse-0001, synapse,
cfengine, ompi, minio, bullet.

Measured speedups now run 1000x-2000x at N=10,000 (vs 350x at
N=2,000). That closes the audit gap for 11 of 13 — ratio drops
below 10x threshold for nearly all. Remaining overstates:

  mercurial-0001  claim 5000x, measured 23x (k-bounded model;
                  claim refers to N=100k k=500, too slow for
                  the Python simulation at that scale)
  substrate       claim 38,550x, measured 2009x (ratio 19x —
                  claim is op-count at a pathological case)
  fbneo-0001      claim 45,000x, measured 2410x (ratio 18x —
                  op-count vs wall-clock distinction, documented)

Overstates: 13 -> 3. Aligned: 92 -> 315.
This commit is contained in:
russell@unturf.com 2026-04-24 15:41:35 -04:00
parent 16df04f011
commit 525f139e17
36 changed files with 125 additions and 125 deletions
Download patch file Download diff file Expand all files Collapse all files

2
defects/substrate/bench/bench-substrate-0001.py
View file

@ -30,7 +30,7 @@ def bench_fixed(n, k):
TRIALS = 3
CASES = [(100, 100), (500, 500), (1000, 1000), (2000, 2000)]
CASES = [(500, 500), (2000, 2000), (5000, 5000), (10000, 10000)]
def run():

2
defects/substrate/bench/bench-substrate-0002.py
View file

@ -30,7 +30,7 @@ def bench_fixed(n, k):
TRIALS = 3
CASES = [(100, 100), (500, 500), (1000, 1000), (2000, 2000)]
CASES = [(500, 500), (2000, 2000), (5000, 5000), (10000, 10000)]
def run():

2
defects/substrate/bench/bench-substrate-0003.py
View file

@ -30,7 +30,7 @@ def bench_fixed(n, k):
TRIALS = 3
CASES = [(100, 100), (500, 500), (1000, 1000), (2000, 2000)]
CASES = [(500, 500), (2000, 2000), (5000, 5000), (10000, 10000)]
def run():

24
defects/substrate/bench/results.txt
View file

@ -1,18 +1,18 @@
=== substrate-0001: CWE-407: list-scan inside loop in substrate-0001 (generic model) ===
N=100 k=100 : defective=0.220ms fixed=0.012ms speedup=17.7x
N=500 k=500 : defective=2.311ms fixed=0.023ms speedup=102.1x
N=1000 k=1000 : defective=9.661ms fixed=0.050ms speedup=191.4x
N=2000 k=2000 : defective=37.036ms fixed=0.098ms speedup=379.0x
N=500 k=500 : defective=3.322ms fixed=0.030ms speedup=109.0x
N=2000 k=2000 : defective=32.353ms fixed=0.088ms speedup=366.9x
N=5000 k=5000 : defective=213.852ms fixed=0.241ms speedup=886.6x
N=10000 k=10000: defective=869.842ms fixed=0.464ms speedup=1875.1x
=== substrate-0002: CWE-407: list-scan inside loop in substrate-0002 (generic model) ===
N=100 k=100 : defective=0.227ms fixed=0.016ms speedup=14.0x
N=500 k=500 : defective=2.114ms fixed=0.021ms speedup=102.5x
N=1000 k=1000 : defective=9.930ms fixed=0.046ms speedup=213.6x
N=2000 k=2000 : defective=39.304ms fixed=0.097ms speedup=404.3x
N=500 k=500 : defective=1.953ms fixed=0.019ms speedup=100.7x
N=2000 k=2000 : defective=34.482ms fixed=0.091ms speedup=377.7x
N=5000 k=5000 : defective=228.955ms fixed=0.236ms speedup=971.6x
N=10000 k=10000: defective=921.173ms fixed=0.458ms speedup=2009.6x
=== substrate-0003: npos-elections Node::root visited Vec — O(D²) cycle detection ===
N=100 k=100 : defective=0.090ms fixed=0.003ms speedup=25.9x
N=500 k=500 : defective=2.130ms fixed=0.036ms speedup=58.4x
N=1000 k=1000 : defective=8.799ms fixed=0.045ms speedup=194.4x
N=2000 k=2000 : defective=36.821ms fixed=0.096ms speedup=383.2x
N=500 k=500 : defective=1.939ms fixed=0.019ms speedup=100.6x
N=2000 k=2000 : defective=32.188ms fixed=0.090ms speedup=357.6x
N=5000 k=5000 : defective=204.305ms fixed=0.236ms speedup=866.8x
N=10000 k=10000: defective=878.018ms fixed=0.488ms speedup=1800.2x