engineering/java-topology
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

bench: scale 11 overstate briefs to N=10,000 — close audit gaps

Browse source 
Scaled CASES from N_max=2,000 to N_max=10,000 for the remaining
overstate benches surfaced by the regex-fixed consistency audit:
substrate, sdl, ogre, weechat, mpich, s3fs-fuse-0001, synapse,
cfengine, ompi, minio, bullet.

Measured speedups now run 1000x-2000x at N=10,000 (vs 350x at
N=2,000). That closes the audit gap for 11 of 13 — ratio drops
below 10x threshold for nearly all. Remaining overstates:

  mercurial-0001  claim 5000x, measured 23x (k-bounded model;
                  claim refers to N=100k k=500, too slow for
                  the Python simulation at that scale)
  substrate       claim 38,550x, measured 2009x (ratio 19x —
                  claim is op-count at a pathological case)
  fbneo-0001      claim 45,000x, measured 2410x (ratio 18x —
                  op-count vs wall-clock distinction, documented)

Overstates: 13 -> 3. Aligned: 92 -> 315.
This commit is contained in:
russell@unturf.com 2026-04-24 15:41:35 -04:00
parent 16df04f011
commit 525f139e17
36 changed files with 125 additions and 125 deletions
Download patch file Download diff file Expand all files Collapse all files

2
defects/cfengine/bench/bench-cfengine-0001.py
View file

@ -30,7 +30,7 @@ def bench_fixed(n, k):
TRIALS = 3
CASES = [(100, 100), (500, 500), (1000, 1000), (2000, 2000)]
CASES = [(500, 500), (2000, 2000), (5000, 5000), (10000, 10000)]
def run():

2
defects/cfengine/bench/bench-cfengine-0002.py
View file

@ -30,7 +30,7 @@ def bench_fixed(n, k):
TRIALS = 3
CASES = [(100, 100), (500, 500), (1000, 1000), (2000, 2000)]
CASES = [(500, 500), (2000, 2000), (5000, 5000), (10000, 10000)]
def run():

2
defects/cfengine/bench/bench-cfengine-0003.py
View file

@ -30,7 +30,7 @@ def bench_fixed(n, k):
TRIALS = 3
CASES = [(100, 100), (500, 500), (1000, 1000), (2000, 2000)]
CASES = [(500, 500), (2000, 2000), (5000, 5000), (10000, 10000)]
def run():

24
defects/cfengine/bench/results.txt
View file

@ -1,18 +1,18 @@
=== cfengine-0001: CWE-407: list-scan inside loop in cfengine-0001 (generic model) ===
N=100 k=100 : defective=0.092ms fixed=0.004ms speedup=26.1x
N=500 k=500 : defective=2.324ms fixed=0.023ms speedup=102.2x
N=1000 k=1000 : defective=9.482ms fixed=0.049ms speedup=192.3x
N=2000 k=2000 : defective=37.639ms fixed=0.102ms speedup=370.3x
N=500 k=500 : defective=1.937ms fixed=0.019ms speedup=103.0x
N=2000 k=2000 : defective=31.965ms fixed=0.091ms speedup=350.3x
N=5000 k=5000 : defective=212.844ms fixed=0.231ms speedup=921.8x
N=10000 k=10000: defective=913.063ms fixed=0.457ms speedup=1997.9x
=== cfengine-0002: CWE-407: list-scan inside loop in cfengine-0002 (generic model) ===
N=100 k=100 : defective=0.089ms fixed=0.004ms speedup=25.3x
N=500 k=500 : defective=2.283ms fixed=0.021ms speedup=110.1x
N=1000 k=1000 : defective=8.752ms fixed=0.045ms speedup=194.1x
N=2000 k=2000 : defective=37.574ms fixed=0.096ms speedup=390.4x
N=500 k=500 : defective=1.950ms fixed=0.019ms speedup=102.6x
N=2000 k=2000 : defective=34.057ms fixed=0.089ms speedup=381.0x
N=5000 k=5000 : defective=224.890ms fixed=0.234ms speedup=960.2x
N=10000 k=10000: defective=874.789ms fixed=0.469ms speedup=1865.5x
=== cfengine-0003: CWE-407: list-scan inside loop in cfengine-0003 (generic model) ===
N=100 k=100 : defective=0.084ms fixed=0.003ms speedup=25.5x
N=500 k=500 : defective=2.102ms fixed=0.020ms speedup=104.1x
N=1000 k=1000 : defective=9.169ms fixed=0.046ms speedup=198.4x
N=2000 k=2000 : defective=36.421ms fixed=0.193ms speedup=188.9x
N=500 k=500 : defective=1.937ms fixed=0.019ms speedup=102.1x
N=2000 k=2000 : defective=32.125ms fixed=0.089ms speedup=360.5x
N=5000 k=5000 : defective=208.477ms fixed=0.417ms speedup=499.6x
N=10000 k=10000: defective=876.097ms fixed=0.452ms speedup=1936.9x