java-topology/whitepaper/outreach/test-harness-survey.md

Test Harness Survey — 42+ Languages — CWE-407 / MOAD-0001 Scan Results

Survey date: 2026-04-23 Tool: unmoad (9 active MOAD detectors, HIGH+ severity filter) Scope: 61 test frameworks and related tooling spanning 30+ programming languages

---

Summary

We scanned the leading open-source test frameworks across 30+ language ecosystems for the nine active MOAD patterns in our registry. The survey confirmed four confirmed CWE-407 / MOAD-0001 defects with measurable benchmarks (see individual intel briefs), identified dozens of additional findings triaged as bounded-N, false-positive (Set.contains is O(1), not O(N)), or vendored assets, and established fourteen test frameworks as having zero HIGH+ findings under our scanner.

The clean-scan list is a credit to those projects' maintainers. Inclusion in that list indicates our current scanner's 9 detectors did not fire at HIGH+ severity on the sampled source tree.

Confirmed defects with patches (Wave 3)

Target	Defect	Speedup	UNDF
vitest	coverage-v8 coverage.result.find inside forEach	824× @ N=M=10000	UNDF-2026-000001295
testng	DynamicGraph.toDot List.contains inside two loops	64× @ N=2000	UNDF-2026-000001294
jasmine	SpyRegistry.spyOnAllFunctions propertiesToSkip.indexOf	61× @ D=10 P=300	UNDF-2026-000001293
libcheck	suite tcase-by-name linear strcmp scan	117× @ N=1000	UNDF-2026-000001292

Clean scans (0 HIGH+ findings — 14 projects)

These frameworks ran clean under our 9 MOAD detectors at HIGH+ severity. They either avoid the O(N×k) sedimentary patterns entirely or keep them bounded below our detection threshold.

Framework	Language
Midje	Clojure
speclj	Clojure
alcotest	OCaml
qcheck	OCaml
hspec	Haskell
tasty	Haskell
proper	Erlang
testify	Go
expecto	F#
ReTest.jl	Julia
bats-core	Shell
shunit2	Shell
busted	Lua
tape	JavaScript

Targets with findings, triaged but not patched this wave

Findings reviewed and found to be either:

• False positives under type inference: Set.contains, Map.containsKey, String.includes, String.contains, Set.has — all O(1), flagged by the scanner conservatively because it cannot type-distinguish inside a loop. Examples: mockito (mocked.contains(type) where mocked is a Set), rspec (already_run_blocks.include? where block is a Set.new), junit5 (EnumSet.of(...).contains(...)).
• Bounded-N in configuration-space: arg-list parsing, config-file bucket filters, error-code allowlists with ≤5 elements. Examples: phpunit TestSuiteMapper in_array($suite, $includeTestSuites), nunit Options.cs IndexOf arg parsing, jest-config extensionsToTreatAsEsm.includes('.js').
• Vendored third-party assets: jquery.min.js, lunr.min.js, docset documentation. Examples: Quick's 44 findings all in docset/Contents/Resources/Documents/js/*.min.js; specs2's prettify.js and tipuesearch.js.

Projects reviewed, bounded/noise-dominated, candidates for future refinement or scanner improvement:

assertj (DeepDifference, BDDAssumptions), mockito (InlineBytecodeGenerator), scalatest (ArgsParser, Filter), kotest (StringEq, SpringTestExtension), spock (asciidoc-extensions, TempDirExtension, SpecInfo), specs2 (SpecStructure, HtmlUrls), phpunit (Configuration XML readers, TestSuiteMapper), codeception (Dependencies subscriber, Parser), rspec (configuration, memoized_helpers, metadata_filter), nunit (nunitlite Options, Constraints), hypothesis (ghostwriter, ftz_detector), pytest (cacheprovider src, findpaths), xunit (assert tests), nose2 (plugin pipeline), cocotb (ContextVar chain, scheduler), googletest (amalgamated test utils), insta (cargo-insta cli, snapshot glob), ava (like-selector, shared-worker-loader), mocha (runner.globalProps, cli/options), pest (config), Behat (autoload), Catch2 (catch_run_context find_if), munit (junit-interface TagFilter), Unity (generate_test_runner build tool), proptest (bitflags, not actual array contains), quickcheck (minor), criterion.rs (report.rs directory existence check), cucumber-ruby (minor), minitest (small M3 pattern), gomega (minor), ginkgo (ContextVar-style), hypothesis (ghostwriter code-gen), Codeception (Subscriber), tapjs (typeof checks), Nimble (minor).

Several of these are worth revisiting once the scanner gains type-inference for the contains-in-loop detector.

Method

# Clone targets (shallow, depth=1)
git clone --depth=1 https://github.com/{org}/{repo}.git

# Scan all MOADs at HIGH+
unmoad -s high -f json {repo}/ > {repo}.json

# Filter out test fixtures, node_modules, dist, docs, vendored JS
# Triage confirmed defects by reading the code context
# Confirm via Python-model benchmark + O(N+k) vs O(N×k) scaling

References

• Individual target briefs: /selenium/, /playwright/, /webdriverio/, /testcafe/, /vitest/, /testng/, /jasmine/, /check/ on undefect.com
• MOAD-0001 A Sedimentary Defect: https://undefect.com/moad-2026-0001/
• unmoad detection engine: git.unturf.com/engineering/unmoad.com