russell@unturf.com d67ec93a5d test-frameworks wave 3: vitest + testng + jasmine + libcheck (4 patches)

vitest-0001: coverage-v8 coverage.result.find inside merged.result.forEach
  -> Map<url, result> lookup. Bench: 824x at N=M=10000 coverage entries.

testng-0001: DynamicGraph.toDot freeNodes.contains inside two for-each
  loops -> Map<T, String> color lookup via getOrDefault. Bench: 64x at N=2000.

jasmine-0001: SpyRegistry.spyOnAllFunctions propertiesToSkip.indexOf inside
  Array.filter + .concat growth across D prototype levels -> Set.has + O(1)
  growth. Bench: 61x at D=10, P=300.

check-0001: libcheck suite_tcase linear strcmp scan over tclst List
  -> parallel hashtable for O(1) lookup amortized. Bench: 117x at N=1000.
  Shipped as design sketch; full integration requires companion hashtable.

Also ships whitepaper/outreach/test-harness-survey.md documenting 14
clean-scan frameworks across Clojure, OCaml, Haskell, Erlang, Go, F#,
Julia, Shell, Lua, JS. Scope covered 61 targets across 30+ languages.

UNDF IDs: 1292 (check), 1293 (jasmine), 1294 (testng), 1295 (vitest).
All 12 tests pass.

2026-04-23 08:54:44 -04:00

2.5 KiB

Raw Blame History

check-0001: Suite test-case lookup — O(N) strcmp linear scan per call

Target: libcheck/check Severity: LOW-MEDIUM CWE: CWE-407 (Inefficient Algorithmic Complexity) MOAD: MOAD-0001 (A Sedimentary Defect) File: src/check.c:76-94, 186-229 Language: C Status: open

Description

libcheck looks up test cases by name via a linear scan of a List, calling strcmp per entry. The same pattern lives in two places: suite_tcase (helper for tcase-by-name lookup) and the suite-runner filter (lines 186-229) that applies sname/tcname filters to decide whether to execute a given suite/tcase. On large test suites (hundreds of suites × hundreds of tcases each), per-call cost is O(N) per lookup, O(N²) if the runner iterates all tcases checking name membership.

Root Cause

// src/check.c:76-94
int suite_tcase(Suite *s, const char *tcname)
{
    List *l;

    if(s == NULL) return 0;

    l = s->tclst;
    for(check_list_front(l); !check_list_at_end(l); check_list_advance(l))
    {
        TCase *tc = (TCase *)check_list_val(l);
        if(strcmp(tcname, tc->name) == 0)
            return 1;
    }
    return 0;
}

// src/check_run.c:186-229 — runner filter
// For each suite (sname filter) and each tcase (tcname filter), strcmp is
// invoked per list entry per call.

Fix

Maintain a parallel hashtable (or sorted array with binary search) keyed by name alongside the List, updated whenever tcase_add/suite_add_tcase is called. Lookup drops to O(1) amortized. The List is preserved for ordered iteration (test-run order matters).

// Add to Suite:
struct hashtable *tcname_index;  // maps char* name -> TCase*

// In tcase_add, on insert:
hashtable_insert(s->tcname_index, tc->name, tc);

// Rewrite suite_tcase:
int suite_tcase(Suite *s, const char *tcname) {
    if (s == NULL || s->tcname_index == NULL) return 0;
    return hashtable_search(s->tcname_index, tcname) != NULL;
}

Total cost drops from O(N) per lookup to O(1) amortized. For the runner filter, the net speedup across a test-run is O(N²) → O(N).

Severity Note

Impact scales with test suite size. Typical unit-test codebases have <50 tcases per suite, where the effect is milliseconds at most. Larger suites (integration/e2e harnesses with hundreds of tcases and filter patterns) see measurable slowdown. LOW-MEDIUM priority; cleanup rather than hotspot.

Complexity Gate

N=500 tcases per suite, 500 lookups: fixed must complete in <5ms
k-scaling 5×: time ratio must be <17.5×

2.5 KiB Raw Blame History Unescape Escape