From 8b3a38bcab07702849ca3b6efbac6743bf3cb3c4 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 26 Apr 2026 07:24:28 -0400 Subject: [PATCH] scanner-enhancement intel: 10 unmoad fixes, ~51679 FPs cleared across 19 codebases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents this autonomous-loop session's pivot from breadth wave scanning to unmoad scanner enhancements. Each fix targeted a documented FP class (noted across waves 11/15/18/21/24/26 + others), shipped + validated end-to-end against re-cloned source. Architecture: per-file symbol tables on ScanState (hash_vars for M1, tl_vars for M3) + path/header/argument-shape heuristics. Three repos in sync, all unmoad commits remain local (no remote per project CLAUDE.md). Notable byproduct: with M3 noise dropped wildfly 4840 -> 37 (commit 1f48798), one real ThreadLocal leak surfaces as a MOAD-0003 candidate for fox's review — ElytronSecurityIntegration.securityContext.set(context) with zero remove() calls anywhere in the WildFly codebase. Out of CWE-407 scope; MOAD-0003 follow-up. --- .../outreach/scanner-enhancement-session.md | 198 ++++++++++++++++++ 1 file changed, 198 insertions(+) create mode 100644 whitepaper/outreach/scanner-enhancement-session.md diff --git a/whitepaper/outreach/scanner-enhancement-session.md b/whitepaper/outreach/scanner-enhancement-session.md new file mode 100644 index 000000000..b6247b5d9 --- /dev/null +++ b/whitepaper/outreach/scanner-enhancement-session.md @@ -0,0 +1,198 @@ +# unmoad scanner enhancement session — 10 fixes, ~51,679 FPs cleared + +**Session date:** 2026-04-26 +**Scope:** Single autonomous-loop session targeting the false-positive backlog +documented across waves 7-28 (scanner FPs noted 30+ times). + +--- + +## Summary + +Ten scanner enhancements landed in `unmoad.com`, all validated end-to-end +against re-cloned source. **~51,679 false positives cleared across 19 +codebases.** Three repos in sync after each commit. All commits remain +local on `unmoad.com` (no remote configured per the project's CLAUDE.md). + +The session pattern: pick the most-documented FP from triage backlog, +write a per-file symbol-table or path-shape suppression, validate against +two real codebases that exhibited the FP, commit. Rinse, repeat. + +## Commits + validated impact + +| # | Commit | Detector | Languages | FPs cleared (validated) | +|---|--------|----------|-----------|-------------------------| +| 1 | `73caddd` | M1 | Java/Kotlin/Scala/Groovy | wildfly **65** | +| 2 | `79dbac6` | M1 | Rust | meilisearch **25** | +| 3 | `1f48798` | M3 | Java/Kotlin/Groovy | wildfly **4803** | +| 4 | `9f62cdf` | M1 | C# | PowerShell **42** | +| 5 | `e94ec26` | M3 | Python | paperless-ngx **108** + patroni **49** | +| 6 | `1048451` | scanner | All | WordPress **540** + TYPO3 **427** | +| 7 | `c152ad9` | scanner | All | aws-sdk-go-v2 **18,582** + azure-sdk-for-go **25,755** | +| 8 | `26958c0` | M1 | Lisp/Scheme/Racket | sbcl **706** + racket **306** | +| 9 | `0fa41ae` | M7 | All | hydrogen **47** + Hyprland **37** + abseil **22** + spack **6** | +| 10 | `7770325` | M1+M7 | Rust + all | helix **37** + alacritty **40** + wezterm **81** | + +**Total: ~51,679 FPs cleared across 19 codebases.** + +## Architecture + +Three reusable pieces of state added to `ScanState`: + +```c +char hash_vars[64][64]; /* moad_0001: vars declared as Set/Map/HashSet */ +int hash_vars_count; +char tl_vars[64][64]; /* moad_0003: vars declared as ThreadLocal/ContextVar */ +int tl_vars_count; +``` + +Eight of ten enhancements use a per-file symbol-table tracker that +recognizes declarations on each line and consults the table at call sites: + +| Tracker | Recognizes | Suppresses | +|---------|------------|------------| +| `track_java_hash_decls` | `Set<...>`, `HashSet<...>`, `EnumSet<...>`, `Map<...>`, `HashMap<...>`, `ConcurrentHashMap<...>`, `ImmutableSet/Map<...>` | `var.contains(...)` / `var.indexOf(...)` | +| `track_rust_hash_decls` | `let var: HashSet<...>`, `let var = HashSet::new()`, plus `FxHashSet`, `BTreeSet`, `RoaringBitmap`, `SmallBitmap`, `IndexSet`, `BitSet`, `DashSet`, etc. | `var.contains(...)` / `var.position(...)` | +| `track_csharp_hash_decls` | `HashSet<...>`, `Dictionary<...>`, `ImmutableHashSet<...>`, `FrozenSet<...>` | `var.Contains(...)` / `var.IndexOf(...)` | +| `track_java_tl_decls` | `ThreadLocal<...>`, `InheritableThreadLocal<...>`, `ScopedValue<...>`, `FastThreadLocal<...>` | `var.set(...)` (no-remove leak check) | +| `track_python_tl_decls` | `var = ContextVar(...)`, `var: ContextVar[T] = ...`, `_local = threading.local()` | `var.set(...)` (no-reset leak check) | + +Two enhancements use file-level path/header heuristics: + +| Heuristic | Suppresses | +|-----------|------------| +| `should_skip_filename` | `*.min.js`, `*.min.css`, `*.bundle.js`, `*.umd.js`, `*.global.prod.js`, `*-bundle.js`, `*-min.js` | +| `is_generated` | First-16-line scan for `// Code generated by ...`, `DO NOT EDIT`, ``, `@generated`, `Auto-generated`, `automatically generated`, `This file was generated` | + +Plus expanded `SKIP_DIRS`: `3rdParty`, `external`, `Externals`, +`third_party`, `thirdparty`, `Contrib`. + +Two enhancements operate on argument-shape and definition-shape: + +| Helper | Recognizes | +|--------|-----------| +| `csharp_indexof_is_char_literal` | `IndexOf('X')` (PowerShell single-char position scan, not collection lookup) | +| `looks_like_definition` (M7) | `bool ClassName::contains(...)`, function body `) {` opener, `def`/`fn`/`function` keywords | +| `looks_like_string_contains` (M7) | `.contains('X')` / `.contains("X")` / `.contains(b'X')` substring search | +| `contains_arg_is_path_constant` (M1+M7) | `.contains(Type::SCREAMING_SNAKE)` — bitflags / enum flag check | + +One severity-only adjustment: + +| Rule | Change | +|------|--------| +| `list-member-in-loop` (Lisp/Scheme/Racket) | `SEV_HIGH` → `SEV_MEDIUM`. Lisp `(member ...)` returns the matched-tail not just bool; idiomatic for small-list membership in graph algorithms. | + +## Test surface + +Throughout the session, the test count grew from 105 → 111 integration + +29 unit + 33 functional. Eight new fixtures: + +- `moad_0001/clean_java_set_decl.java` — `Set.contains` + `EnumSet.contains` + `HashMap.containsKey` +- `moad_0001/clean_rust_hash_decl.rs` — `HashSet.contains` + `BTreeSet.contains` + `let _ = HashSet::new()` constructor pattern +- `moad_0001/clean_csharp_set_decl.cs` — `HashSet.Contains` + `Dictionary.ContainsKey` + `ImmutableHashSet.Contains` + `IndexOf(':')` char position +- `moad_0001/clean_rust_bitflags.rs` — `flags.contains(KeyModifiers::SHIFT)` + `mods.contains(Self::CTRL)` +- `moad_0003/clean_atomic_set_no_threadlocal.java` — `AtomicLong.set(0L)` + `AtomicReference.set(null)` + JavaBean setters +- `moad_0003/clean_python_django_set.py` — Django M2M `relation.set(items)` + `threading.Event.set()` + custom `cache.set(k,v)` +- Functional: vendored UI suffix exclusion (`.min.js`/`.bundle.js`/`.umd.js`) +- Functional: codegen marker suppression (`// DO NOT EDIT` + `// @generated`) + +All 173 tests pass after every commit. + +## Validated impact summary + +By codebase (sorted by FPs cleared): + +| Codebase | Before | After | Cleared | Reduction | +|----------|------:|------:|--------:|----------:| +| azure-sdk-for-go (codegen) | 25,898 | 143 | 25,755 | 99.4% | +| aws-sdk-go-v2 (codegen) | 18,582 | 0 | 18,582 | 100% | +| wildfly (Java Set + ThreadLocal) | 4,933 | 65 | 4,868 | 98.7% | +| sbcl (Lisp member) | 727 | 21 | 706 | 97.1% | +| WordPress (vendored UI) | 1,050 | 510 | 540 | 51.4% | +| TYPO3 (vendored UI) | 934 | 507 | 427 | 45.7% | +| racket (Lisp member) | 383 | 77 | 306 | 79.9% | +| paperless-ngx (Python ContextVar) | 109 | 1 | 108 | 99.1% | +| wezterm (Rust bitflags) | 135 | 54 | 81 | 60.0% | +| patroni (Python ContextVar) | 49 | 0 | 49 | 100% | +| hydrogen (Qt geometry) | 64 | 17 | 47 | 73.4% | +| PowerShell (C# IndexOf char) | 169 | 127 | 42 | 24.9% | +| alacritty (Rust bitflags) | 51 | 11 | 40 | 78.4% | +| Hyprland (geometry) | 46 | 9 | 37 | 80.4% | +| helix (Rust bitflags) | 75 | 38 | 37 | 49.3% | +| meilisearch (Rust HashSet) | 68 | 43 | 25 | 36.8% | +| abseil-cpp (M7 definitions) | 60 | 38 | 22 | 36.7% | +| spack (M7 method calls) | 72 | 66 | 6 | 8.3% | +| ripgrep (Rust HashSet) | 5 | 4 | 1 | 20.0% | +| **Total** | **53,510** | **1,831** | **51,679** | **96.6%** | + +## Findings worth investigating from the cleaner output + +After the M3 detector noise dropped from wildfly 4840 → 37 finds (commit +`1f48798`), the remaining 37 are real ThreadLocal patterns. Manual triage +identified **one clear MOAD-0003 candidate** worth follow-up: + +- `connector/src/main/java/org/jboss/as/connector/security/ElytronSecurityIntegration.java:38` + declares `private final ThreadLocal securityContext = new ThreadLocal<>();` + with `setSecurityContext(...)` at line 51 calling `.set(context)` and zero + `securityContext.remove()` / `securityContext.set(null)` anywhere in the + WildFly codebase (verified by `grep -rn`). For a JCA WorkManager that + reuses thread-pool threads across Work items from different security + principals, a leftover `SecurityContext` from a prior Work item could be + visible to a subsequent one if the new item reads via `getSecurityContext()` + before its own `setSecurityContext(...)` call. Real defense-in-depth + candidate at minimum, possibly an exploitable identity-leak. + +This is out of the CWE-407 disclosure pipeline (MOAD-0001) but goes into +the MOAD-0003 (A Leaked Context) follow-up queue for fox's review. + +## Effect on the wave-survey backlog + +The 30+ scanner-enhancement candidates documented across wave surveys +(every wave from 11 onward listed at least one) are now mostly closed: + +- Java declared-type Set/EnumSet awareness — **shipped** (commit 1) +- Rust HashSet/FxHashSet/Bitmap awareness — **shipped** (commits 2, 10) +- JUnit ExtensionContext suppression — **partially shipped** via Java + ThreadLocal-scoping in commit 3 (eliminates the `.set()` noise; the + `getStore().put()` patterns remain unflagged today) +- Java ThreadLocal-scoped `.set()` — **shipped** (commit 3) +- Python ContextVar-scoped `.set()` — **shipped** (commit 5) +- C# HashSet declared-type — **shipped** (commit 4) +- C# IndexOf(char_literal) — **shipped** (commit 4) +- Codegen artifact suppression — **shipped** (commit 7) +- Vendored UI tree exclusion — **shipped** (commit 6) +- Rust bitflags::bitflags! macro — **shipped** (commit 10) +- Lisp `(member ...)` algorithm awareness — **shipped** (commit 8) +- M7 definition / String.contains distinction — **shipped** (commit 9) +- Qt `.contains()` family awareness — **partially shipped** via M7 + definition heuristic (catches `bool QRect::contains(...)` definitions + but call-site `qrect.contains(point)` still fires; needs Qt-typed + receiver tracking) +- Smalltalk language module — **not shipped** (deferred; pharo still + unscannable) +- Spack/conan-style `intersects()` algorithm awareness — **partially + shipped** via M7 definition heuristic (catches definitions; call-site + `spec.intersects(other)` still fires) + +## What's left + +The remaining noise sources after this session are dominated by: + +- **Test files** that exercise patterns the scanner correctly flags as + defects in production code. A path-prefix suppression for `*Test.java`, + `*_test.go`, `tests/`, `test/`, `__tests__/` directories would help + but requires care not to mask real defects in test-helper code. +- **Spack/conan-style algorithm methods** at call site (intersects / + overlaps / contains / distance) where the method is the algorithm but + the receiver type isn't a recognized geometry primitive. +- **Method-chain receivers** (`a.getX().contains(b)` — extract_receiver + rejects chained calls). This is conservative-correct but misses + legitimate suppressions when the chain returns an O(1) container. + +## References + +- `unmoad` detection engine: `git.unturf.com/engineering/unmoad.com` +- ten unmoad commits: `73caddd` through `7770325`, all on `main`, + no remote configured (per CLAUDE.md) +- prior session shipped 5 flagship CWE-407 patches (UNDF-1300 through 1304): + weaviate, pyroscope, ghost, ghidra-0001, ghidra-0002 +- waves 7-28 of breadth scanning: 192 cumulative honor roll, 5 flagships