scanner-enhancement intel: 10 unmoad fixes, ~51679 FPs cleared across 19 codebases

Documents this autonomous-loop session's pivot from breadth wave scanning to unmoad scanner enhancements. Each fix targeted a documented FP class (noted across waves 11/15/18/21/24/26 + others), shipped + validated end-to-end against re-cloned source. Architecture: per-file symbol tables on ScanState (hash_vars for M1, tl_vars for M3) + path/header/argument-shape heuristics. Three repos in sync, all unmoad commits remain local (no remote per project CLAUDE.md). Notable byproduct: with M3 noise dropped wildfly 4840 -> 37 (commit 1f48798), one real ThreadLocal leak surfaces as a MOAD-0003 candidate for fox's review — ElytronSecurityIntegration.securityContext.set(context) with zero remove() calls anywhere in the WildFly codebase. Out of CWE-407 scope; MOAD-0003 follow-up.
2026-04-26 07:24:28 -04:00 · 2026-04-26 07:24:28 -04:00 · 8b3a38bcab
commit 8b3a38bcab
parent 9f7d28ffb9
1 changed files with 198 additions and 0 deletions
--- a/whitepaper/outreach/scanner-enhancement-session.md
+++ b/whitepaper/outreach/scanner-enhancement-session.md
@ -0,0 +1,198 @@
+# unmoad scanner enhancement session — 10 fixes, ~51,679 FPs cleared
+
+**Session date:** 2026-04-26
+**Scope:** Single autonomous-loop session targeting the false-positive backlog
+documented across waves 7-28 (scanner FPs noted 30+ times).
+
+---
+
+## Summary
+
+Ten scanner enhancements landed in `unmoad.com`, all validated end-to-end
+against re-cloned source. **~51,679 false positives cleared across 19
+codebases.** Three repos in sync after each commit. All commits remain
+local on `unmoad.com` (no remote configured per the project's CLAUDE.md).
+
+The session pattern: pick the most-documented FP from triage backlog,
+write a per-file symbol-table or path-shape suppression, validate against
+two real codebases that exhibited the FP, commit. Rinse, repeat.
+
+## Commits + validated impact
+
+| # | Commit | Detector | Languages | FPs cleared (validated) |
+|---|--------|----------|-----------|-------------------------|
+| 1 | `73caddd` | M1 | Java/Kotlin/Scala/Groovy | wildfly **65** |
+| 2 | `79dbac6` | M1 | Rust | meilisearch **25** |
+| 3 | `1f48798` | M3 | Java/Kotlin/Groovy | wildfly **4803** |
+| 4 | `9f62cdf` | M1 | C# | PowerShell **42** |
+| 5 | `e94ec26` | M3 | Python | paperless-ngx **108** + patroni **49** |
+| 6 | `1048451` | scanner | All | WordPress **540** + TYPO3 **427** |
+| 7 | `c152ad9` | scanner | All | aws-sdk-go-v2 **18,582** + azure-sdk-for-go **25,755** |
+| 8 | `26958c0` | M1 | Lisp/Scheme/Racket | sbcl **706** + racket **306** |
+| 9 | `0fa41ae` | M7 | All | hydrogen **47** + Hyprland **37** + abseil **22** + spack **6** |
+| 10 | `7770325` | M1+M7 | Rust + all | helix **37** + alacritty **40** + wezterm **81** |
+
+**Total: ~51,679 FPs cleared across 19 codebases.**
+
+## Architecture
+
+Three reusable pieces of state added to `ScanState`:
+
+```c
+char hash_vars[64][64];   /* moad_0001: vars declared as Set/Map/HashSet */
+int  hash_vars_count;
+char tl_vars[64][64];     /* moad_0003: vars declared as ThreadLocal/ContextVar */
+int  tl_vars_count;
+```
+
+Eight of ten enhancements use a per-file symbol-table tracker that
+recognizes declarations on each line and consults the table at call sites:
+
+| Tracker | Recognizes | Suppresses |
+|---------|------------|------------|
+| `track_java_hash_decls` | `Set<...>`, `HashSet<...>`, `EnumSet<...>`, `Map<...>`, `HashMap<...>`, `ConcurrentHashMap<...>`, `ImmutableSet/Map<...>` | `var.contains(...)` / `var.indexOf(...)` |
+| `track_rust_hash_decls` | `let var: HashSet<...>`, `let var = HashSet::new()`, plus `FxHashSet`, `BTreeSet`, `RoaringBitmap`, `SmallBitmap`, `IndexSet`, `BitSet`, `DashSet`, etc. | `var.contains(...)` / `var.position(...)` |
+| `track_csharp_hash_decls` | `HashSet<...>`, `Dictionary<...>`, `ImmutableHashSet<...>`, `FrozenSet<...>` | `var.Contains(...)` / `var.IndexOf(...)` |
+| `track_java_tl_decls` | `ThreadLocal<...>`, `InheritableThreadLocal<...>`, `ScopedValue<...>`, `FastThreadLocal<...>` | `var.set(...)` (no-remove leak check) |
+| `track_python_tl_decls` | `var = ContextVar(...)`, `var: ContextVar[T] = ...`, `_local = threading.local()` | `var.set(...)` (no-reset leak check) |
+
+Two enhancements use file-level path/header heuristics:
+
+| Heuristic | Suppresses |
+|-----------|------------|
+| `should_skip_filename` | `*.min.js`, `*.min.css`, `*.bundle.js`, `*.umd.js`, `*.global.prod.js`, `*-bundle.js`, `*-min.js` |
+| `is_generated` | First-16-line scan for `// Code generated by ...`, `DO NOT EDIT`, `<auto-generated>`, `@generated`, `Auto-generated`, `automatically generated`, `This file was generated` |
+
+Plus expanded `SKIP_DIRS`: `3rdParty`, `external`, `Externals`,
+`third_party`, `thirdparty`, `Contrib`.
+
+Two enhancements operate on argument-shape and definition-shape:
+
+| Helper | Recognizes |
+|--------|-----------|
+| `csharp_indexof_is_char_literal` | `IndexOf('X')` (PowerShell single-char position scan, not collection lookup) |
+| `looks_like_definition` (M7) | `bool ClassName::contains(...)`, function body `) {` opener, `def`/`fn`/`function` keywords |
+| `looks_like_string_contains` (M7) | `.contains('X')` / `.contains("X")` / `.contains(b'X')` substring search |
+| `contains_arg_is_path_constant` (M1+M7) | `.contains(Type::SCREAMING_SNAKE)` — bitflags / enum flag check |
+
+One severity-only adjustment:
+
+| Rule | Change |
+|------|--------|
+| `list-member-in-loop` (Lisp/Scheme/Racket) | `SEV_HIGH` → `SEV_MEDIUM`. Lisp `(member ...)` returns the matched-tail not just bool; idiomatic for small-list membership in graph algorithms. |
+
+## Test surface
+
+Throughout the session, the test count grew from 105 → 111 integration +
+29 unit + 33 functional. Eight new fixtures:
+
+- `moad_0001/clean_java_set_decl.java` — `Set<String>.contains` + `EnumSet.contains` + `HashMap.containsKey`
+- `moad_0001/clean_rust_hash_decl.rs` — `HashSet<u32>.contains` + `BTreeSet.contains` + `let _ = HashSet::new()` constructor pattern
+- `moad_0001/clean_csharp_set_decl.cs` — `HashSet<string>.Contains` + `Dictionary.ContainsKey` + `ImmutableHashSet.Contains` + `IndexOf(':')` char position
+- `moad_0001/clean_rust_bitflags.rs` — `flags.contains(KeyModifiers::SHIFT)` + `mods.contains(Self::CTRL)`
+- `moad_0003/clean_atomic_set_no_threadlocal.java` — `AtomicLong.set(0L)` + `AtomicReference.set(null)` + JavaBean setters
+- `moad_0003/clean_python_django_set.py` — Django M2M `relation.set(items)` + `threading.Event.set()` + custom `cache.set(k,v)`
+- Functional: vendored UI suffix exclusion (`.min.js`/`.bundle.js`/`.umd.js`)
+- Functional: codegen marker suppression (`// DO NOT EDIT` + `// @generated`)
+
+All 173 tests pass after every commit.
+
+## Validated impact summary
+
+By codebase (sorted by FPs cleared):
+
+| Codebase | Before | After | Cleared | Reduction |
+|----------|------:|------:|--------:|----------:|
+| azure-sdk-for-go (codegen) | 25,898 | 143 | 25,755 | 99.4% |
+| aws-sdk-go-v2 (codegen) | 18,582 | 0 | 18,582 | 100% |
+| wildfly (Java Set + ThreadLocal) | 4,933 | 65 | 4,868 | 98.7% |
+| sbcl (Lisp member) | 727 | 21 | 706 | 97.1% |
+| WordPress (vendored UI) | 1,050 | 510 | 540 | 51.4% |
+| TYPO3 (vendored UI) | 934 | 507 | 427 | 45.7% |
+| racket (Lisp member) | 383 | 77 | 306 | 79.9% |
+| paperless-ngx (Python ContextVar) | 109 | 1 | 108 | 99.1% |
+| wezterm (Rust bitflags) | 135 | 54 | 81 | 60.0% |
+| patroni (Python ContextVar) | 49 | 0 | 49 | 100% |
+| hydrogen (Qt geometry) | 64 | 17 | 47 | 73.4% |
+| PowerShell (C# IndexOf char) | 169 | 127 | 42 | 24.9% |
+| alacritty (Rust bitflags) | 51 | 11 | 40 | 78.4% |
+| Hyprland (geometry) | 46 | 9 | 37 | 80.4% |
+| helix (Rust bitflags) | 75 | 38 | 37 | 49.3% |
+| meilisearch (Rust HashSet) | 68 | 43 | 25 | 36.8% |
+| abseil-cpp (M7 definitions) | 60 | 38 | 22 | 36.7% |
+| spack (M7 method calls) | 72 | 66 | 6 | 8.3% |
+| ripgrep (Rust HashSet) | 5 | 4 | 1 | 20.0% |
+| **Total** | **53,510** | **1,831** | **51,679** | **96.6%** |
+
+## Findings worth investigating from the cleaner output
+
+After the M3 detector noise dropped from wildfly 4840 → 37 finds (commit
+`1f48798`), the remaining 37 are real ThreadLocal patterns. Manual triage
+identified **one clear MOAD-0003 candidate** worth follow-up:
+
+- `connector/src/main/java/org/jboss/as/connector/security/ElytronSecurityIntegration.java:38`
+  declares `private final ThreadLocal<SecurityContext> securityContext = new ThreadLocal<>();`
+  with `setSecurityContext(...)` at line 51 calling `.set(context)` and zero
+  `securityContext.remove()` / `securityContext.set(null)` anywhere in the
+  WildFly codebase (verified by `grep -rn`). For a JCA WorkManager that
+  reuses thread-pool threads across Work items from different security
+  principals, a leftover `SecurityContext` from a prior Work item could be
+  visible to a subsequent one if the new item reads via `getSecurityContext()`
+  before its own `setSecurityContext(...)` call. Real defense-in-depth
+  candidate at minimum, possibly an exploitable identity-leak.
+
+This is out of the CWE-407 disclosure pipeline (MOAD-0001) but goes into
+the MOAD-0003 (A Leaked Context) follow-up queue for fox's review.
+
+## Effect on the wave-survey backlog
+
+The 30+ scanner-enhancement candidates documented across wave surveys
+(every wave from 11 onward listed at least one) are now mostly closed:
+
+- Java declared-type Set/EnumSet awareness — **shipped** (commit 1)
+- Rust HashSet/FxHashSet/Bitmap awareness — **shipped** (commits 2, 10)
+- JUnit ExtensionContext suppression — **partially shipped** via Java
+  ThreadLocal-scoping in commit 3 (eliminates the `.set()` noise; the
+  `getStore().put()` patterns remain unflagged today)
+- Java ThreadLocal-scoped `.set()` — **shipped** (commit 3)
+- Python ContextVar-scoped `.set()` — **shipped** (commit 5)
+- C# HashSet declared-type — **shipped** (commit 4)
+- C# IndexOf(char_literal) — **shipped** (commit 4)
+- Codegen artifact suppression — **shipped** (commit 7)
+- Vendored UI tree exclusion — **shipped** (commit 6)
+- Rust bitflags::bitflags! macro — **shipped** (commit 10)
+- Lisp `(member ...)` algorithm awareness — **shipped** (commit 8)
+- M7 definition / String.contains distinction — **shipped** (commit 9)
+- Qt `.contains()` family awareness — **partially shipped** via M7
+  definition heuristic (catches `bool QRect::contains(...)` definitions
+  but call-site `qrect.contains(point)` still fires; needs Qt-typed
+  receiver tracking)
+- Smalltalk language module — **not shipped** (deferred; pharo still
+  unscannable)
+- Spack/conan-style `intersects()` algorithm awareness — **partially
+  shipped** via M7 definition heuristic (catches definitions; call-site
+  `spec.intersects(other)` still fires)
+
+## What's left
+
+The remaining noise sources after this session are dominated by:
+
+- **Test files** that exercise patterns the scanner correctly flags as
+  defects in production code. A path-prefix suppression for `*Test.java`,
+  `*_test.go`, `tests/`, `test/`, `__tests__/` directories would help
+  but requires care not to mask real defects in test-helper code.
+- **Spack/conan-style algorithm methods** at call site (intersects /
+  overlaps / contains / distance) where the method is the algorithm but
+  the receiver type isn't a recognized geometry primitive.
+- **Method-chain receivers** (`a.getX().contains(b)` — extract_receiver
+  rejects chained calls). This is conservative-correct but misses
+  legitimate suppressions when the chain returns an O(1) container.
+
+## References
+
+- `unmoad` detection engine: `git.unturf.com/engineering/unmoad.com`
+- ten unmoad commits: `73caddd` through `7770325`, all on `main`,
+  no remote configured (per CLAUDE.md)
+- prior session shipped 5 flagship CWE-407 patches (UNDF-1300 through 1304):
+  weaviate, pyroscope, ghost, ghidra-0001, ghidra-0002
+- waves 7-28 of breadth scanning: 192 cumulative honor roll, 5 flagships