wave10a: 462/209 hudi/iceberg/beam-clean/samza-clean

defects/beam/patch/beam-CLEAN.md

@@ -0,0 +1,13 @@
+# beam — CLEAN
+
+Scanned:
+- `sdks/java/core` — transforms, values, construction graph, PipelineValidator
+- `runners/core-java` — SimplePushbackSideInputDoFnRunner, SideInputHandler, WatermarkManager, TimerUpdate
+- `runners/direct-java` — KeyedPValueTrackingVisitor, EvaluationContext, SideInputContainer, DirectRunner
+- `runners/flink` — FlinkRunner, SideInputInitializer, FlinkStateInternals
+- `runners/spark` — SparkSideInputReader, CachedSideInputReader
+- `runners/google-cloud-dataflow-java` — DataflowRunner (experiments List.contains — but O(4) fixed-length, not a loop)
+- `sdks/python/apache_beam/transforms/` — Python transforms
+
+All `contains()` calls in hot paths use HashSet, ImmutableSet, or operate on trivially small
+fixed-length lists (experiment flags, ~4 items). No CWE-407 defects found.

defects/hudi/patch/hudi-0001-appendloadedinstants-list-contains.md

@@ -0,0 +1,67 @@
+# hudi-0001 — BaseHoodieTimeline.appendLoadedInstants List.contains O(N×M)
+
+## File
+`hudi-common/src/main/java/org/apache/hudi/common/table/timeline/BaseHoodieTimeline.java`
+Line 123–131
+
+## Defect
+
+```java
+protected void appendLoadedInstants(List<HoodieInstant> loadedInstants) {
+    List<HoodieInstant> existingInstants = getInstants();          // List<HoodieInstant>
+    List<HoodieInstant> newInstants = loadedInstants.stream()
+        .filter(instant -> !existingInstants.contains(instant))   // O(M) per call
+        .collect(Collectors.toList());
+    if (!newInstants.isEmpty()) {
+      appendInstants(newInstants);
+    }
+}
+```
+
+`existingInstants` is a `List<HoodieInstant>` (the backing field `private List<HoodieInstant> instants`).
+`List.contains()` performs a linear scan — O(M) per invocation.
+The stream filter calls it once per element of `loadedInstants` — N calls total.
+
+Total complexity: **O(N × M)** where N = loadedInstants.size(), M = existing timeline size.
+
+In a Hudi table with a long history (thousands of commits), M can be large. This method is called
+during incremental timeline loading (time-range and limit-based), so both N and M can be in the
+thousands during compaction or restore operations.
+
+## Fix
+
+Convert `existingInstants` to a `HashSet` before the filter. `HoodieInstant` implements `equals()`
+and `hashCode()` based on timestamp+action+state — so a HashSet works correctly.
+
+```java
+protected void appendLoadedInstants(List<HoodieInstant> loadedInstants) {
+    Set<HoodieInstant> existingSet = new HashSet<>(getInstants());  // O(M) one-time
+    List<HoodieInstant> newInstants = loadedInstants.stream()
+        .filter(instant -> !existingSet.contains(instant))          // O(1) per call
+        .collect(Collectors.toList());
+    if (!newInstants.isEmpty()) {
+      appendInstants(newInstants);
+    }
+}
+```
+
+## Complexity
+
+| | Before | After |
+|-|--------|-------|
+| per-element membership test | O(M) | O(1) |
+| full filter pass | O(N × M) | O(N + M) |
+
+At N=M=10 000: 100 000 000 comparisons → 20 000 comparisons. **5000x fewer operations.**
+
+## Severity
+
+MEDIUM — triggered during incremental timeline load on large tables (restore, compaction, archival).
+Degrades linearly with table history depth.
+
+## Import required
+
+```java
+import java.util.HashSet;
+import java.util.Set;
+```

defects/hudi/patch/hudi-0002-pruneinternalschema-list-contains.md

@@ -0,0 +1,76 @@
+# hudi-0002 — InternalSchemaUtils.pruneInternalSchema ArrayList.contains O(N²) + pruneType O(F×D)
+
+## File
+`hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java`
+Lines 66–72 (topParentFieldIds dedup) and 105–160 (pruneType)
+
+## Defect — Part A: topParentFieldIds dedup (O(N²))
+
+```java
+List<Integer> topParentFieldIds = new ArrayList<>();
+names.stream().forEach(f -> {
+    int id = schema.findIdByName(f.split("\\.")[0]);
+    if (!topParentFieldIds.contains(id)) {      // O(N) scan of ArrayList per call
+        topParentFieldIds.add(id);
+    }
+});
+```
+
+`topParentFieldIds` is `ArrayList<Integer>`. `.contains(id)` is O(N) — called once per name.
+Total: O(N²) where N = names.size(). For a schema with many projected columns this becomes quadratic.
+
+## Defect — Part B: pruneType field membership (O(F×D))
+
+```java
+private static Type pruneType(Type type, List<Integer> fieldIds) {
+    // ...RECORD case:
+    for (Types.Field f : fields) {
+        Type newType = pruneType(f.type(), fieldIds);
+        if (fieldIds.contains(f.fieldId())) {    // O(D) scan per field
+            newTypes.add(f.type());
+        }
+    }
+    // ...ARRAY case:
+    if (fieldIds.contains(array.elementId())) {  // O(D) per array
+    // ...MAP case:
+    if (fieldIds.contains(map.valueId())) {      // O(D) per map
+```
+
+`fieldIds` is `List<Integer>`. Called recursively over the full schema tree (F nodes).
+Total: O(F × D) where F = total schema fields, D = projected field count.
+
+## Fix
+
+```java
+// Part A: use LinkedHashSet to preserve insertion order and deduplicate in O(1)
+Set<Integer> topParentFieldIdSet = new LinkedHashSet<>();
+names.stream().forEach(f -> {
+    int id = schema.findIdByName(f.split("\\.")[0]);
+    topParentFieldIdSet.add(id);   // HashSet.add deduplicates — O(1) amortized
+});
+List<Integer> topParentFieldIds = new ArrayList<>(topParentFieldIdSet);
+```
+
+```java
+// Part B: convert fieldIds to HashSet before entering pruneType
+private static Type pruneType(Type type, Set<Integer> fieldIds) {
+    // ...same logic, but fieldIds.contains() is O(1)
+}
+// Call site: pruneType(schema.getRecord(), new HashSet<>(fieldIds))
+```
+
+## Complexity
+
+| | Before | After |
+|-|--------|-------|
+| topParentFieldIds dedup | O(N²) | O(N) |
+| pruneType per-field check | O(D) | O(1) |
+| full pruneType traversal | O(F × D) | O(F) |
+
+At F=D=500 fields: 250 000 comparisons → 500. **500x fewer operations.**
+
+## Severity
+
+MEDIUM — triggered on every call to `pruneInternalSchema()`, which is called during query
+projection pushdown, Spark read, and schema evolution. Schemas with many nested columns amplify
+both defects simultaneously.

defects/hudi/patch/hudi-0003-metadatautil-logfilepaths-list-contains.md

@@ -0,0 +1,62 @@
+# hudi-0003 — HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs List.contains O(N×M)
+
+## File
+`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java`
+Line 1005–1007
+
+## Defect
+
+```java
+public static <T> Pair<Set<String>, Set<String>> getRevivedAndDeletedKeysFromMergedLogs(
+    ..., List<String> logFilePaths, ..., List<String> currentLogFilePaths, ...) {
+
+    List<String> logFilePathsWithoutCurrentLogFiles = logFilePaths.stream()
+        .filter(logFilePath -> !currentLogFilePaths.contains(logFilePath))  // O(M) per element
+        .collect(toList());
+```
+
+`currentLogFilePaths` is a `List<String>` (constructed at line 925 via `Collectors.toList()`).
+`List.contains()` does a linear scan — O(M) per call where M = currentLogFilePaths.size().
+The stream filter applies this once per element of `logFilePaths` — N calls total.
+
+Total complexity: **O(N × M)** where N = total log file paths, M = current log file paths count.
+
+This method is called during every Record-Level Index (RLI) update, which happens on each delta
+commit. In a table with many log files per partition (large MOR tables, frequent compaction), both
+N and M can be in the hundreds.
+
+## Fix
+
+Convert `currentLogFilePaths` to a `HashSet<String>` before the filter. String equality is
+well-defined and path strings are unique identifiers.
+
+```java
+Set<String> currentLogFilePathSet = new HashSet<>(currentLogFilePaths);  // O(M) once
+List<String> logFilePathsWithoutCurrentLogFiles = logFilePaths.stream()
+    .filter(logFilePath -> !currentLogFilePathSet.contains(logFilePath))   // O(1) per element
+    .collect(toList());
+```
+
+The call site (line 925–927) creates `currentLogFilePaths` as a List and passes it to this method.
+Alternatively, build it as a `HashSet` at the call site.
+
+## Complexity
+
+| | Before | After |
+|-|--------|-------|
+| per-element membership test | O(M) | O(1) |
+| full filter pass | O(N × M) | O(N + M) |
+
+At N=M=500 log files: 250 000 comparisons → 1000. **250x fewer operations.**
+
+## Severity
+
+MEDIUM — triggered on every delta commit when RLI is enabled. MOR (Merge-on-Read) tables with
+high write frequency and many log files per filegroup amplify this significantly.
+
+## Import required
+
+```java
+import java.util.HashSet;
+import java.util.Set;
+```

defects/hudi/unit/HudiLogFilePathsAlgorithm.java

@@ -0,0 +1,135 @@
+package unit;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * hudi-0003: HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs
+ *            List<String>.contains O(N×M) → HashSet O(N+M)
+ *
+ * Simulates the log file path deduplication filter in the RLI update path.
+ *
+ * Compile: javac -d . HudiLogFilePathsAlgorithm.java
+ * Run:     java -ea unit.HudiLogFilePathsAlgorithm
+ */
+public class HudiLogFilePathsAlgorithm {
+
+    static long compareCount = 0;
+
+    // Defective: List<String>.contains — O(N×M)
+    static List<String> filterLogPaths_slow(List<String> allPaths, List<String> currentPaths) {
+        return allPaths.stream()
+            .filter(path -> {
+                for (String c : currentPaths) {
+                    compareCount++;
+                    if (c.equals(path)) return false;
+                }
+                return true;
+            })
+            .collect(Collectors.toList());
+    }
+
+    // Fixed: HashSet<String>.contains — O(N+M)
+    static List<String> filterLogPaths_fast(List<String> allPaths, List<String> currentPaths) {
+        Set<String> currentSet = new HashSet<>(currentPaths);
+        return allPaths.stream()
+            .filter(path -> {
+                compareCount++;
+                return !currentSet.contains(path);
+            })
+            .collect(Collectors.toList());
+    }
+
+    static void assertEq(String label, Object expected, Object actual) {
+        if (!expected.equals(actual)) {
+            throw new AssertionError(label + ": expected " + expected + " but got " + actual);
+        }
+    }
+
+    public static void main(String[] args) {
+        int pass = 0;
+        int fail = 0;
+
+        System.out.println("hudi-0003: HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs");
+
+        // Test 1: correctness — small overlap
+        try {
+            List<String> all = new ArrayList<>();
+            List<String> current = new ArrayList<>();
+            for (int i = 0; i < 20; i++) all.add("/table/part/.hoodie_meta/file_" + i + ".log");
+            for (int i = 0; i < 5; i++) current.add(all.get(i));
+
+            compareCount = 0;
+            List<String> slow = filterLogPaths_slow(all, current);
+            compareCount = 0;
+            List<String> fast = filterLogPaths_fast(all, current);
+
+            assertEq("size", slow.size(), fast.size());
+            assertEq("content", new HashSet<>(slow), new HashSet<>(fast));
+            System.out.println("  PASS correctness-small overlap=" + current.size());
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL correctness: " + e.getMessage()); fail++; }
+
+        // Test 2: correctness — no overlap
+        try {
+            List<String> all = new ArrayList<>();
+            List<String> current = new ArrayList<>();
+            for (int i = 0; i < 10; i++) all.add("/part/file_a_" + i + ".log");
+            for (int i = 0; i < 5; i++) current.add("/part/file_b_" + i + ".log");
+
+            List<String> slow = filterLogPaths_slow(all, current);
+            List<String> fast = filterLogPaths_fast(all, current);
+            assertEq("no-overlap-size", slow.size(), fast.size());
+            assertEq("no-overlap-all", 10, slow.size());
+            System.out.println("  PASS no-overlap");
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL no-overlap: " + e.getMessage()); fail++; }
+
+        // Test 3: correctness — full overlap
+        try {
+            List<String> all = new ArrayList<>();
+            for (int i = 0; i < 10; i++) all.add("/part/file_" + i + ".log");
+            List<String> current = new ArrayList<>(all);
+
+            List<String> slow = filterLogPaths_slow(all, current);
+            List<String> fast = filterLogPaths_fast(all, current);
+            assertEq("full-overlap-size", slow.size(), fast.size());
+            assertEq("full-overlap-empty", 0, slow.size());
+            System.out.println("  PASS full-overlap");
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL full-overlap: " + e.getMessage()); fail++; }
+
+        // Test 4: N=500, M=500 — operation count
+        {
+            List<String> all = new ArrayList<>();
+            List<String> current = new ArrayList<>();
+            for (int i = 0; i < 500; i++) all.add("/table/p0/.hoodie_meta/file_" + i + ".log");
+            for (int i = 0; i < 250; i++) current.add(all.get(i));        // half overlap
+            for (int i = 500; i < 750; i++) current.add("/table/p0/.hoodie_meta/file_" + i + ".log");  // non-overlap current
+
+            compareCount = 0;
+            List<String> slowResult = filterLogPaths_slow(all, current);
+            long slowOps = compareCount;
+
+            compareCount = 0;
+            List<String> fastResult = filterLogPaths_fast(all, current);
+            long fastOps = compareCount;
+
+            assertEq("N500-result", new HashSet<>(slowResult), new HashSet<>(fastResult));
+            System.out.printf("  N=500 M=500: slow=%8d fast=%6d ratio=%4dx%n",
+                slowOps, fastOps, fastOps > 0 ? slowOps / fastOps : 0);
+
+            if (slowOps > fastOps * 10) {
+                System.out.println("  PASS slow > 10x fast at N=500"); pass++;
+            } else {
+                System.out.println("  FAIL expected slow > 10x fast"); fail++;
+            }
+        }
+
+        System.out.println(pass + "/" + (pass + fail) + " PASS");
+        if (fail > 0) throw new RuntimeException(fail + " tests failed");
+    }
+}

defects/hudi/unit/HudiPruneSchemaAlgorithm.java

@@ -0,0 +1,171 @@
+package unit;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * hudi-0002: InternalSchemaUtils.pruneInternalSchema ArrayList.contains O(N²) + pruneType O(F×D)
+ *
+ * Part A: topParentFieldIds dedup — ArrayList.contains in forEach = O(N²)
+ * Part B: pruneType field membership — List.contains per field visit = O(F×D)
+ *
+ * Compile: javac -d . HudiPruneSchemaAlgorithm.java
+ * Run:     java -ea unit.HudiPruneSchemaAlgorithm
+ */
+public class HudiPruneSchemaAlgorithm {
+
+    static long compareCount = 0;
+
+    // --- Part A: topParentFieldIds dedup ---
+
+    // Defective: ArrayList.contains dedup — O(N²)
+    static List<Integer> deduplicateParentIds_slow(List<Integer> ids) {
+        List<Integer> result = new ArrayList<>();
+        for (int id : ids) {
+            // ArrayList.contains = linear scan
+            boolean found = false;
+            for (int existing : result) {
+                compareCount++;
+                if (existing == id) { found = true; break; }
+            }
+            if (!found) result.add(id);
+        }
+        return result;
+    }
+
+    // Fixed: LinkedHashSet — O(N), preserves insertion order
+    static List<Integer> deduplicateParentIds_fast(List<Integer> ids) {
+        Set<Integer> seen = new LinkedHashSet<>();
+        for (int id : ids) {
+            compareCount++;  // one hash op per entry
+            seen.add(id);
+        }
+        return new ArrayList<>(seen);
+    }
+
+    // --- Part B: pruneType field membership ---
+
+    // Defective: List<Integer> fieldIds with contains() — O(F×D)
+    static int pruneType_slow(int[] schema, List<Integer> fieldIds) {
+        int visited = 0;
+        for (int fieldId : schema) {
+            visited++;
+            for (int fid : fieldIds) {
+                compareCount++;
+                if (fid == fieldId) break;
+            }
+        }
+        return visited;
+    }
+
+    // Fixed: Set<Integer> fieldIds with contains() — O(F)
+    static int pruneType_fast(int[] schema, Set<Integer> fieldIds) {
+        int visited = 0;
+        for (int fieldId : schema) {
+            visited++;
+            compareCount++;  // O(1) hash lookup
+            fieldIds.contains(fieldId);
+        }
+        return visited;
+    }
+
+    static void assertEq(String label, Object expected, Object actual) {
+        if (!expected.equals(actual)) {
+            throw new AssertionError(label + ": expected " + expected + " but got " + actual);
+        }
+    }
+
+    public static void main(String[] args) {
+        int pass = 0;
+        int fail = 0;
+
+        System.out.println("hudi-0002: InternalSchemaUtils.pruneInternalSchema");
+
+        // --- Part A tests ---
+
+        // Test 1: correctness of dedup
+        try {
+            List<Integer> ids = new ArrayList<>();
+            for (int i = 0; i < 5; i++) { ids.add(i % 3); }  // [0,1,2,0,1]
+            compareCount = 0;
+            List<Integer> slow = deduplicateParentIds_slow(ids);
+            compareCount = 0;
+            List<Integer> fast = deduplicateParentIds_fast(ids);
+            assertEq("dedup-size", slow.size(), fast.size());
+            assertEq("dedup-content", new HashSet<>(slow), new HashSet<>(fast));
+            System.out.println("  PASS dedup-correctness");
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL dedup-correctness: " + e.getMessage()); fail++; }
+
+        // Test 2: slow > fast at N=300
+        {
+            List<Integer> ids300 = new ArrayList<>();
+            for (int i = 0; i < 300; i++) ids300.add(i % 50);  // 50 unique, many repeats
+
+            compareCount = 0;
+            deduplicateParentIds_slow(ids300);
+            long slowOps = compareCount;
+
+            compareCount = 0;
+            deduplicateParentIds_fast(ids300);
+            long fastOps = compareCount;
+
+            System.out.printf("  dedup N=300(50unique): slow=%6d fast=%6d%n", slowOps, fastOps);
+            if (slowOps > fastOps * 5) {
+                System.out.println("  PASS slow > 5x fast for dedup"); pass++;
+            } else {
+                System.out.println("  FAIL expected slow > 5x fast for dedup"); fail++;
+            }
+        }
+
+        // --- Part B tests ---
+
+        // Test 3: correctness — both visit same fields
+        try {
+            int[] schema = new int[100];
+            for (int i = 0; i < 100; i++) schema[i] = i;
+            List<Integer> fieldIdList = new ArrayList<>();
+            Set<Integer> fieldIdSet = new HashSet<>();
+            for (int i = 0; i < 20; i++) { fieldIdList.add(i * 5); fieldIdSet.add(i * 5); }
+
+            compareCount = 0;
+            int slowVisited = pruneType_slow(schema, fieldIdList);
+            compareCount = 0;
+            int fastVisited = pruneType_fast(schema, fieldIdSet);
+            assertEq("pruneType-visited", slowVisited, fastVisited);
+            System.out.println("  PASS pruneType-correctness");
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL pruneType-correctness: " + e.getMessage()); fail++; }
+
+        // Test 4: slow > fast at F=500, D=100
+        {
+            int[] schema500 = new int[500];
+            for (int i = 0; i < 500; i++) schema500[i] = i;
+            List<Integer> fieldIdList = new ArrayList<>();
+            Set<Integer> fieldIdSet = new HashSet<>();
+            for (int i = 0; i < 100; i++) { fieldIdList.add(i); fieldIdSet.add(i); }
+
+            compareCount = 0;
+            pruneType_slow(schema500, fieldIdList);
+            long slowOps = compareCount;
+
+            compareCount = 0;
+            pruneType_fast(schema500, fieldIdSet);
+            long fastOps = compareCount;
+
+            System.out.printf("  pruneType F=500 D=100: slow=%6d fast=%6d%n", slowOps, fastOps);
+            if (slowOps > fastOps * 10) {
+                System.out.println("  PASS slow > 10x fast for pruneType"); pass++;
+            } else {
+                System.out.println("  FAIL expected slow > 10x fast for pruneType"); fail++;
+            }
+        }
+
+        System.out.println(pass + "/" + (pass + fail) + " PASS");
+        if (fail > 0) throw new RuntimeException(fail + " tests failed");
+    }
+}

defects/hudi/unit/HudiTimelineListContainsAlgorithm.java

@@ -0,0 +1,148 @@
+package unit;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * hudi-0001: BaseHoodieTimeline.appendLoadedInstants List.contains O(N×M) → HashSet O(N+M)
+ *
+ * Simulates the deduplication filter in appendLoadedInstants using a lightweight
+ * Instant stand-in (just an integer ID) to count comparison operations.
+ *
+ * Compile: javac -d . HudiTimelineListContainsAlgorithm.java
+ * Run:     java -ea unit.HudiTimelineListContainsAlgorithm
+ */
+public class HudiTimelineListContainsAlgorithm {
+
+    // Lightweight stand-in for HoodieInstant
+    static class Instant {
+        final int id;
+        static long compareCount = 0;
+
+        Instant(int id) {
+            this.id = id;
+        }
+
+        @Override
+        public boolean equals(Object o) {
+            compareCount++;
+            if (this == o) return true;
+            if (!(o instanceof Instant)) return false;
+            return this.id == ((Instant) o).id;
+        }
+
+        @Override
+        public int hashCode() {
+            return Integer.hashCode(id);
+        }
+    }
+
+    // Defective: List.contains — O(N×M)
+    static List<Instant> appendLoadedInstants_slow(List<Instant> existing, List<Instant> loaded) {
+        // existingInstants is a plain List
+        List<Instant> existingInstants = new ArrayList<>(existing);
+        return loaded.stream()
+            .filter(instant -> !existingInstants.contains(instant))
+            .collect(Collectors.toList());
+    }
+
+    // Fixed: Set.contains — O(N+M)
+    static List<Instant> appendLoadedInstants_fast(List<Instant> existing, List<Instant> loaded) {
+        Set<Instant> existingSet = new HashSet<>(existing);
+        return loaded.stream()
+            .filter(instant -> !existingSet.contains(instant))
+            .collect(Collectors.toList());
+    }
+
+    static void test(String name, int existingSize, int loadedSize, int overlapSize) {
+        List<Instant> existing = new ArrayList<>();
+        for (int i = 0; i < existingSize; i++) existing.add(new Instant(i));
+
+        // loaded = first overlapSize are duplicates, rest are new
+        List<Instant> loaded = new ArrayList<>();
+        for (int i = 0; i < overlapSize; i++) loaded.add(new Instant(i));            // duplicates
+        for (int i = existingSize; i < existingSize + (loadedSize - overlapSize); i++) {
+            loaded.add(new Instant(i));  // new instants
+        }
+
+        Instant.compareCount = 0;
+        List<Instant> resultSlow = appendLoadedInstants_slow(existing, loaded);
+        long slowCount = Instant.compareCount;
+
+        Instant.compareCount = 0;
+        List<Instant> resultFast = appendLoadedInstants_fast(existing, loaded);
+        long fastCount = Instant.compareCount;
+
+        // Correctness: both must return the same new instants
+        assert resultSlow.size() == resultFast.size()
+            : "Size mismatch slow=" + resultSlow.size() + " fast=" + resultFast.size();
+
+        Set<Integer> slowIds = resultSlow.stream().map(x -> x.id).collect(Collectors.toSet());
+        Set<Integer> fastIds = resultFast.stream().map(x -> x.id).collect(Collectors.toSet());
+        assert slowIds.equals(fastIds) : "Result mismatch";
+
+        long ratio = fastCount == 0 ? 0 : slowCount / fastCount;
+        System.out.printf("  %-40s existing=%4d loaded=%4d overlap=%4d  slow=%8d fast=%8d ratio=%4dx%n",
+            name, existingSize, loadedSize, overlapSize, slowCount, fastCount, ratio);
+    }
+
+    public static void main(String[] args) {
+        int pass = 0;
+        int fail = 0;
+
+        System.out.println("hudi-0001: BaseHoodieTimeline.appendLoadedInstants");
+
+        // Test 1: correctness with small input
+        try {
+            test("correctness-small", 10, 10, 5);
+            pass++;
+        } catch (AssertionError e) {
+            System.out.println("  FAIL correctness-small: " + e.getMessage()); fail++;
+        }
+
+        // Test 2: slow path is measurably more expensive than fast path at N=500
+        Instant.compareCount = 0;
+        List<Instant> existing500 = new ArrayList<>();
+        for (int i = 0; i < 500; i++) existing500.add(new Instant(i));
+        List<Instant> loaded500 = new ArrayList<>();
+        for (int i = 0; i < 250; i++) loaded500.add(new Instant(i));         // duplicates
+        for (int i = 500; i < 750; i++) loaded500.add(new Instant(i));       // new
+
+        Instant.compareCount = 0;
+        appendLoadedInstants_slow(existing500, loaded500);
+        long slowOps = Instant.compareCount;
+
+        Instant.compareCount = 0;
+        appendLoadedInstants_fast(existing500, loaded500);
+        long fastOps = Instant.compareCount;
+
+        System.out.printf("  %-40s slow=%8d fast=%8d%n", "N=500-ops-comparison", slowOps, fastOps);
+        if (slowOps > fastOps * 10) {
+            System.out.println("  PASS slow-path > 10x fast-path at N=500"); pass++;
+        } else {
+            System.out.println("  FAIL expected slow > 10x fast"); fail++;
+        }
+
+        // Test 3: no overlap — all new instants
+        try {
+            test("all-new", 200, 200, 0);
+            pass++;
+        } catch (AssertionError e) {
+            System.out.println("  FAIL all-new: " + e.getMessage()); fail++;
+        }
+
+        // Test 4: full overlap — all duplicates
+        try {
+            test("all-dup", 200, 100, 100);
+            pass++;
+        } catch (AssertionError e) {
+            System.out.println("  FAIL all-dup: " + e.getMessage()); fail++;
+        }
+
+        System.out.println(pass + "/" + (pass + fail) + " PASS");
+        if (fail > 0) throw new RuntimeException(fail + " tests failed");
+    }
+}

defects/iceberg/patch/iceberg-0001-schemaupdate-applychanges-list-deletes.md

@@ -0,0 +1,82 @@
+# iceberg-0001 — SchemaUpdate.ApplyChanges List<Integer> deletes O(F×D)
+
+## File
+`core/src/main/java/org/apache/iceberg/SchemaUpdate.java`
+Lines 59, 661, 717
+
+## Defect
+
+```java
+// Line 59 — field declaration
+private final List<Integer> deletes = Lists.newArrayList();  // ArrayList
+
+// Line 660–663 — field() visitor — called ONCE PER SCHEMA FIELD
+@Override
+public Type field(Types.NestedField field, Type fieldResult) {
+    int fieldId = field.fieldId();
+    if (deletes.contains(fieldId)) {    // O(D) scan per field visit
+        return null;
+    }
+    // ...
+}
+
+// Line 716–718 — map() visitor
+public Type map(Types.MapType map, Type kResult, Type valueResult) {
+    int keyId = map.fields().get(0).fieldId();
+    if (deletes.contains(keyId)) {      // O(D) scan per map
+        throw new IllegalArgumentException("Cannot delete map keys: " + map);
+    }
+```
+
+`TypeUtil.visit(schema, new ApplyChanges(...))` walks every field in the schema tree —
+O(F) calls to `field()`. Each call does `deletes.contains()` on a `List<Integer>` — O(D) per call.
+
+Also in the outer `applyChanges()` (line 533–588), the loop over `identifierFieldNames` calls
+`deletes.contains()` twice per identifier field (lines 549, 556) plus a while-loop over parent
+chain — in total O(I × depth × D) for the validation pass.
+
+Total schema visitor cost: **O(F × D)** where F = schema field count, D = delete count.
+
+## Fix
+
+Change the backing store of `deletes` from `ArrayList` to `HashSet`. The field is only ever
+used for `contains()` checks and `add()` — no index-based access — so a `HashSet<Integer>` is
+a drop-in replacement.
+
+```java
+// Before:
+private final List<Integer> deletes = Lists.newArrayList();
+
+// After:
+private final Set<Integer> deletes = Sets.newHashSet();
+```
+
+The `ApplyChanges` inner class also holds a `List<Integer> deletes` at line 591. Change it to
+`Set<Integer>` and update the constructor parameter at line 597.
+
+```java
+// Before (line 591):
+private final List<Integer> deletes;
+
+// After:
+private final Set<Integer> deletes;
+```
+
+All `.contains()` calls become O(1). No other code changes required — `add()` and `contains()`
+are both valid `Set` operations.
+
+## Complexity
+
+| | Before | After |
+|-|--------|-------|
+| deletes.contains() per field | O(D) | O(1) |
+| full schema traversal | O(F × D) | O(F) |
+
+At F=1000 fields, D=100 deletes: 100 000 comparisons → 1000. **100x fewer operations.**
+For Iceberg tables with wide schemas (Parquet files with 500+ columns), this is significant.
+
+## Severity
+
+HIGH — `applyChanges()` is called on every `updateSchema()` transaction commit and on every
+`SchemaUpdate.apply()` in the table metadata path. Wide schemas with batch deletes amplify
+the O(F×D) factor. Schema evolution on large analytical tables hits this on every DDL operation.

defects/iceberg/unit/IcebergSchemaUpdateDeletesAlgorithm.java

@@ -0,0 +1,186 @@
+package unit;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * iceberg-0001: SchemaUpdate.ApplyChanges List<Integer> deletes O(F×D) → HashSet O(F)
+ *
+ * Simulates the ApplyChanges.field() visitor pattern: called once per schema field,
+ * each call checks if the field is in the deletes collection.
+ *
+ * Compile: javac -d . IcebergSchemaUpdateDeletesAlgorithm.java
+ * Run:     java -ea unit.IcebergSchemaUpdateDeletesAlgorithm
+ */
+public class IcebergSchemaUpdateDeletesAlgorithm {
+
+    static long compareCount = 0;
+
+    // Defective: List<Integer> deletes — O(D) per field visit
+    static class ApplyChanges_slow {
+        private final List<Integer> deletes;
+
+        ApplyChanges_slow(List<Integer> deletes) {
+            this.deletes = deletes;
+        }
+
+        // Called once per field in schema traversal
+        boolean isDeleted(int fieldId) {
+            for (int d : deletes) {
+                compareCount++;
+                if (d == fieldId) return true;
+            }
+            return false;
+        }
+
+        int visitSchema(int[] fieldIds) {
+            int deletedCount = 0;
+            for (int fid : fieldIds) {
+                if (isDeleted(fid)) deletedCount++;
+            }
+            return deletedCount;
+        }
+    }
+
+    // Fixed: Set<Integer> deletes — O(1) per field visit
+    static class ApplyChanges_fast {
+        private final Set<Integer> deletes;
+
+        ApplyChanges_fast(Set<Integer> deletes) {
+            this.deletes = deletes;
+        }
+
+        boolean isDeleted(int fieldId) {
+            compareCount++;  // one hash lookup
+            return deletes.contains(fieldId);
+        }
+
+        int visitSchema(int[] fieldIds) {
+            int deletedCount = 0;
+            for (int fid : fieldIds) {
+                if (isDeleted(fid)) deletedCount++;
+            }
+            return deletedCount;
+        }
+    }
+
+    static void assertEq(String label, Object expected, Object actual) {
+        if (!expected.equals(actual)) {
+            throw new AssertionError(label + ": expected " + expected + " but got " + actual);
+        }
+    }
+
+    public static void main(String[] args) {
+        int pass = 0;
+        int fail = 0;
+
+        System.out.println("iceberg-0001: SchemaUpdate.ApplyChanges deletes List → HashSet");
+
+        // Test 1: correctness — same deleted count for slow and fast
+        try {
+            int fieldCount = 50;
+            int deleteCount = 10;
+            int[] fields = new int[fieldCount];
+            for (int i = 0; i < fieldCount; i++) fields[i] = i;
+
+            List<Integer> deleteList = new ArrayList<>();
+            Set<Integer> deleteSet = new HashSet<>();
+            for (int i = 0; i < deleteCount; i++) {
+                deleteList.add(i * 5);
+                deleteSet.add(i * 5);
+            }
+
+            compareCount = 0;
+            int slowDeleted = new ApplyChanges_slow(deleteList).visitSchema(fields);
+            compareCount = 0;
+            int fastDeleted = new ApplyChanges_fast(deleteSet).visitSchema(fields);
+            assertEq("deleted-count", slowDeleted, fastDeleted);
+            System.out.println("  PASS correctness-F=50-D=10 deleted=" + slowDeleted);
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL correctness: " + e.getMessage()); fail++; }
+
+        // Test 2: no deletes — correctness
+        try {
+            int[] fields = new int[]{1, 2, 3, 4, 5};
+            int slow = new ApplyChanges_slow(new ArrayList<>()).visitSchema(fields);
+            int fast = new ApplyChanges_fast(new HashSet<>()).visitSchema(fields);
+            assertEq("no-deletes", slow, fast);
+            assertEq("no-deletes-zero", 0, slow);
+            System.out.println("  PASS no-deletes");
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL no-deletes: " + e.getMessage()); fail++; }
+
+        // Test 3: all fields deleted — correctness
+        try {
+            int[] fields = new int[]{10, 20, 30};
+            List<Integer> dl = new ArrayList<>(); dl.add(10); dl.add(20); dl.add(30);
+            Set<Integer> ds = new HashSet<>(dl);
+            int slow = new ApplyChanges_slow(dl).visitSchema(fields);
+            int fast = new ApplyChanges_fast(ds).visitSchema(fields);
+            assertEq("all-deleted", slow, fast);
+            assertEq("all-deleted-count", 3, slow);
+            System.out.println("  PASS all-deleted");
+            pass++;
+        } catch (AssertionError e) { System.out.println("  FAIL all-deleted: " + e.getMessage()); fail++; }
+
+        // Test 4: operation count at F=1000, D=100
+        {
+            int F = 1000, D = 100;
+            int[] fields = new int[F];
+            for (int i = 0; i < F; i++) fields[i] = i;
+            List<Integer> dl = new ArrayList<>();
+            Set<Integer> ds = new HashSet<>();
+            for (int i = 0; i < D; i++) { dl.add(i); ds.add(i); }
+
+            compareCount = 0;
+            new ApplyChanges_slow(dl).visitSchema(fields);
+            long slowOps = compareCount;
+
+            compareCount = 0;
+            new ApplyChanges_fast(ds).visitSchema(fields);
+            long fastOps = compareCount;
+
+            System.out.printf("  F=1000 D=100: slow=%8d fast=%6d ratio=%4dx%n",
+                slowOps, fastOps, slowOps / fastOps);
+
+            if (slowOps > fastOps * 10) {
+                System.out.println("  PASS slow > 10x fast at F=1000 D=100"); pass++;
+            } else {
+                System.out.println("  FAIL expected slow > 10x fast"); fail++;
+            }
+        }
+
+        // Test 5: F=500, D=50 — wide schema typical for Iceberg analytics
+        {
+            int F = 500, D = 50;
+            int[] fields = new int[F];
+            for (int i = 0; i < F; i++) fields[i] = i;
+            List<Integer> dl = new ArrayList<>();
+            Set<Integer> ds = new HashSet<>();
+            for (int i = 0; i < D; i++) { dl.add(i * 10); ds.add(i * 10); }
+
+            compareCount = 0;
+            int slowResult = new ApplyChanges_slow(dl).visitSchema(fields);
+            long slowOps = compareCount;
+
+            compareCount = 0;
+            int fastResult = new ApplyChanges_fast(ds).visitSchema(fields);
+            long fastOps = compareCount;
+
+            assertEq("wide-schema-result", slowResult, fastResult);
+            System.out.printf("  F=500 D=50: slow=%8d fast=%6d ratio=%4dx%n",
+                slowOps, fastOps, slowOps / fastOps);
+
+            if (slowOps > fastOps * 5) {
+                System.out.println("  PASS slow > 5x fast at F=500 D=50"); pass++;
+            } else {
+                System.out.println("  FAIL expected slow > 5x fast"); fail++;
+            }
+        }
+
+        System.out.println(pass + "/" + (pass + fail) + " PASS");
+        if (fail > 0) throw new RuntimeException(fail + " tests failed");
+    }
+}

defects/samza/patch/samza-CLEAN.md

@@ -0,0 +1,12 @@
+# samza — CLEAN
+
+Scanned:
+- `samza-core/src/main/java/org/apache/samza/execution/` — JobGraph, JobNode, IntermediateStreamManager
+- `samza-core/src/main/java/org/apache/samza/container/grouper/` — GroupByContainerIds, SSPGrouperProxy, GroupByPartition
+- `samza-core/src/main/java/org/apache/samza/clustermanager/` — StandbyContainerManager
+
+JobGraph.topologicalSort() and findReachable() use `Set<JobNode> visited = new HashSet<>()` —
+O(1) membership test. GroupByContainerIds uses `Set<TaskName> assignedTasks = new HashSet<>()`.
+StandbyContainerManager resourceRequests is a `HashSet<SamzaResourceRequest>`.
+
+No CWE-407 defects found.

whitepaper/MD5SUMS

@@ -1 +1 @@
-ddfb6a4041328defcb9c9db0abb2bc2b  undefect-cwe407-2026-03-27.pdf
+93e63b76eed03e223cacf4b27544c527  undefect-cwe407-2026-03-27.pdf

whitepaper/full-paper.md

@@ -39,8 +39,8 @@ A single well-crafted implementation serves as the genetic blueprint.
 4. **Harvest Stage:** Mature implementations compile into comprehensive documentation, ready for use
 
 Code propagates according to its kind — clean architecture begets clean implementations,
-elegant solutions inspire elegant variations. The process of generating 458 validated
-defect patches across 207 ecosystems in a single research wave demonstrates how truth,
+elegant solutions inspire elegant variations. The process of generating 462 validated
+defect patches across 209 ecosystems in a single research wave demonstrates how truth,
 properly seeded, multiplies. Each tested patch validates the correctness of the original
 diagnosis & extends light into new programming paradigms.
 
@@ -159,7 +159,7 @@ the missing linkages, applied them, tested them, and benchmarked them across eve
 confirmed site — compiler, routing, database, build tool, event streaming, web framework,
 query optimizer, and browser runtime.
 
-**458 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds).
+**462 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds).
 1 fixable-upstream (Erlang OTP). 1 fixable-pending (swipl-0003). 2 not-worth-fixing.
 3 unpatched (Minecraft, Create mod). No language left behind.
 
@@ -555,6 +555,10 @@ stacks, Spark schemas — this is the dominant build cost.
 | hive-0002 | Apache Hive | `optimizer/GenMRProcContext.java:142` — `List<FileSinkOperator>.contains()` in file sink dedup | **PATCHED** |
 | spark-0001 | Apache Spark | `sql/catalyst/.../analysis/Analyzer.scala:3286` — `ArrayBuffer[AggregateExpression].contains(agg)` in window func extraction | **PATCHED** |
 | spark-0002 | Apache Spark | `core/src/main/scala/.../scheduler/DAGScheduler.scala` — 6 BFS traversal functions use `ListBuffer.remove(0)` O(N) dequeue; O(N²) total; fix: `ArrayDeque` | **PATCHED** |
+| hudi-0001 | Apache Hudi | `BaseHoodieTimeline.java:126` — `List<HoodieInstant>.contains()` in appendLoadedInstants stream filter; O(N×M) (625×) | **PATCHED** |
+| hudi-0002 | Apache Hudi | `InternalSchemaUtils.java:69,113` — `ArrayList<Integer>.contains()` in pruneInternalSchema forEach+pruneType; O(N²)+O(F×D) (90×) | **PATCHED** |
+| hudi-0003 | Apache Hudi | `HoodieTableMetadataUtil.java:1006` — `List<String>.contains()` in log file dedup filter; O(N×M) (312×) | **PATCHED** |
+| iceberg-0001 | Apache Iceberg | `SchemaUpdate.java:59,661,717` — `List<Integer> deletes.contains()` per field in schema visitor; O(F×D) (95×) | **PATCHED** |
 | luigi-0001 | Luigi (Python) | `luigi/tools/deps.py:dfs_paths` — `set(path)` rebuilt from list on every recursive DFS call | **PATCHED** |
 | ray-0001 | Ray | `python/ray/autoscaler/_private/local/node_provider.py:79-83,147-149` — `list_of_node_ips = list(...)` then `for worker_ip in workers: if worker_ip not in list_of_node_ips`; O(N²) cluster reconciliation in `ClusterState` and `OnPremCoordinatorState`; fix: `set(worker_ips)` (300×) | **PATCHED** |
 | cel-0001 | Celery | `celery/canvas.py:702-706` — `append_to_list_option()` uses `if value not in items` where items is a list; called inside chain-build loops O(T×E) times; O(T×E×L) total; fix: parallel set mirror for O(1) dedup | **PATCHED** |
@@ -733,7 +737,7 @@ where D is the depth of the diamond chain. For a diamond of depth 10, that is 2^
 1,024 redundant node visits per edge check. Large modpacks produce diamond dependency
 chains with depths in this range.
 
-**458 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP — sltab patch). 1 fixable-pending (swipl-0003 attr_unify_hook). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). 10 CLEAN (WireGuard-tools, Solana, git, JGit, Dask, OSRM, Buck2, DGL, Protocol Buffers, gRPC Python).**
+**462 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP — sltab patch). 1 fixable-pending (swipl-0003 attr_unify_hook). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). 12 CLEAN (WireGuard-tools, Solana, git, JGit, Dask, OSRM, Buck2, DGL, Protocol Buffers, gRPC Python, Apache Beam, Apache Samza).**
 
 ---
 
@@ -1304,6 +1308,24 @@ task, making each reconciliation cycle O(N²). At N=300 tasks the defective path
 comparisons vs 900 map lookups (150×). The fix is a `map[string]*wfv1.DAGTask` built at context
 construction — identical to what `dagValidationContext` in `validate.go` already does correctly.
 
+**Apache Hudi — hudi-0001/2/3 (HIGH/MEDIUM, 625×/90×/312×)**
+
+Hudi's timeline layer contains three independent CWE-407 sites. `BaseHoodieTimeline.appendLoadedInstants()`
+filters duplicates via `List<HoodieInstant>.contains()` inside a stream filter — O(N×M) per incremental
+load — fixed by pre-converting the existing timeline to `HashSet` (625× at N=500). `InternalSchemaUtils.pruneInternalSchema()`
+builds `topParentFieldIds` as `ArrayList` and calls `.contains()` per projected column (O(N²)) while
+the recursive `pruneType()` also scans `fieldIds` per schema tree node (O(F×D)) — fixed with `LinkedHashSet`/`HashSet`
+(90×). `HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs()` filters log file paths with a
+`List<String>.contains()` stream predicate — O(N×M) on every RLI delta commit — fixed with `HashSet<String>` (312×).
+
+**Apache Iceberg — iceberg-0001 (HIGH, 95×)**
+
+`SchemaUpdate.ApplyChanges` holds `private final List<Integer> deletes` and calls `deletes.contains(fieldId)`
+once per field during `TypeUtil.visit()` schema traversal — O(F×D) for F fields and D pending deletes.
+This fires on every `updateSchema()` DDL commit. Fix: change `deletes` to `HashSet<Integer>`. Apache Beam
+and Apache Samza are CLEAN: Beam's pipeline graph uses `ImmutableSet`/`HashSet` throughout; Samza's
+`topologicalSort()` uses `HashSet<JobNode> visited`.
+
 **CFEngine** — **cfe-0001/0002/0003 PATCHED.** `getindices()`, `unique()`, and
 `maparray()` all used `RlistAppendScalarIdemp()` — which calls `RlistKeyIn()`, an O(N)
 linked-list walk — as a dedup primitive. `unique()` is a first-class CFEngine policy