diff --git a/defects/beam/patch/beam-CLEAN.md b/defects/beam/patch/beam-CLEAN.md new file mode 100644 index 000000000..39ddb4fac --- /dev/null +++ b/defects/beam/patch/beam-CLEAN.md @@ -0,0 +1,13 @@ +# beam — CLEAN + +Scanned: +- `sdks/java/core` — transforms, values, construction graph, PipelineValidator +- `runners/core-java` — SimplePushbackSideInputDoFnRunner, SideInputHandler, WatermarkManager, TimerUpdate +- `runners/direct-java` — KeyedPValueTrackingVisitor, EvaluationContext, SideInputContainer, DirectRunner +- `runners/flink` — FlinkRunner, SideInputInitializer, FlinkStateInternals +- `runners/spark` — SparkSideInputReader, CachedSideInputReader +- `runners/google-cloud-dataflow-java` — DataflowRunner (experiments List.contains — but O(4) fixed-length, not a loop) +- `sdks/python/apache_beam/transforms/` — Python transforms + +All `contains()` calls in hot paths use HashSet, ImmutableSet, or operate on trivially small +fixed-length lists (experiment flags, ~4 items). No CWE-407 defects found. diff --git a/defects/hudi/patch/hudi-0001-appendloadedinstants-list-contains.md b/defects/hudi/patch/hudi-0001-appendloadedinstants-list-contains.md new file mode 100644 index 000000000..4add6c870 --- /dev/null +++ b/defects/hudi/patch/hudi-0001-appendloadedinstants-list-contains.md @@ -0,0 +1,67 @@ +# hudi-0001 — BaseHoodieTimeline.appendLoadedInstants List.contains O(N×M) + +## File +`hudi-common/src/main/java/org/apache/hudi/common/table/timeline/BaseHoodieTimeline.java` +Line 123–131 + +## Defect + +```java +protected void appendLoadedInstants(List loadedInstants) { + List existingInstants = getInstants(); // List + List newInstants = loadedInstants.stream() + .filter(instant -> !existingInstants.contains(instant)) // O(M) per call + .collect(Collectors.toList()); + if (!newInstants.isEmpty()) { + appendInstants(newInstants); + } +} +``` + +`existingInstants` is a `List` (the backing field `private List instants`). +`List.contains()` performs a linear scan — O(M) per invocation. +The stream filter calls it once per element of `loadedInstants` — N calls total. + +Total complexity: **O(N × M)** where N = loadedInstants.size(), M = existing timeline size. + +In a Hudi table with a long history (thousands of commits), M can be large. This method is called +during incremental timeline loading (time-range and limit-based), so both N and M can be in the +thousands during compaction or restore operations. + +## Fix + +Convert `existingInstants` to a `HashSet` before the filter. `HoodieInstant` implements `equals()` +and `hashCode()` based on timestamp+action+state — so a HashSet works correctly. + +```java +protected void appendLoadedInstants(List loadedInstants) { + Set existingSet = new HashSet<>(getInstants()); // O(M) one-time + List newInstants = loadedInstants.stream() + .filter(instant -> !existingSet.contains(instant)) // O(1) per call + .collect(Collectors.toList()); + if (!newInstants.isEmpty()) { + appendInstants(newInstants); + } +} +``` + +## Complexity + +| | Before | After | +|-|--------|-------| +| per-element membership test | O(M) | O(1) | +| full filter pass | O(N × M) | O(N + M) | + +At N=M=10 000: 100 000 000 comparisons → 20 000 comparisons. **5000x fewer operations.** + +## Severity + +MEDIUM — triggered during incremental timeline load on large tables (restore, compaction, archival). +Degrades linearly with table history depth. + +## Import required + +```java +import java.util.HashSet; +import java.util.Set; +``` diff --git a/defects/hudi/patch/hudi-0002-pruneinternalschema-list-contains.md b/defects/hudi/patch/hudi-0002-pruneinternalschema-list-contains.md new file mode 100644 index 000000000..f9745dca6 --- /dev/null +++ b/defects/hudi/patch/hudi-0002-pruneinternalschema-list-contains.md @@ -0,0 +1,76 @@ +# hudi-0002 — InternalSchemaUtils.pruneInternalSchema ArrayList.contains O(N²) + pruneType O(F×D) + +## File +`hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java` +Lines 66–72 (topParentFieldIds dedup) and 105–160 (pruneType) + +## Defect — Part A: topParentFieldIds dedup (O(N²)) + +```java +List topParentFieldIds = new ArrayList<>(); +names.stream().forEach(f -> { + int id = schema.findIdByName(f.split("\\.")[0]); + if (!topParentFieldIds.contains(id)) { // O(N) scan of ArrayList per call + topParentFieldIds.add(id); + } +}); +``` + +`topParentFieldIds` is `ArrayList`. `.contains(id)` is O(N) — called once per name. +Total: O(N²) where N = names.size(). For a schema with many projected columns this becomes quadratic. + +## Defect — Part B: pruneType field membership (O(F×D)) + +```java +private static Type pruneType(Type type, List fieldIds) { + // ...RECORD case: + for (Types.Field f : fields) { + Type newType = pruneType(f.type(), fieldIds); + if (fieldIds.contains(f.fieldId())) { // O(D) scan per field + newTypes.add(f.type()); + } + } + // ...ARRAY case: + if (fieldIds.contains(array.elementId())) { // O(D) per array + // ...MAP case: + if (fieldIds.contains(map.valueId())) { // O(D) per map +``` + +`fieldIds` is `List`. Called recursively over the full schema tree (F nodes). +Total: O(F × D) where F = total schema fields, D = projected field count. + +## Fix + +```java +// Part A: use LinkedHashSet to preserve insertion order and deduplicate in O(1) +Set topParentFieldIdSet = new LinkedHashSet<>(); +names.stream().forEach(f -> { + int id = schema.findIdByName(f.split("\\.")[0]); + topParentFieldIdSet.add(id); // HashSet.add deduplicates — O(1) amortized +}); +List topParentFieldIds = new ArrayList<>(topParentFieldIdSet); +``` + +```java +// Part B: convert fieldIds to HashSet before entering pruneType +private static Type pruneType(Type type, Set fieldIds) { + // ...same logic, but fieldIds.contains() is O(1) +} +// Call site: pruneType(schema.getRecord(), new HashSet<>(fieldIds)) +``` + +## Complexity + +| | Before | After | +|-|--------|-------| +| topParentFieldIds dedup | O(N²) | O(N) | +| pruneType per-field check | O(D) | O(1) | +| full pruneType traversal | O(F × D) | O(F) | + +At F=D=500 fields: 250 000 comparisons → 500. **500x fewer operations.** + +## Severity + +MEDIUM — triggered on every call to `pruneInternalSchema()`, which is called during query +projection pushdown, Spark read, and schema evolution. Schemas with many nested columns amplify +both defects simultaneously. diff --git a/defects/hudi/patch/hudi-0003-metadatautil-logfilepaths-list-contains.md b/defects/hudi/patch/hudi-0003-metadatautil-logfilepaths-list-contains.md new file mode 100644 index 000000000..8daf27680 --- /dev/null +++ b/defects/hudi/patch/hudi-0003-metadatautil-logfilepaths-list-contains.md @@ -0,0 +1,62 @@ +# hudi-0003 — HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs List.contains O(N×M) + +## File +`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java` +Line 1005–1007 + +## Defect + +```java +public static Pair, Set> getRevivedAndDeletedKeysFromMergedLogs( + ..., List logFilePaths, ..., List currentLogFilePaths, ...) { + + List logFilePathsWithoutCurrentLogFiles = logFilePaths.stream() + .filter(logFilePath -> !currentLogFilePaths.contains(logFilePath)) // O(M) per element + .collect(toList()); +``` + +`currentLogFilePaths` is a `List` (constructed at line 925 via `Collectors.toList()`). +`List.contains()` does a linear scan — O(M) per call where M = currentLogFilePaths.size(). +The stream filter applies this once per element of `logFilePaths` — N calls total. + +Total complexity: **O(N × M)** where N = total log file paths, M = current log file paths count. + +This method is called during every Record-Level Index (RLI) update, which happens on each delta +commit. In a table with many log files per partition (large MOR tables, frequent compaction), both +N and M can be in the hundreds. + +## Fix + +Convert `currentLogFilePaths` to a `HashSet` before the filter. String equality is +well-defined and path strings are unique identifiers. + +```java +Set currentLogFilePathSet = new HashSet<>(currentLogFilePaths); // O(M) once +List logFilePathsWithoutCurrentLogFiles = logFilePaths.stream() + .filter(logFilePath -> !currentLogFilePathSet.contains(logFilePath)) // O(1) per element + .collect(toList()); +``` + +The call site (line 925–927) creates `currentLogFilePaths` as a List and passes it to this method. +Alternatively, build it as a `HashSet` at the call site. + +## Complexity + +| | Before | After | +|-|--------|-------| +| per-element membership test | O(M) | O(1) | +| full filter pass | O(N × M) | O(N + M) | + +At N=M=500 log files: 250 000 comparisons → 1000. **250x fewer operations.** + +## Severity + +MEDIUM — triggered on every delta commit when RLI is enabled. MOR (Merge-on-Read) tables with +high write frequency and many log files per filegroup amplify this significantly. + +## Import required + +```java +import java.util.HashSet; +import java.util.Set; +``` diff --git a/defects/hudi/unit/HudiLogFilePathsAlgorithm.java b/defects/hudi/unit/HudiLogFilePathsAlgorithm.java new file mode 100644 index 000000000..1660e02cf --- /dev/null +++ b/defects/hudi/unit/HudiLogFilePathsAlgorithm.java @@ -0,0 +1,135 @@ +package unit; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +/** + * hudi-0003: HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs + * List.contains O(N×M) → HashSet O(N+M) + * + * Simulates the log file path deduplication filter in the RLI update path. + * + * Compile: javac -d . HudiLogFilePathsAlgorithm.java + * Run: java -ea unit.HudiLogFilePathsAlgorithm + */ +public class HudiLogFilePathsAlgorithm { + + static long compareCount = 0; + + // Defective: List.contains — O(N×M) + static List filterLogPaths_slow(List allPaths, List currentPaths) { + return allPaths.stream() + .filter(path -> { + for (String c : currentPaths) { + compareCount++; + if (c.equals(path)) return false; + } + return true; + }) + .collect(Collectors.toList()); + } + + // Fixed: HashSet.contains — O(N+M) + static List filterLogPaths_fast(List allPaths, List currentPaths) { + Set currentSet = new HashSet<>(currentPaths); + return allPaths.stream() + .filter(path -> { + compareCount++; + return !currentSet.contains(path); + }) + .collect(Collectors.toList()); + } + + static void assertEq(String label, Object expected, Object actual) { + if (!expected.equals(actual)) { + throw new AssertionError(label + ": expected " + expected + " but got " + actual); + } + } + + public static void main(String[] args) { + int pass = 0; + int fail = 0; + + System.out.println("hudi-0003: HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs"); + + // Test 1: correctness — small overlap + try { + List all = new ArrayList<>(); + List current = new ArrayList<>(); + for (int i = 0; i < 20; i++) all.add("/table/part/.hoodie_meta/file_" + i + ".log"); + for (int i = 0; i < 5; i++) current.add(all.get(i)); + + compareCount = 0; + List slow = filterLogPaths_slow(all, current); + compareCount = 0; + List fast = filterLogPaths_fast(all, current); + + assertEq("size", slow.size(), fast.size()); + assertEq("content", new HashSet<>(slow), new HashSet<>(fast)); + System.out.println(" PASS correctness-small overlap=" + current.size()); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL correctness: " + e.getMessage()); fail++; } + + // Test 2: correctness — no overlap + try { + List all = new ArrayList<>(); + List current = new ArrayList<>(); + for (int i = 0; i < 10; i++) all.add("/part/file_a_" + i + ".log"); + for (int i = 0; i < 5; i++) current.add("/part/file_b_" + i + ".log"); + + List slow = filterLogPaths_slow(all, current); + List fast = filterLogPaths_fast(all, current); + assertEq("no-overlap-size", slow.size(), fast.size()); + assertEq("no-overlap-all", 10, slow.size()); + System.out.println(" PASS no-overlap"); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL no-overlap: " + e.getMessage()); fail++; } + + // Test 3: correctness — full overlap + try { + List all = new ArrayList<>(); + for (int i = 0; i < 10; i++) all.add("/part/file_" + i + ".log"); + List current = new ArrayList<>(all); + + List slow = filterLogPaths_slow(all, current); + List fast = filterLogPaths_fast(all, current); + assertEq("full-overlap-size", slow.size(), fast.size()); + assertEq("full-overlap-empty", 0, slow.size()); + System.out.println(" PASS full-overlap"); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL full-overlap: " + e.getMessage()); fail++; } + + // Test 4: N=500, M=500 — operation count + { + List all = new ArrayList<>(); + List current = new ArrayList<>(); + for (int i = 0; i < 500; i++) all.add("/table/p0/.hoodie_meta/file_" + i + ".log"); + for (int i = 0; i < 250; i++) current.add(all.get(i)); // half overlap + for (int i = 500; i < 750; i++) current.add("/table/p0/.hoodie_meta/file_" + i + ".log"); // non-overlap current + + compareCount = 0; + List slowResult = filterLogPaths_slow(all, current); + long slowOps = compareCount; + + compareCount = 0; + List fastResult = filterLogPaths_fast(all, current); + long fastOps = compareCount; + + assertEq("N500-result", new HashSet<>(slowResult), new HashSet<>(fastResult)); + System.out.printf(" N=500 M=500: slow=%8d fast=%6d ratio=%4dx%n", + slowOps, fastOps, fastOps > 0 ? slowOps / fastOps : 0); + + if (slowOps > fastOps * 10) { + System.out.println(" PASS slow > 10x fast at N=500"); pass++; + } else { + System.out.println(" FAIL expected slow > 10x fast"); fail++; + } + } + + System.out.println(pass + "/" + (pass + fail) + " PASS"); + if (fail > 0) throw new RuntimeException(fail + " tests failed"); + } +} diff --git a/defects/hudi/unit/HudiPruneSchemaAlgorithm.java b/defects/hudi/unit/HudiPruneSchemaAlgorithm.java new file mode 100644 index 000000000..bcda3272a --- /dev/null +++ b/defects/hudi/unit/HudiPruneSchemaAlgorithm.java @@ -0,0 +1,171 @@ +package unit; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +/** + * hudi-0002: InternalSchemaUtils.pruneInternalSchema ArrayList.contains O(N²) + pruneType O(F×D) + * + * Part A: topParentFieldIds dedup — ArrayList.contains in forEach = O(N²) + * Part B: pruneType field membership — List.contains per field visit = O(F×D) + * + * Compile: javac -d . HudiPruneSchemaAlgorithm.java + * Run: java -ea unit.HudiPruneSchemaAlgorithm + */ +public class HudiPruneSchemaAlgorithm { + + static long compareCount = 0; + + // --- Part A: topParentFieldIds dedup --- + + // Defective: ArrayList.contains dedup — O(N²) + static List deduplicateParentIds_slow(List ids) { + List result = new ArrayList<>(); + for (int id : ids) { + // ArrayList.contains = linear scan + boolean found = false; + for (int existing : result) { + compareCount++; + if (existing == id) { found = true; break; } + } + if (!found) result.add(id); + } + return result; + } + + // Fixed: LinkedHashSet — O(N), preserves insertion order + static List deduplicateParentIds_fast(List ids) { + Set seen = new LinkedHashSet<>(); + for (int id : ids) { + compareCount++; // one hash op per entry + seen.add(id); + } + return new ArrayList<>(seen); + } + + // --- Part B: pruneType field membership --- + + // Defective: List fieldIds with contains() — O(F×D) + static int pruneType_slow(int[] schema, List fieldIds) { + int visited = 0; + for (int fieldId : schema) { + visited++; + for (int fid : fieldIds) { + compareCount++; + if (fid == fieldId) break; + } + } + return visited; + } + + // Fixed: Set fieldIds with contains() — O(F) + static int pruneType_fast(int[] schema, Set fieldIds) { + int visited = 0; + for (int fieldId : schema) { + visited++; + compareCount++; // O(1) hash lookup + fieldIds.contains(fieldId); + } + return visited; + } + + static void assertEq(String label, Object expected, Object actual) { + if (!expected.equals(actual)) { + throw new AssertionError(label + ": expected " + expected + " but got " + actual); + } + } + + public static void main(String[] args) { + int pass = 0; + int fail = 0; + + System.out.println("hudi-0002: InternalSchemaUtils.pruneInternalSchema"); + + // --- Part A tests --- + + // Test 1: correctness of dedup + try { + List ids = new ArrayList<>(); + for (int i = 0; i < 5; i++) { ids.add(i % 3); } // [0,1,2,0,1] + compareCount = 0; + List slow = deduplicateParentIds_slow(ids); + compareCount = 0; + List fast = deduplicateParentIds_fast(ids); + assertEq("dedup-size", slow.size(), fast.size()); + assertEq("dedup-content", new HashSet<>(slow), new HashSet<>(fast)); + System.out.println(" PASS dedup-correctness"); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL dedup-correctness: " + e.getMessage()); fail++; } + + // Test 2: slow > fast at N=300 + { + List ids300 = new ArrayList<>(); + for (int i = 0; i < 300; i++) ids300.add(i % 50); // 50 unique, many repeats + + compareCount = 0; + deduplicateParentIds_slow(ids300); + long slowOps = compareCount; + + compareCount = 0; + deduplicateParentIds_fast(ids300); + long fastOps = compareCount; + + System.out.printf(" dedup N=300(50unique): slow=%6d fast=%6d%n", slowOps, fastOps); + if (slowOps > fastOps * 5) { + System.out.println(" PASS slow > 5x fast for dedup"); pass++; + } else { + System.out.println(" FAIL expected slow > 5x fast for dedup"); fail++; + } + } + + // --- Part B tests --- + + // Test 3: correctness — both visit same fields + try { + int[] schema = new int[100]; + for (int i = 0; i < 100; i++) schema[i] = i; + List fieldIdList = new ArrayList<>(); + Set fieldIdSet = new HashSet<>(); + for (int i = 0; i < 20; i++) { fieldIdList.add(i * 5); fieldIdSet.add(i * 5); } + + compareCount = 0; + int slowVisited = pruneType_slow(schema, fieldIdList); + compareCount = 0; + int fastVisited = pruneType_fast(schema, fieldIdSet); + assertEq("pruneType-visited", slowVisited, fastVisited); + System.out.println(" PASS pruneType-correctness"); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL pruneType-correctness: " + e.getMessage()); fail++; } + + // Test 4: slow > fast at F=500, D=100 + { + int[] schema500 = new int[500]; + for (int i = 0; i < 500; i++) schema500[i] = i; + List fieldIdList = new ArrayList<>(); + Set fieldIdSet = new HashSet<>(); + for (int i = 0; i < 100; i++) { fieldIdList.add(i); fieldIdSet.add(i); } + + compareCount = 0; + pruneType_slow(schema500, fieldIdList); + long slowOps = compareCount; + + compareCount = 0; + pruneType_fast(schema500, fieldIdSet); + long fastOps = compareCount; + + System.out.printf(" pruneType F=500 D=100: slow=%6d fast=%6d%n", slowOps, fastOps); + if (slowOps > fastOps * 10) { + System.out.println(" PASS slow > 10x fast for pruneType"); pass++; + } else { + System.out.println(" FAIL expected slow > 10x fast for pruneType"); fail++; + } + } + + System.out.println(pass + "/" + (pass + fail) + " PASS"); + if (fail > 0) throw new RuntimeException(fail + " tests failed"); + } +} diff --git a/defects/hudi/unit/HudiTimelineListContainsAlgorithm.java b/defects/hudi/unit/HudiTimelineListContainsAlgorithm.java new file mode 100644 index 000000000..042680ab9 --- /dev/null +++ b/defects/hudi/unit/HudiTimelineListContainsAlgorithm.java @@ -0,0 +1,148 @@ +package unit; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +/** + * hudi-0001: BaseHoodieTimeline.appendLoadedInstants List.contains O(N×M) → HashSet O(N+M) + * + * Simulates the deduplication filter in appendLoadedInstants using a lightweight + * Instant stand-in (just an integer ID) to count comparison operations. + * + * Compile: javac -d . HudiTimelineListContainsAlgorithm.java + * Run: java -ea unit.HudiTimelineListContainsAlgorithm + */ +public class HudiTimelineListContainsAlgorithm { + + // Lightweight stand-in for HoodieInstant + static class Instant { + final int id; + static long compareCount = 0; + + Instant(int id) { + this.id = id; + } + + @Override + public boolean equals(Object o) { + compareCount++; + if (this == o) return true; + if (!(o instanceof Instant)) return false; + return this.id == ((Instant) o).id; + } + + @Override + public int hashCode() { + return Integer.hashCode(id); + } + } + + // Defective: List.contains — O(N×M) + static List appendLoadedInstants_slow(List existing, List loaded) { + // existingInstants is a plain List + List existingInstants = new ArrayList<>(existing); + return loaded.stream() + .filter(instant -> !existingInstants.contains(instant)) + .collect(Collectors.toList()); + } + + // Fixed: Set.contains — O(N+M) + static List appendLoadedInstants_fast(List existing, List loaded) { + Set existingSet = new HashSet<>(existing); + return loaded.stream() + .filter(instant -> !existingSet.contains(instant)) + .collect(Collectors.toList()); + } + + static void test(String name, int existingSize, int loadedSize, int overlapSize) { + List existing = new ArrayList<>(); + for (int i = 0; i < existingSize; i++) existing.add(new Instant(i)); + + // loaded = first overlapSize are duplicates, rest are new + List loaded = new ArrayList<>(); + for (int i = 0; i < overlapSize; i++) loaded.add(new Instant(i)); // duplicates + for (int i = existingSize; i < existingSize + (loadedSize - overlapSize); i++) { + loaded.add(new Instant(i)); // new instants + } + + Instant.compareCount = 0; + List resultSlow = appendLoadedInstants_slow(existing, loaded); + long slowCount = Instant.compareCount; + + Instant.compareCount = 0; + List resultFast = appendLoadedInstants_fast(existing, loaded); + long fastCount = Instant.compareCount; + + // Correctness: both must return the same new instants + assert resultSlow.size() == resultFast.size() + : "Size mismatch slow=" + resultSlow.size() + " fast=" + resultFast.size(); + + Set slowIds = resultSlow.stream().map(x -> x.id).collect(Collectors.toSet()); + Set fastIds = resultFast.stream().map(x -> x.id).collect(Collectors.toSet()); + assert slowIds.equals(fastIds) : "Result mismatch"; + + long ratio = fastCount == 0 ? 0 : slowCount / fastCount; + System.out.printf(" %-40s existing=%4d loaded=%4d overlap=%4d slow=%8d fast=%8d ratio=%4dx%n", + name, existingSize, loadedSize, overlapSize, slowCount, fastCount, ratio); + } + + public static void main(String[] args) { + int pass = 0; + int fail = 0; + + System.out.println("hudi-0001: BaseHoodieTimeline.appendLoadedInstants"); + + // Test 1: correctness with small input + try { + test("correctness-small", 10, 10, 5); + pass++; + } catch (AssertionError e) { + System.out.println(" FAIL correctness-small: " + e.getMessage()); fail++; + } + + // Test 2: slow path is measurably more expensive than fast path at N=500 + Instant.compareCount = 0; + List existing500 = new ArrayList<>(); + for (int i = 0; i < 500; i++) existing500.add(new Instant(i)); + List loaded500 = new ArrayList<>(); + for (int i = 0; i < 250; i++) loaded500.add(new Instant(i)); // duplicates + for (int i = 500; i < 750; i++) loaded500.add(new Instant(i)); // new + + Instant.compareCount = 0; + appendLoadedInstants_slow(existing500, loaded500); + long slowOps = Instant.compareCount; + + Instant.compareCount = 0; + appendLoadedInstants_fast(existing500, loaded500); + long fastOps = Instant.compareCount; + + System.out.printf(" %-40s slow=%8d fast=%8d%n", "N=500-ops-comparison", slowOps, fastOps); + if (slowOps > fastOps * 10) { + System.out.println(" PASS slow-path > 10x fast-path at N=500"); pass++; + } else { + System.out.println(" FAIL expected slow > 10x fast"); fail++; + } + + // Test 3: no overlap — all new instants + try { + test("all-new", 200, 200, 0); + pass++; + } catch (AssertionError e) { + System.out.println(" FAIL all-new: " + e.getMessage()); fail++; + } + + // Test 4: full overlap — all duplicates + try { + test("all-dup", 200, 100, 100); + pass++; + } catch (AssertionError e) { + System.out.println(" FAIL all-dup: " + e.getMessage()); fail++; + } + + System.out.println(pass + "/" + (pass + fail) + " PASS"); + if (fail > 0) throw new RuntimeException(fail + " tests failed"); + } +} diff --git a/defects/iceberg/patch/iceberg-0001-schemaupdate-applychanges-list-deletes.md b/defects/iceberg/patch/iceberg-0001-schemaupdate-applychanges-list-deletes.md new file mode 100644 index 000000000..60c075d0e --- /dev/null +++ b/defects/iceberg/patch/iceberg-0001-schemaupdate-applychanges-list-deletes.md @@ -0,0 +1,82 @@ +# iceberg-0001 — SchemaUpdate.ApplyChanges List deletes O(F×D) + +## File +`core/src/main/java/org/apache/iceberg/SchemaUpdate.java` +Lines 59, 661, 717 + +## Defect + +```java +// Line 59 — field declaration +private final List deletes = Lists.newArrayList(); // ArrayList + +// Line 660–663 — field() visitor — called ONCE PER SCHEMA FIELD +@Override +public Type field(Types.NestedField field, Type fieldResult) { + int fieldId = field.fieldId(); + if (deletes.contains(fieldId)) { // O(D) scan per field visit + return null; + } + // ... +} + +// Line 716–718 — map() visitor +public Type map(Types.MapType map, Type kResult, Type valueResult) { + int keyId = map.fields().get(0).fieldId(); + if (deletes.contains(keyId)) { // O(D) scan per map + throw new IllegalArgumentException("Cannot delete map keys: " + map); + } +``` + +`TypeUtil.visit(schema, new ApplyChanges(...))` walks every field in the schema tree — +O(F) calls to `field()`. Each call does `deletes.contains()` on a `List` — O(D) per call. + +Also in the outer `applyChanges()` (line 533–588), the loop over `identifierFieldNames` calls +`deletes.contains()` twice per identifier field (lines 549, 556) plus a while-loop over parent +chain — in total O(I × depth × D) for the validation pass. + +Total schema visitor cost: **O(F × D)** where F = schema field count, D = delete count. + +## Fix + +Change the backing store of `deletes` from `ArrayList` to `HashSet`. The field is only ever +used for `contains()` checks and `add()` — no index-based access — so a `HashSet` is +a drop-in replacement. + +```java +// Before: +private final List deletes = Lists.newArrayList(); + +// After: +private final Set deletes = Sets.newHashSet(); +``` + +The `ApplyChanges` inner class also holds a `List deletes` at line 591. Change it to +`Set` and update the constructor parameter at line 597. + +```java +// Before (line 591): +private final List deletes; + +// After: +private final Set deletes; +``` + +All `.contains()` calls become O(1). No other code changes required — `add()` and `contains()` +are both valid `Set` operations. + +## Complexity + +| | Before | After | +|-|--------|-------| +| deletes.contains() per field | O(D) | O(1) | +| full schema traversal | O(F × D) | O(F) | + +At F=1000 fields, D=100 deletes: 100 000 comparisons → 1000. **100x fewer operations.** +For Iceberg tables with wide schemas (Parquet files with 500+ columns), this is significant. + +## Severity + +HIGH — `applyChanges()` is called on every `updateSchema()` transaction commit and on every +`SchemaUpdate.apply()` in the table metadata path. Wide schemas with batch deletes amplify +the O(F×D) factor. Schema evolution on large analytical tables hits this on every DDL operation. diff --git a/defects/iceberg/unit/IcebergSchemaUpdateDeletesAlgorithm.java b/defects/iceberg/unit/IcebergSchemaUpdateDeletesAlgorithm.java new file mode 100644 index 000000000..a0d84d6cb --- /dev/null +++ b/defects/iceberg/unit/IcebergSchemaUpdateDeletesAlgorithm.java @@ -0,0 +1,186 @@ +package unit; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +/** + * iceberg-0001: SchemaUpdate.ApplyChanges List deletes O(F×D) → HashSet O(F) + * + * Simulates the ApplyChanges.field() visitor pattern: called once per schema field, + * each call checks if the field is in the deletes collection. + * + * Compile: javac -d . IcebergSchemaUpdateDeletesAlgorithm.java + * Run: java -ea unit.IcebergSchemaUpdateDeletesAlgorithm + */ +public class IcebergSchemaUpdateDeletesAlgorithm { + + static long compareCount = 0; + + // Defective: List deletes — O(D) per field visit + static class ApplyChanges_slow { + private final List deletes; + + ApplyChanges_slow(List deletes) { + this.deletes = deletes; + } + + // Called once per field in schema traversal + boolean isDeleted(int fieldId) { + for (int d : deletes) { + compareCount++; + if (d == fieldId) return true; + } + return false; + } + + int visitSchema(int[] fieldIds) { + int deletedCount = 0; + for (int fid : fieldIds) { + if (isDeleted(fid)) deletedCount++; + } + return deletedCount; + } + } + + // Fixed: Set deletes — O(1) per field visit + static class ApplyChanges_fast { + private final Set deletes; + + ApplyChanges_fast(Set deletes) { + this.deletes = deletes; + } + + boolean isDeleted(int fieldId) { + compareCount++; // one hash lookup + return deletes.contains(fieldId); + } + + int visitSchema(int[] fieldIds) { + int deletedCount = 0; + for (int fid : fieldIds) { + if (isDeleted(fid)) deletedCount++; + } + return deletedCount; + } + } + + static void assertEq(String label, Object expected, Object actual) { + if (!expected.equals(actual)) { + throw new AssertionError(label + ": expected " + expected + " but got " + actual); + } + } + + public static void main(String[] args) { + int pass = 0; + int fail = 0; + + System.out.println("iceberg-0001: SchemaUpdate.ApplyChanges deletes List → HashSet"); + + // Test 1: correctness — same deleted count for slow and fast + try { + int fieldCount = 50; + int deleteCount = 10; + int[] fields = new int[fieldCount]; + for (int i = 0; i < fieldCount; i++) fields[i] = i; + + List deleteList = new ArrayList<>(); + Set deleteSet = new HashSet<>(); + for (int i = 0; i < deleteCount; i++) { + deleteList.add(i * 5); + deleteSet.add(i * 5); + } + + compareCount = 0; + int slowDeleted = new ApplyChanges_slow(deleteList).visitSchema(fields); + compareCount = 0; + int fastDeleted = new ApplyChanges_fast(deleteSet).visitSchema(fields); + assertEq("deleted-count", slowDeleted, fastDeleted); + System.out.println(" PASS correctness-F=50-D=10 deleted=" + slowDeleted); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL correctness: " + e.getMessage()); fail++; } + + // Test 2: no deletes — correctness + try { + int[] fields = new int[]{1, 2, 3, 4, 5}; + int slow = new ApplyChanges_slow(new ArrayList<>()).visitSchema(fields); + int fast = new ApplyChanges_fast(new HashSet<>()).visitSchema(fields); + assertEq("no-deletes", slow, fast); + assertEq("no-deletes-zero", 0, slow); + System.out.println(" PASS no-deletes"); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL no-deletes: " + e.getMessage()); fail++; } + + // Test 3: all fields deleted — correctness + try { + int[] fields = new int[]{10, 20, 30}; + List dl = new ArrayList<>(); dl.add(10); dl.add(20); dl.add(30); + Set ds = new HashSet<>(dl); + int slow = new ApplyChanges_slow(dl).visitSchema(fields); + int fast = new ApplyChanges_fast(ds).visitSchema(fields); + assertEq("all-deleted", slow, fast); + assertEq("all-deleted-count", 3, slow); + System.out.println(" PASS all-deleted"); + pass++; + } catch (AssertionError e) { System.out.println(" FAIL all-deleted: " + e.getMessage()); fail++; } + + // Test 4: operation count at F=1000, D=100 + { + int F = 1000, D = 100; + int[] fields = new int[F]; + for (int i = 0; i < F; i++) fields[i] = i; + List dl = new ArrayList<>(); + Set ds = new HashSet<>(); + for (int i = 0; i < D; i++) { dl.add(i); ds.add(i); } + + compareCount = 0; + new ApplyChanges_slow(dl).visitSchema(fields); + long slowOps = compareCount; + + compareCount = 0; + new ApplyChanges_fast(ds).visitSchema(fields); + long fastOps = compareCount; + + System.out.printf(" F=1000 D=100: slow=%8d fast=%6d ratio=%4dx%n", + slowOps, fastOps, slowOps / fastOps); + + if (slowOps > fastOps * 10) { + System.out.println(" PASS slow > 10x fast at F=1000 D=100"); pass++; + } else { + System.out.println(" FAIL expected slow > 10x fast"); fail++; + } + } + + // Test 5: F=500, D=50 — wide schema typical for Iceberg analytics + { + int F = 500, D = 50; + int[] fields = new int[F]; + for (int i = 0; i < F; i++) fields[i] = i; + List dl = new ArrayList<>(); + Set ds = new HashSet<>(); + for (int i = 0; i < D; i++) { dl.add(i * 10); ds.add(i * 10); } + + compareCount = 0; + int slowResult = new ApplyChanges_slow(dl).visitSchema(fields); + long slowOps = compareCount; + + compareCount = 0; + int fastResult = new ApplyChanges_fast(ds).visitSchema(fields); + long fastOps = compareCount; + + assertEq("wide-schema-result", slowResult, fastResult); + System.out.printf(" F=500 D=50: slow=%8d fast=%6d ratio=%4dx%n", + slowOps, fastOps, slowOps / fastOps); + + if (slowOps > fastOps * 5) { + System.out.println(" PASS slow > 5x fast at F=500 D=50"); pass++; + } else { + System.out.println(" FAIL expected slow > 5x fast"); fail++; + } + } + + System.out.println(pass + "/" + (pass + fail) + " PASS"); + if (fail > 0) throw new RuntimeException(fail + " tests failed"); + } +} diff --git a/defects/samza/patch/samza-CLEAN.md b/defects/samza/patch/samza-CLEAN.md new file mode 100644 index 000000000..32fab750a --- /dev/null +++ b/defects/samza/patch/samza-CLEAN.md @@ -0,0 +1,12 @@ +# samza — CLEAN + +Scanned: +- `samza-core/src/main/java/org/apache/samza/execution/` — JobGraph, JobNode, IntermediateStreamManager +- `samza-core/src/main/java/org/apache/samza/container/grouper/` — GroupByContainerIds, SSPGrouperProxy, GroupByPartition +- `samza-core/src/main/java/org/apache/samza/clustermanager/` — StandbyContainerManager + +JobGraph.topologicalSort() and findReachable() use `Set visited = new HashSet<>()` — +O(1) membership test. GroupByContainerIds uses `Set assignedTasks = new HashSet<>()`. +StandbyContainerManager resourceRequests is a `HashSet`. + +No CWE-407 defects found. diff --git a/whitepaper/MD5SUMS b/whitepaper/MD5SUMS index 8c2bbd08d..11faed0fa 100644 --- a/whitepaper/MD5SUMS +++ b/whitepaper/MD5SUMS @@ -1 +1 @@ -ddfb6a4041328defcb9c9db0abb2bc2b undefect-cwe407-2026-03-27.pdf +93e63b76eed03e223cacf4b27544c527 undefect-cwe407-2026-03-27.pdf diff --git a/whitepaper/full-paper.md b/whitepaper/full-paper.md index ffaf63a40..62a9524c1 100644 --- a/whitepaper/full-paper.md +++ b/whitepaper/full-paper.md @@ -39,8 +39,8 @@ A single well-crafted implementation serves as the genetic blueprint. 4. **Harvest Stage:** Mature implementations compile into comprehensive documentation, ready for use Code propagates according to its kind — clean architecture begets clean implementations, -elegant solutions inspire elegant variations. The process of generating 458 validated -defect patches across 207 ecosystems in a single research wave demonstrates how truth, +elegant solutions inspire elegant variations. The process of generating 462 validated +defect patches across 209 ecosystems in a single research wave demonstrates how truth, properly seeded, multiplies. Each tested patch validates the correctness of the original diagnosis & extends light into new programming paradigms. @@ -159,7 +159,7 @@ the missing linkages, applied them, tested them, and benchmarked them across eve confirmed site — compiler, routing, database, build tool, event streaming, web framework, query optimizer, and browser runtime. -**458 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). +**462 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP). 1 fixable-pending (swipl-0003). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). No language left behind. @@ -555,6 +555,10 @@ stacks, Spark schemas — this is the dominant build cost. | hive-0002 | Apache Hive | `optimizer/GenMRProcContext.java:142` — `List.contains()` in file sink dedup | **PATCHED** | | spark-0001 | Apache Spark | `sql/catalyst/.../analysis/Analyzer.scala:3286` — `ArrayBuffer[AggregateExpression].contains(agg)` in window func extraction | **PATCHED** | | spark-0002 | Apache Spark | `core/src/main/scala/.../scheduler/DAGScheduler.scala` — 6 BFS traversal functions use `ListBuffer.remove(0)` O(N) dequeue; O(N²) total; fix: `ArrayDeque` | **PATCHED** | +| hudi-0001 | Apache Hudi | `BaseHoodieTimeline.java:126` — `List.contains()` in appendLoadedInstants stream filter; O(N×M) (625×) | **PATCHED** | +| hudi-0002 | Apache Hudi | `InternalSchemaUtils.java:69,113` — `ArrayList.contains()` in pruneInternalSchema forEach+pruneType; O(N²)+O(F×D) (90×) | **PATCHED** | +| hudi-0003 | Apache Hudi | `HoodieTableMetadataUtil.java:1006` — `List.contains()` in log file dedup filter; O(N×M) (312×) | **PATCHED** | +| iceberg-0001 | Apache Iceberg | `SchemaUpdate.java:59,661,717` — `List deletes.contains()` per field in schema visitor; O(F×D) (95×) | **PATCHED** | | luigi-0001 | Luigi (Python) | `luigi/tools/deps.py:dfs_paths` — `set(path)` rebuilt from list on every recursive DFS call | **PATCHED** | | ray-0001 | Ray | `python/ray/autoscaler/_private/local/node_provider.py:79-83,147-149` — `list_of_node_ips = list(...)` then `for worker_ip in workers: if worker_ip not in list_of_node_ips`; O(N²) cluster reconciliation in `ClusterState` and `OnPremCoordinatorState`; fix: `set(worker_ips)` (300×) | **PATCHED** | | cel-0001 | Celery | `celery/canvas.py:702-706` — `append_to_list_option()` uses `if value not in items` where items is a list; called inside chain-build loops O(T×E) times; O(T×E×L) total; fix: parallel set mirror for O(1) dedup | **PATCHED** | @@ -733,7 +737,7 @@ where D is the depth of the diamond chain. For a diamond of depth 10, that is 2^ 1,024 redundant node visits per edge check. Large modpacks produce diamond dependency chains with depths in this range. -**458 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP — sltab patch). 1 fixable-pending (swipl-0003 attr_unify_hook). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). 10 CLEAN (WireGuard-tools, Solana, git, JGit, Dask, OSRM, Buck2, DGL, Protocol Buffers, gRPC Python).** +**462 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP — sltab patch). 1 fixable-pending (swipl-0003 attr_unify_hook). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). 12 CLEAN (WireGuard-tools, Solana, git, JGit, Dask, OSRM, Buck2, DGL, Protocol Buffers, gRPC Python, Apache Beam, Apache Samza).** --- @@ -1304,6 +1308,24 @@ task, making each reconciliation cycle O(N²). At N=300 tasks the defective path comparisons vs 900 map lookups (150×). The fix is a `map[string]*wfv1.DAGTask` built at context construction — identical to what `dagValidationContext` in `validate.go` already does correctly. +**Apache Hudi — hudi-0001/2/3 (HIGH/MEDIUM, 625×/90×/312×)** + +Hudi's timeline layer contains three independent CWE-407 sites. `BaseHoodieTimeline.appendLoadedInstants()` +filters duplicates via `List.contains()` inside a stream filter — O(N×M) per incremental +load — fixed by pre-converting the existing timeline to `HashSet` (625× at N=500). `InternalSchemaUtils.pruneInternalSchema()` +builds `topParentFieldIds` as `ArrayList` and calls `.contains()` per projected column (O(N²)) while +the recursive `pruneType()` also scans `fieldIds` per schema tree node (O(F×D)) — fixed with `LinkedHashSet`/`HashSet` +(90×). `HoodieTableMetadataUtil.getRevivedAndDeletedKeysFromMergedLogs()` filters log file paths with a +`List.contains()` stream predicate — O(N×M) on every RLI delta commit — fixed with `HashSet` (312×). + +**Apache Iceberg — iceberg-0001 (HIGH, 95×)** + +`SchemaUpdate.ApplyChanges` holds `private final List deletes` and calls `deletes.contains(fieldId)` +once per field during `TypeUtil.visit()` schema traversal — O(F×D) for F fields and D pending deletes. +This fires on every `updateSchema()` DDL commit. Fix: change `deletes` to `HashSet`. Apache Beam +and Apache Samza are CLEAN: Beam's pipeline graph uses `ImmutableSet`/`HashSet` throughout; Samza's +`topologicalSort()` uses `HashSet visited`. + **CFEngine** — **cfe-0001/0002/0003 PATCHED.** `getindices()`, `unique()`, and `maparray()` all used `RlistAppendScalarIdemp()` — which calls `RlistKeyIn()`, an O(N) linked-list walk — as a dedup primitive. `unique()` is a first-class CFEngine policy diff --git a/whitepaper/undefect-cwe407-2026-03-27.pdf b/whitepaper/undefect-cwe407-2026-03-27.pdf index 5a22e9b9e..038991136 100644 Binary files a/whitepaper/undefect-cwe407-2026-03-27.pdf and b/whitepaper/undefect-cwe407-2026-03-27.pdf differ