whitepaper: re-add 10 missing entries + 11 new defects this session, count 578→590; rebuild PDF

defects/spark/patch/spark-0001-analyzer-seenwindowaggregates-linkedhashset.patch

@@ -1,3 +1,4 @@
+# UNDF: UNDF-2026-000000290
 diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

defects/spark/patch/spark-0003.patch

@@ -1,3 +1,4 @@
+# UNDF: UNDF-2026-000000291
 diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
 --- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
 +++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala

defects/spark/patch/spark-deeper-CLEAN.md

@@ -0,0 +1,50 @@
+# spark-deeper — Scheduler + Catalyst + Physical Plan + Storage + Shuffle CLEAN
+
+## Areas Scanned
+
+### Core Scheduler
+- `core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala` — BFS traversals (spark-0002), waitingStages filter (spark-0004), `submitWaitingChildStages`, `stageDependsOn`, `getMissingParentStages`, all 6 BFS methods — all already covered by existing defects
+- `core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala` — all `contains`/`exists` calls use `HashMap`/`mutable.Map` → O(1)
+- `core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala` — no List.contains in hot paths
+- `core/src/main/scala/org/apache/spark/scheduler/TaskSetExcludeList.scala` — `excludedExecs`, `excludedNodes` are `HashSet` → O(1)
+- `core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala` — clean
+- `core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala` — `queues.asScala.find(_.name == queue)` on bounded small list (typically 4 queues)
+
+### Catalyst Optimizer
+- `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala` — `excludeList` is `AttributeSet` (O(1)), `projectList.exists(...)` is O(E) but called once per plan node, not in a nested loop
+- `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala` — all `projectList.exists()` / `aggList.exists()` calls are O(E) single-pass pattern-match guards
+- `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala` — `assignments.exists(isEqual(_, fieldPath))` is O(A×F) in `filterSchema`, but this is a schema-time operation on small data
+- `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CapturesConfig.scala` — `configPrefixDenyList` has ~11 fixed entries, not a scaling concern
+
+### Physical Plan Operators
+- `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/` — all `contains`/`exists` calls use `HashSet`, `OpenHashSet`, `AttributeSet`, or `ExpressionSet` → O(1)
+- `sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala` — `childrenIndexes` is bounded (≤ 2 for joins)
+- `sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/` — clean
+- `sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/` — `modes.contains(Final)` etc. is on a small fixed-size list
+
+### Dynamic Partition Pruning (Spark SQL)
+- `sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala` — `joinKeys.indexOf(filteringKeys.head)` is O(K) single call, not in a loop
+
+### Storage and Shuffle
+- `core/src/main/scala/org/apache/spark/storage/` — no list.contains patterns in hot paths
+- `core/src/main/scala/org/apache/spark/shuffle/` — no list.contains patterns in hot paths
+
+### RDD Operations
+- `core/src/main/scala/org/apache/spark/rdd/RDD.scala` — `getNarrowAncestors` uses `mutable.HashSet` → O(1)
+- `core/src/main/scala/org/apache/spark/rdd/OrderedRDDFunctions.scala` — `partitionIndices.contains` uses Scala `Range.contains` → O(1)
+- `core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala` — all `contains` calls use `HashMap`-backed structures
+
+### MLlib (spot check)
+- `mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala` — `dotTerms.contains` is O(D) per filter call on plan-time formula parsing — negligible at runtime
+
+## Verdict: No new CWE-407 defects found beyond spark-0001 through spark-0004
+
+All identified `contains`/`exists`/`find` calls in hot paths either:
+1. Operate on hash-based structures (O(1) lookup), or
+2. Are called once per plan node / RDD construction (not in a doubly-nested loop), or
+3. Operate on bounded-size data (≤ join side count, ≤ mode count, etc.)
+
+The 6 BFS `ListBuffer.remove(0)` defects are already captured as spark-0002.
+The `submitWaitingChildStages` O(W×P) defect is already captured as spark-0004.
+
+## Scan date: 2026-03-27