engineering/java-topology
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

wave7: 433/194 — kafka/flink/pulsar, spring/micronaut/quarkus, nginx/haproxy/traefik, linux/nomad/consul, numpy/pandas/sklearn, ES/OS/pg/sqlite/rustc/cargo

Browse source
This commit is contained in:
russell@unturf.com 2026-03-27 16:20:58 -04:00
parent 3735145aa5
commit 5fe6da7cc2
69 changed files with 6793 additions and 32 deletions
Download patch file Download diff file Expand all files Collapse all files

46
defects/flink/patch/flink-0002-rowtypeutils-checklist-hashset.md Normal file
View file

@ -0,0 +1,46 @@
# flink-0002: RowTypeUtils.getUniqueName — List.contains() inside nested for+do-while
## Defect ID
flink-0002
## File:Line
`flink-table/flink-table-common/src/main/java/org/apache/flink/table/typeutils/RowTypeUtils.java:43,49`
## Description
`getUniqueName(List<String> oldNames, List<String> checklist)` does:
```java
for (String oldName : oldNames) { // O(N)
if (checklist.contains(oldName) || result.contains(oldName)) { // O(M) + O(R)
do {
changedName = oldName + "_" + suffix++;
} while (checklist.contains(changedName) || result.contains(changedName)); // O(M) + O(R) per iteration
result.add(changedName);
} else {
result.add(oldName);
}
}
```
Both `checklist` and `result` are `List<String>`. Each `.contains()` is O(M) or O(R).
The do-while iterates up to K times per name (until a unique suffix is found).
Total complexity: O(N × (M + R) × K) which approaches O(N × M²) in the worst case
when many names clash and checklist is large.
This method is called during SQL query planning whenever column names must be
deduplicated (e.g., join projections, window aggregations) — it runs on every query.
## Complexity
- Slow: O(N × M × K) — List.contains() = O(M)
- Fast: O(N × K) — HashSet.contains() = O(1)
## Severity
MEDIUM
## Speedup Estimate
~M× improvement; at M=100 column names per relation, ~100x.
## Fix
Convert `checklist` to `HashSet<String>` at call sites, or convert internally at
method entry. Build a `HashSet<String> seen` from `checklist` plus accumulate
`result` into a parallel `HashSet<String>` for O(1) membership checks.

40
defects/flink/patch/flink-0003-aggregatereducegroupingrule-newgroupinglist-hashset.md Normal file
View file

@ -0,0 +1,40 @@
# flink-0003: AggregateReduceGroupingRule — List<Integer>.contains() inside for loop
## Defect ID
flink-0003
## File:Line
`flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/AggregateReduceGroupingRule.java:88`
## Description
In `onMatch()`:
```java
List<Integer> newGroupingList = newGrouping.toList(); // List<Integer>
for (int column : originalGrouping) { // O(G) iterations
if (newGroupingList.contains(column)) { // O(G) linear scan each time
...
}
}
```
`newGroupingList` is a `List<Integer>`. Each `.contains()` call does a linear scan
O(G) where G = number of grouping columns. Total: O(G²).
This fires during query planning on every aggregate that has reducible grouping keys —
i.e., every GROUP BY with functional dependencies. In queries with wide GROUP BY
clauses (e.g., 50+ columns in analytics), this is O(2500) instead of O(50).
## Complexity
- Slow: O(G²) — List.contains() = O(G)
- Fast: O(G) — HashSet.contains() = O(1)
## Severity
MEDIUM
## Speedup Estimate
~G× improvement; at G=50 grouping columns, ~50x speedup in the planning phase.
## Fix
Replace `newGroupingList` with `Set<Integer> newGroupingSet = new HashSet<>(newGrouping.toList())`.
The list is only used for `.contains()` and `.size()` — both work identically on HashSet.

173
defects/flink/unit/FlinkAggregateGroupingRuleTest.java Normal file
View file

@ -0,0 +1,173 @@
package unit;
import java.util.*;
/**
* flink-0003: AggregateReduceGroupingRule — List<Integer>.contains() in for loop.
*
* newGroupingList is a List<Integer> from ImmutableBitSet.toList().
* Each .contains(column) is O(G). With G grouping columns, total is O(G^2).
* Fix: use Set<Integer> = new HashSet<>(newGrouping.toList()) for O(G).
*/
public class FlinkAggregateGroupingRuleTest {
// Slow path: List<Integer>.contains() — simulates AggregateReduceGroupingRule.onMatch()
// Returns (indexOldToNewMap, opCount)
static Object[] slowBuildIndexMap(List<Integer> originalGrouping, List<Integer> newGroupingList) {
Map<Integer, Integer> indexOldToNewMap = new HashMap<>();
int idxOfNewGrouping = 0;
int idxOfAggCallsForDropped = newGroupingList.size();
int index = 0;
long ops = 0;
for (int column : originalGrouping) {
// List.contains: linear scan
boolean found = false;
for (Integer g : newGroupingList) {
ops++;
if (g == column) { found = true; break; }
}
if (found) {
indexOldToNewMap.put(index, idxOfNewGrouping++);
} else {
indexOldToNewMap.put(index, idxOfAggCallsForDropped++);
}
index++;
}
return new Object[]{indexOldToNewMap, ops};
}
// Fast path: Set<Integer>.contains() — O(1) lookup
static Object[] fastBuildIndexMap(List<Integer> originalGrouping, Set<Integer> newGroupingSet, int newGroupingSize) {
Map<Integer, Integer> indexOldToNewMap = new HashMap<>();
int idxOfNewGrouping = 0;
int idxOfAggCallsForDropped = newGroupingSize;
int index = 0;
long ops = 0;
for (int column : originalGrouping) {
ops++; // O(1) HashSet lookup
if (newGroupingSet.contains(column)) {
indexOldToNewMap.put(index, idxOfNewGrouping++);
} else {
indexOldToNewMap.put(index, idxOfAggCallsForDropped++);
}
index++;
}
return new Object[]{indexOldToNewMap, ops};
}
public static void main(String[] args) {
int passed = 0;
int total = 0;
// Test 1: correctness — all columns retained
{
total++;
List<Integer> original = Arrays.asList(0, 1, 2, 3, 4);
List<Integer> newGrouping = Arrays.asList(0, 1, 2, 3, 4);
Set<Integer> newGroupingSet = new HashSet<>(newGrouping);
@SuppressWarnings("unchecked")
Map<Integer,Integer> slowMap = (Map<Integer,Integer>) slowBuildIndexMap(original, newGrouping)[0];
@SuppressWarnings("unchecked")
Map<Integer,Integer> fastMap = (Map<Integer,Integer>) fastBuildIndexMap(original, newGroupingSet, newGrouping.size())[0];
assert slowMap.equals(fastMap)
: "All retained: slow and fast must agree. slow=" + slowMap + " fast=" + fastMap;
// All mapped to 0..4
for (int i = 0; i < 5; i++) {
assert slowMap.get(i) == i : "column " + i + " should map to " + i;
}
System.out.println(" Test 1 PASS: all-retained case, map=" + slowMap);
passed++;
}
// Test 2: correctness — some columns dropped
{
total++;
List<Integer> original = Arrays.asList(0, 1, 2, 3, 4);
List<Integer> newGrouping = Arrays.asList(1, 3); // only cols 1 and 3 are unique
Set<Integer> newGroupingSet = new HashSet<>(newGrouping);
@SuppressWarnings("unchecked")
Map<Integer,Integer> slowMap = (Map<Integer,Integer>) slowBuildIndexMap(original, newGrouping)[0];
@SuppressWarnings("unchecked")
Map<Integer,Integer> fastMap = (Map<Integer,Integer>) fastBuildIndexMap(original, newGroupingSet, newGrouping.size())[0];
assert slowMap.equals(fastMap)
: "Partial drop: slow and fast must agree. slow=" + slowMap + " fast=" + fastMap;
// col 0 not in newGrouping → agg call index (starts at 2)
assert slowMap.get(0) == 2 : "col 0 not in newGrouping → idx 2, got " + slowMap.get(0);
assert slowMap.get(1) == 0 : "col 1 in newGrouping → idx 0, got " + slowMap.get(1);
assert slowMap.get(2) == 3 : "col 2 not in newGrouping → idx 3, got " + slowMap.get(2);
assert slowMap.get(3) == 1 : "col 3 in newGrouping → idx 1, got " + slowMap.get(3);
assert slowMap.get(4) == 4 : "col 4 not in newGrouping → idx 4, got " + slowMap.get(4);
System.out.println(" Test 2 PASS: partial-drop case, map=" + slowMap);
passed++;
}
// Test 3: op count — O(G^2) vs O(G)
{
total++;
int G = 60; // grouping columns
List<Integer> original = new ArrayList<>();
List<Integer> newGrouping = new ArrayList<>();
for (int i = 0; i < G; i++) {
original.add(i);
newGrouping.add(i);
}
Set<Integer> newGroupingSet = new HashSet<>(newGrouping);
long slowOps = (Long) slowBuildIndexMap(original, newGrouping)[1];
long fastOps = (Long) fastBuildIndexMap(original, newGroupingSet, G)[1];
// Slow: best case (all found on first scan) = G * 1 = G ops
// Worst case: not found = G * G ops
// In our worst-case-measuring sim above, found on first match (column == 0..G-1 sequential)
// so slowOps = sum(pos+1) for each element. Let's just verify slowOps > fastOps
// and fastOps == G.
assert fastOps == G : "Fast path must make exactly G=" + G + " ops, got " + fastOps;
assert slowOps >= G : "Slow path must make at least G ops";
long speedup = slowOps / fastOps;
System.out.println(" Test 3 PASS: slowOps=" + slowOps + " fastOps=" + fastOps + " speedup=" + speedup + "x");
passed++;
}
// Test 4: worst-case O(G^2) when target always at end of list
{
total++;
int G = 50;
// original grouping has elements NOT in newGrouping → every contains() scans all G
List<Integer> original = new ArrayList<>();
List<Integer> newGrouping = new ArrayList<>();
// newGrouping = [0..G-1], original = [G..2G-1] (no overlap → all scan full list)
for (int i = 0; i < G; i++) newGrouping.add(i);
for (int i = G; i < 2 * G; i++) original.add(i);
Set<Integer> newGroupingSet = new HashSet<>(newGrouping);
// slow: each original element scans all G elements of newGrouping (no match)
long expectedSlowOps = (long) G * G; // G * G scans
long actualSlowOps = 0;
for (int col : original) {
for (int g : newGrouping) {
actualSlowOps++;
if (g == col) break; // never matches
}
}
assert actualSlowOps == expectedSlowOps
: "Expected " + expectedSlowOps + " slow ops, got " + actualSlowOps;
long fastOps = G; // G × O(1)
long speedup = actualSlowOps / fastOps;
assert speedup == G : "Speedup should equal G=" + G + " in worst case, got " + speedup;
System.out.println(" Test 4 PASS: O(G^2)=" + actualSlowOps + " vs O(G)=" + fastOps + " speedup=" + speedup + "x (G=" + G + ")");
passed++;
}
System.out.println(passed + "/" + total + " PASS");
if (passed != total) System.exit(1);
}
}

192
defects/flink/unit/FlinkRowTypeUtilsTest.java Normal file
View file

@ -0,0 +1,192 @@
package unit;
import java.util.*;
/**
* flink-0002: RowTypeUtils.getUniqueName — List.contains() in for + do-while nested loop.
*
* Both checklist and result are List<String>. Each .contains() is O(M).
* With N names and M checklist entries, worst-case is O(N * M^2) on collisions.
* Fix: convert checklist and result-so-far to HashSet for O(1) membership.
*/
public class FlinkRowTypeUtilsTest {
// ---- Slow path: exact replica of RowTypeUtils.getUniqueName logic ----
// Returns (result, opCount) — opCount tracks every element comparison in contains()
static Object[] slowGetUniqueName(List<String> oldNames, List<String> checklist) {
List<String> result = new ArrayList<>();
long ops = 0;
for (String oldName : oldNames) {
// checklist.contains(oldName) — scan checklist
boolean inChecklist = false;
for (String s : checklist) { ops++; if (s.equals(oldName)) { inChecklist = true; break; } }
// result.contains(oldName) — scan result (only if not already found)
boolean inResult = false;
if (!inChecklist) {
for (String s : result) { ops++; if (s.equals(oldName)) { inResult = true; break; } }
}
if (inChecklist || inResult) {
int suffix = -1;
String changedName;
do {
suffix++;
changedName = oldName + "_" + suffix;
// checklist.contains(changedName)
boolean chkContains = false;
for (String s : checklist) { ops++; if (s.equals(changedName)) { chkContains = true; break; } }
// result.contains(changedName)
boolean resContains = false;
if (!chkContains) {
for (String s : result) { ops++; if (s.equals(changedName)) { resContains = true; break; } }
}
if (!chkContains && !resContains) break;
} while (true);
result.add(changedName);
} else {
result.add(oldName);
}
}
return new Object[]{result, ops};
}
// ---- Fast path: HashSet for both checklist and result tracking ----
@SuppressWarnings("unchecked")
static Object[] fastGetUniqueName(List<String> oldNames, List<String> checklist) {
Set<String> checkSet = new HashSet<>(checklist);
List<String> result = new ArrayList<>();
Set<String> resultSet = new HashSet<>();
long ops = 0;
for (String oldName : oldNames) {
ops++; // one HashSet.contains for checklist
ops++; // one HashSet.contains for resultSet
if (checkSet.contains(oldName) || resultSet.contains(oldName)) {
int suffix = -1;
String changedName;
do {
suffix++;
changedName = oldName + "_" + suffix;
ops++; // checkSet
ops++; // resultSet
} while (checkSet.contains(changedName) || resultSet.contains(changedName));
result.add(changedName);
resultSet.add(changedName);
} else {
result.add(oldName);
resultSet.add(oldName);
}
}
return new Object[]{result, ops};
}
public static void main(String[] args) {
int passed = 0;
int total = 0;
// Test 1: correctness — no collisions
{
total++;
List<String> oldNames = Arrays.asList("a", "b", "c");
List<String> checklist = Arrays.asList("x", "y", "z");
@SuppressWarnings("unchecked")
List<String> slowResult = (List<String>) slowGetUniqueName(oldNames, checklist)[0];
@SuppressWarnings("unchecked")
List<String> fastResult = (List<String>) fastGetUniqueName(oldNames, checklist)[0];
assert slowResult.equals(fastResult)
: "No collision: expected same result. slow=" + slowResult + " fast=" + fastResult;
assert slowResult.equals(Arrays.asList("a", "b", "c"))
: "No collision: result should be unchanged. got=" + slowResult;
System.out.println(" Test 1 PASS: no-collision case, result=" + slowResult);
passed++;
}
// Test 2: correctness — with collisions
{
total++;
List<String> oldNames = Arrays.asList("a", "b", "a"); // duplicate "a"
List<String> checklist = Arrays.asList("b", "c");
@SuppressWarnings("unchecked")
List<String> slowResult = (List<String>) slowGetUniqueName(oldNames, checklist)[0];
@SuppressWarnings("unchecked")
List<String> fastResult = (List<String>) fastGetUniqueName(oldNames, checklist)[0];
assert slowResult.equals(fastResult)
: "Collision: slow and fast must agree. slow=" + slowResult + " fast=" + fastResult;
// "b" clashes with checklist → b_0
// second "a" clashes with result "a" → a_0
assert slowResult.get(0).equals("a") : "First 'a' is unchanged";
assert slowResult.get(1).equals("b_0") : "'b' collides with checklist → b_0, got=" + slowResult.get(1);
assert slowResult.get(2).equals("a_0") : "Second 'a' collides with result → a_0, got=" + slowResult.get(2);
System.out.println(" Test 2 PASS: collision case, result=" + slowResult);
passed++;
}
// Test 3: op count — slow >> fast with large checklist
{
total++;
int M = 100; // checklist size
int N = 50; // names
List<String> checklist = new ArrayList<>();
for (int i = 0; i < M; i++) checklist.add("existing-" + i);
// All names collide with checklist (worst case: every name needs renaming)
List<String> oldNames = new ArrayList<>();
for (int i = 0; i < N; i++) oldNames.add("existing-" + i);
Object[] slowOut = slowGetUniqueName(oldNames, checklist);
Object[] fastOut = fastGetUniqueName(oldNames, checklist);
@SuppressWarnings("unchecked")
List<String> slowResult = (List<String>) slowOut[0];
@SuppressWarnings("unchecked")
List<String> fastResult = (List<String>) fastOut[0];
long slowOps = (Long) slowOut[1];
long fastOps = (Long) fastOut[1];
assert slowResult.equals(fastResult)
: "Large test: slow and fast must agree";
assert slowOps > fastOps
: "Slow path must do more ops than fast: slowOps=" + slowOps + " fastOps=" + fastOps;
long speedup = slowOps / Math.max(fastOps, 1);
System.out.println(" Test 3 PASS: slowOps=" + slowOps + " fastOps=" + fastOps + " speedup=" + speedup + "x");
passed++;
}
// Test 4: pure complexity: O(N*M) slow vs O(N) fast with no collisions
{
total++;
int N = 80, M = 80;
List<String> checklist = new ArrayList<>();
for (int i = 0; i < M; i++) checklist.add("check-" + i);
Set<String> checkSet = new HashSet<>(checklist);
// names don't collide — just checklist.contains() called N times
List<String> names = new ArrayList<>();
for (int i = M; i < M + N; i++) names.add("name-" + i);
long slowOps = 0;
for (String name : names) {
// List.contains scans all M elements (name not in checklist)
slowOps += M;
}
long fastOps = N; // HashSet: N × O(1)
assert slowOps == (long) N * M : "slowOps should be N*M=" + (long)N*M;
assert fastOps == N : "fastOps should be N=" + N;
long speedup = slowOps / fastOps;
assert speedup == M : "speedup should == M=" + M + ", got " + speedup;
System.out.println(" Test 4 PASS: O(N*M)=" + slowOps + " vs O(N)=" + fastOps + " speedup=" + speedup + "x");
passed++;
}
System.out.println(passed + "/" + total + " PASS");
if (passed != total) System.exit(1);
}
}