wave9 complete: 458/207 — airflow/argo/pytorch-0003/tf/jax/pyg/grpc/thrift + graphhopper/valhalla
This commit is contained in:
parent
81bc62b9cb
commit
ab10b2f555
22 changed files with 2889 additions and 16 deletions
131
defects/airflow/patch/airflow-0001-topological-sort-quadratic.md
Normal file
131
defects/airflow/patch/airflow-0001-topological-sort-quadratic.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# airflow-0001 — O(N²) Topological Sort in TaskGroup
|
||||
|
||||
**Severity:** HIGH
|
||||
**Complexity:** O(N²) → O(N)
|
||||
**CWE:** CWE-407 (Algorithmic Complexity)
|
||||
|
||||
## Affected Files
|
||||
|
||||
| File | Lines | Notes |
|
||||
|------|-------|-------|
|
||||
| `task-sdk/src/airflow/sdk/definitions/taskgroup.py` | 536–568 | Runtime task group sort |
|
||||
| `airflow-core/src/airflow/serialization/definitions/taskgroup.py` | 230–248 | Serialized task group sort |
|
||||
|
||||
## Defective Code
|
||||
|
||||
### `task-sdk/src/airflow/sdk/definitions/taskgroup.py` lines 536–568
|
||||
|
||||
```python
|
||||
while graph_unsorted:
|
||||
acyclic = False
|
||||
for node in list(graph_unsorted.values()): # O(N) scan every round
|
||||
for edge in node.upstream_list:
|
||||
if edge.node_id in graph_unsorted:
|
||||
break
|
||||
tg = edge.task_group
|
||||
while tg:
|
||||
if tg.node_id in graph_unsorted:
|
||||
break
|
||||
tg = tg.parent_group
|
||||
if tg:
|
||||
break
|
||||
else:
|
||||
acyclic = True
|
||||
del graph_unsorted[node.node_id]
|
||||
graph_sorted.append(node)
|
||||
|
||||
if not acyclic:
|
||||
raise AirflowDagCycleException(...)
|
||||
```
|
||||
|
||||
**Why it is O(N²):** On each pass of the outer `while graph_unsorted:` loop the inner
|
||||
`for node in list(graph_unsorted.values())` rescans every remaining unsorted node from
|
||||
scratch. In the worst case (a linear chain of N tasks) only one node is resolved per
|
||||
round, so the total work is N + (N−1) + … + 1 = O(N²/2).
|
||||
|
||||
Same pattern is copy-pasted verbatim into the serialization layer
|
||||
(`airflow-core/src/airflow/serialization/definitions/taskgroup.py:230–248`).
|
||||
|
||||
## Fixed Code
|
||||
|
||||
Standard Kahn's algorithm: pre-compute in-degrees, maintain a ready queue, process
|
||||
each node exactly once — O(N + E).
|
||||
|
||||
```python
|
||||
def topological_sort(self):
|
||||
"""
|
||||
Sorts children in topographical order, such that a task comes after any of
|
||||
its upstream dependencies.
|
||||
|
||||
Uses Kahn's algorithm with an explicit ready-queue: O(N + E).
|
||||
"""
|
||||
from collections import deque
|
||||
|
||||
graph_unsorted = dict(self.children)
|
||||
graph_sorted: list[DAGNode] = []
|
||||
|
||||
if not graph_unsorted:
|
||||
return graph_sorted
|
||||
|
||||
# Build in-degree counts and adjacency (upstream → dependents).
|
||||
in_degree: dict[str, int] = {node_id: 0 for node_id in graph_unsorted}
|
||||
dependents: dict[str, list[str]] = {node_id: [] for node_id in graph_unsorted}
|
||||
|
||||
for node_id, node in graph_unsorted.items():
|
||||
for edge in node.upstream_list:
|
||||
upstream_id = edge.node_id
|
||||
if upstream_id in graph_unsorted:
|
||||
in_degree[node_id] += 1
|
||||
dependents[upstream_id].append(node_id)
|
||||
else:
|
||||
# Check if task's group is a child of this TG
|
||||
tg = edge.task_group
|
||||
while tg:
|
||||
if tg.node_id in graph_unsorted:
|
||||
in_degree[node_id] += 1
|
||||
dependents[tg.node_id].append(node_id)
|
||||
break
|
||||
tg = tg.parent_group
|
||||
|
||||
ready: deque[str] = deque(
|
||||
node_id for node_id, deg in in_degree.items() if deg == 0
|
||||
)
|
||||
|
||||
while ready:
|
||||
node_id = ready.popleft()
|
||||
graph_sorted.append(graph_unsorted[node_id])
|
||||
for dep_id in dependents[node_id]:
|
||||
in_degree[dep_id] -= 1
|
||||
if in_degree[dep_id] == 0:
|
||||
ready.append(dep_id)
|
||||
|
||||
if len(graph_sorted) != len(graph_unsorted):
|
||||
raise AirflowDagCycleException(
|
||||
f"A cyclic dependency occurred in dag: {self.dag_id}"
|
||||
)
|
||||
|
||||
return graph_sorted
|
||||
```
|
||||
|
||||
Apply the same fix to the copy in
|
||||
`airflow-core/src/airflow/serialization/definitions/taskgroup.py`.
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Version | Time | Space |
|
||||
|---------|------|-------|
|
||||
| Defective | O(N²) | O(N) |
|
||||
| Fixed | O(N + E) | O(N + E) |
|
||||
|
||||
For a DAG with N=1000 tasks and E=2000 edges, the defective version performs
|
||||
~500,000 operations per sort; the fixed version performs ~3,000.
|
||||
|
||||
## Trigger Path
|
||||
|
||||
`topological_sort()` is called:
|
||||
- `airflow-core/.../routes/ui/structure.py:93` on every UI structure render
|
||||
- `airflow-core/.../services/ui/task_group.py:36` via `task_group_to_dict`
|
||||
- Indirectly by `DAG.topological_sort()` which delegates to this method
|
||||
|
||||
Production Airflow deployments with large DAGs (hundreds of tasks) will see
|
||||
O(N²) CPU consumption on every API call that renders DAG structure.
|
||||
268
defects/airflow/unit/AirflowTopoSortAlgorithm.java
Normal file
268
defects/airflow/unit/AirflowTopoSortAlgorithm.java
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* airflow-0001 — O(N²) topological sort in TaskGroup.topological_sort()
|
||||
*
|
||||
* Simulates the defective "modified Kahn's" algorithm from:
|
||||
* task-sdk/src/airflow/sdk/definitions/taskgroup.py:536-568
|
||||
*
|
||||
* Defect: Each round of the outer while loop rescans ALL remaining unsorted nodes
|
||||
* (inner for loop), doing O(N) work per round. Worst case (linear chain): O(N²).
|
||||
*
|
||||
* Fix: Pre-compute in-degrees and maintain a ready-queue (true Kahn's), O(N + E).
|
||||
*
|
||||
* The op-count measures "node examination" — how many times a node is inspected
|
||||
* in the scan loop. In the defective version every remaining node is examined
|
||||
* each round; in the fixed version each node is processed exactly once.
|
||||
*/
|
||||
public class AirflowTopoSortAlgorithm {
|
||||
|
||||
static void check(String desc, boolean cond) {
|
||||
System.out.println((cond ? "PASS" : "FAIL") + ": " + desc);
|
||||
if (!cond) throw new AssertionError("FAIL: " + desc);
|
||||
}
|
||||
|
||||
// Node: simulates TaskGroup child with upstream_list
|
||||
static class Node {
|
||||
final String id;
|
||||
final List<String> upstreamIds = new ArrayList<>();
|
||||
|
||||
Node(String id) { this.id = id; }
|
||||
void addUpstream(String uid) { upstreamIds.add(uid); }
|
||||
}
|
||||
|
||||
// Result carrying an op-count alongside the sorted list
|
||||
static class SortResult {
|
||||
final List<String> sorted;
|
||||
final long nodeExaminations; // how many times a node was pulled from the scan
|
||||
|
||||
SortResult(List<String> sorted, long nodeExaminations) {
|
||||
this.sorted = sorted;
|
||||
this.nodeExaminations = nodeExaminations;
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// DEFECTIVE: O(N²) "modified Kahn's" — direct port of Airflow Python
|
||||
// graph_unsorted is Map<id, Node>, mimics Python dict
|
||||
// Each pass through while scans ALL remaining nodes
|
||||
// -----------------------------------------------------------------------
|
||||
static SortResult defectiveTopoSort(Map<String, Node> nodes) {
|
||||
Map<String, Node> graphUnsorted = new LinkedHashMap<>(nodes);
|
||||
List<String> graphSorted = new ArrayList<>();
|
||||
long examinations = 0;
|
||||
|
||||
while (!graphUnsorted.isEmpty()) {
|
||||
boolean acyclic = false;
|
||||
for (Node node : new ArrayList<>(graphUnsorted.values())) {
|
||||
examinations++; // Each node examined once per pass
|
||||
boolean blocked = false;
|
||||
for (String upId : node.upstreamIds) {
|
||||
if (graphUnsorted.containsKey(upId)) {
|
||||
blocked = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!blocked) {
|
||||
acyclic = true;
|
||||
graphUnsorted.remove(node.id);
|
||||
graphSorted.add(node.id);
|
||||
}
|
||||
}
|
||||
if (!acyclic) throw new RuntimeException("Cycle detected");
|
||||
}
|
||||
return new SortResult(graphSorted, examinations);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FIXED: True Kahn's with ready-queue — O(N + E)
|
||||
// Each node is dequeued exactly once → examinations == N
|
||||
// -----------------------------------------------------------------------
|
||||
static SortResult fixedTopoSort(Map<String, Node> nodes) {
|
||||
Map<String, Integer> inDegree = new HashMap<>();
|
||||
Map<String, List<String>> dependents = new HashMap<>();
|
||||
|
||||
for (String id : nodes.keySet()) {
|
||||
inDegree.put(id, 0);
|
||||
dependents.put(id, new ArrayList<>());
|
||||
}
|
||||
|
||||
for (Node node : nodes.values()) {
|
||||
for (String upId : node.upstreamIds) {
|
||||
if (nodes.containsKey(upId)) {
|
||||
inDegree.merge(node.id, 1, Integer::sum);
|
||||
dependents.get(upId).add(node.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Deque<String> ready = new ArrayDeque<>();
|
||||
for (Map.Entry<String, Integer> e : inDegree.entrySet()) {
|
||||
if (e.getValue() == 0) ready.add(e.getKey());
|
||||
}
|
||||
|
||||
List<String> result = new ArrayList<>();
|
||||
long examinations = 0;
|
||||
while (!ready.isEmpty()) {
|
||||
String id = ready.poll();
|
||||
examinations++; // Each node dequeued exactly once
|
||||
result.add(id);
|
||||
for (String dep : dependents.get(id)) {
|
||||
inDegree.merge(dep, -1, Integer::sum);
|
||||
if (inDegree.get(dep) == 0) ready.add(dep);
|
||||
}
|
||||
}
|
||||
|
||||
if (result.size() != nodes.size()) throw new RuntimeException("Cycle detected");
|
||||
return new SortResult(result, examinations);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Build a reversed linear chain: nodes inserted in dependency-last order.
|
||||
//
|
||||
// Insertion order: t(N-1), t(N-2), ..., t1, t0
|
||||
// Dependency: t(k) depends on t(k-1)
|
||||
//
|
||||
// When graph_unsorted preserves insertion order (LinkedHashMap, like Python dict),
|
||||
// each round of the while loop scans the full snapshot but can only free the LAST
|
||||
// node (t0) on the first pass, then (t1) on the second, etc.
|
||||
//
|
||||
// Round 1: examine all N nodes, free t0 (it has no upstreams)
|
||||
// Round 2: examine remaining N-1 nodes, free t1 (t0 now gone)
|
||||
// ...
|
||||
// Total: N + (N-1) + ... + 1 = N(N+1)/2 = O(N²) node examinations
|
||||
// -----------------------------------------------------------------------
|
||||
static Map<String, Node> buildLinearChain(int n) {
|
||||
Map<String, Node> nodes = new LinkedHashMap<>();
|
||||
// Insert in REVERSE order (t(N-1) first) — worst case for the defective algo
|
||||
for (int i = n - 1; i >= 0; i--) {
|
||||
nodes.put("t" + i, new Node("t" + i));
|
||||
}
|
||||
// t(k) depends on t(k-1): dependency is at the END of the insertion order
|
||||
for (int i = 1; i < n; i++) {
|
||||
nodes.get("t" + i).addUpstream("t" + (i - 1));
|
||||
}
|
||||
return nodes;
|
||||
}
|
||||
|
||||
// Build a wide DAG: single source, all others depend on it
|
||||
static Map<String, Node> buildWideDag(int n) {
|
||||
Map<String, Node> nodes = new LinkedHashMap<>();
|
||||
nodes.put("root", new Node("root"));
|
||||
for (int i = 1; i < n; i++) {
|
||||
Node node = new Node("t" + i);
|
||||
node.addUpstream("root");
|
||||
nodes.put("t" + i, node);
|
||||
}
|
||||
return nodes;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
int passed = 0;
|
||||
int total = 0;
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 1: Correctness on small chain (3 nodes)
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
Map<String, Node> dag = buildLinearChain(3);
|
||||
List<String> defResult = defectiveTopoSort(new LinkedHashMap<>(dag)).sorted;
|
||||
List<String> fixResult = fixedTopoSort(new LinkedHashMap<>(dag)).sorted;
|
||||
// Both must produce a valid topological order: t0 before t1 before t2
|
||||
boolean defOk = defResult.indexOf("t0") < defResult.indexOf("t1")
|
||||
&& defResult.indexOf("t1") < defResult.indexOf("t2");
|
||||
boolean fixOk = fixResult.indexOf("t0") < fixResult.indexOf("t1")
|
||||
&& fixResult.indexOf("t1") < fixResult.indexOf("t2");
|
||||
check("Correctness: defective produces valid topo order for chain-3", defOk);
|
||||
check("Correctness: fixed produces valid topo order for chain-3", fixOk);
|
||||
passed += 2;
|
||||
total++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 2: Correctness on wide dag
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
Map<String, Node> dag = buildWideDag(11);
|
||||
List<String> defResult = defectiveTopoSort(new LinkedHashMap<>(dag)).sorted;
|
||||
List<String> fixResult = fixedTopoSort(new LinkedHashMap<>(dag)).sorted;
|
||||
boolean rootFirst = defResult.get(0).equals("root") && fixResult.get(0).equals("root");
|
||||
check("Correctness: wide dag — root is first in both", rootFirst);
|
||||
passed++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 3: Defective shows O(N²) node examinations on linear chain
|
||||
// For N=200 chain: expected = 200 + 199 + ... + 1 = 200*201/2 = 20100
|
||||
// Fixed: expected = N = 200
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
int n = 200;
|
||||
long expected = (long) n * (n + 1) / 2;
|
||||
SortResult defR = defectiveTopoSort(new LinkedHashMap<>(buildLinearChain(n)));
|
||||
SortResult fixR = fixedTopoSort(new LinkedHashMap<>(buildLinearChain(n)));
|
||||
|
||||
System.out.printf(" N=%d chain: defective=%d exams (expected %d), fixed=%d exams (expected %d)%n",
|
||||
n, defR.nodeExaminations, expected, fixR.nodeExaminations, n);
|
||||
|
||||
check("Defective: N=200 chain examinations == N(N+1)/2=" + expected,
|
||||
defR.nodeExaminations == expected);
|
||||
check("Fixed: N=200 chain examinations == N=" + n,
|
||||
fixR.nodeExaminations == n);
|
||||
passed += 2;
|
||||
total++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 4: Ratio confirms O(N²) vs O(N)
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
int n = 500;
|
||||
SortResult defR = defectiveTopoSort(new LinkedHashMap<>(buildLinearChain(n)));
|
||||
SortResult fixR = fixedTopoSort(new LinkedHashMap<>(buildLinearChain(n)));
|
||||
|
||||
double ratio = (double) defR.nodeExaminations / fixR.nodeExaminations;
|
||||
System.out.printf(" N=%d chain: ratio = %.1fx%n", n, ratio);
|
||||
// For N=500: defective = 500*501/2 = 125250, fixed = 500 → ratio = 250.5
|
||||
check("Op-count ratio >= 100x for linear chain N=" + n, ratio >= 100.0);
|
||||
passed++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 5: Quadratic growth — defective at N=200 should be ~4x N=100
|
||||
// N=100: 100*101/2 = 5050; N=200: 200*201/2 = 20100; ratio = 20100/5050 = 3.98
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
long ops100 = defectiveTopoSort(new LinkedHashMap<>(buildLinearChain(100))).nodeExaminations;
|
||||
long ops200 = defectiveTopoSort(new LinkedHashMap<>(buildLinearChain(200))).nodeExaminations;
|
||||
double growthRatio = (double) ops200 / ops100;
|
||||
System.out.printf(" Defective growth ratio (N=200 vs N=100): %.2fx (expected ~3.98x for O(N²))%n",
|
||||
growthRatio);
|
||||
check("Defective shows quadratic growth (ratio >= 3.5x)", growthRatio >= 3.5);
|
||||
passed++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 6: Linear growth — fixed at N=200 should be ~2x N=100
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
long ops100 = fixedTopoSort(new LinkedHashMap<>(buildLinearChain(100))).nodeExaminations;
|
||||
long ops200 = fixedTopoSort(new LinkedHashMap<>(buildLinearChain(200))).nodeExaminations;
|
||||
double growthRatio = (double) ops200 / ops100;
|
||||
System.out.printf(" Fixed growth ratio (N=200 vs N=100): %.2fx (expected ~2x for O(N))%n",
|
||||
growthRatio);
|
||||
check("Fixed shows linear growth (ratio < 3.0x)", growthRatio < 3.0);
|
||||
passed++;
|
||||
}
|
||||
|
||||
System.out.printf("%n%d/%d PASS%n", passed, total);
|
||||
}
|
||||
}
|
||||
130
defects/argo-workflows/patch/argo-0001-get-task-linear-scan.md
Normal file
130
defects/argo-workflows/patch/argo-0001-get-task-linear-scan.md
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
# argo-0001 — O(N) Linear Scan in GetTask() Called Inside DAG Execution Loop
|
||||
|
||||
**Severity:** HIGH
|
||||
**Complexity:** O(N·D) → O(D) per execution cycle, overall O(N²) → O(N)
|
||||
**CWE:** CWE-407 (Algorithmic Complexity)
|
||||
|
||||
## Affected File
|
||||
|
||||
`workflow/controller/dag.go` lines 83–89
|
||||
|
||||
## Defective Code
|
||||
|
||||
```go
|
||||
// dagContext holds context information about this context's DAG
|
||||
type dagContext struct {
|
||||
// tasks are all the tasks in the template
|
||||
tasks []wfv1.DAGTask // <-- stored as a SLICE
|
||||
...
|
||||
}
|
||||
|
||||
// GetTask returns the DAGTask for the given taskName.
|
||||
// Called from: resolveDependencies, assessDAGPhase, executeDAG task loop,
|
||||
// ancestry.go GetTaskDependencies, validate.go
|
||||
func (d *dagContext) GetTask(ctx context.Context, taskName string) *wfv1.DAGTask {
|
||||
for _, task := range d.tasks { // O(N) linear scan
|
||||
if task.Name == taskName {
|
||||
return &task
|
||||
}
|
||||
}
|
||||
panic("target " + taskName + " does not exist")
|
||||
}
|
||||
```
|
||||
|
||||
**Why it is O(N²):** `GetTask()` is called multiple times per task during
|
||||
`executeDAG()`, which itself iterates over all tasks (or target tasks). The
|
||||
outer loop runs once per task being executed. Each call to `GetTask()` scans
|
||||
the entire `tasks` slice. For a DAG with N tasks, this yields O(N²) total
|
||||
comparisons per reconciliation cycle.
|
||||
|
||||
### Call graph (defective paths)
|
||||
|
||||
```
|
||||
executeDAG()
|
||||
└── for _, taskName := range targetTasks [O(N) tasks]
|
||||
├── GetTask(taskName) [O(N) each]
|
||||
├── GetTask(taskName).Hooks → GetTask(taskName) [O(N) each]
|
||||
└── GetTask(taskName).GetExitHook(...) [O(N) each]
|
||||
|
||||
assessDAGPhase()
|
||||
└── for _, depName := range targetTasks [O(N)]
|
||||
└── GetTask(depName) [O(N) each]
|
||||
|
||||
ancestry.go:GetTaskAncestry()
|
||||
└── recursive DFS over all ancestors
|
||||
└── GetTaskDependencies → resolveDependencies
|
||||
└── GetTask(taskName) [O(N) each]
|
||||
|
||||
ancestry.go:getTaskDependsLogic()
|
||||
└── for _, dependency := range dagTask.Dependencies [O(D)]
|
||||
└── GetTask(dependency) [O(N) each]
|
||||
```
|
||||
|
||||
Note: `dagValidationContext.GetTask()` in `validate.go:1315` correctly uses a
|
||||
`map[string]wfv1.DAGTask` for O(1) lookup. The runtime `dagContext` uses a
|
||||
slice — this discrepancy is the defect.
|
||||
|
||||
## Fixed Code
|
||||
|
||||
### Step 1: Add a task lookup map to `dagContext`
|
||||
|
||||
```go
|
||||
type dagContext struct {
|
||||
boundaryName string
|
||||
boundaryID string
|
||||
tasks []wfv1.DAGTask
|
||||
taskByName map[string]*wfv1.DAGTask // <-- add this field
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Populate the map in `executeDAG` where dagContext is constructed
|
||||
|
||||
```go
|
||||
dagCtx := &dagContext{
|
||||
boundaryName: nodeName,
|
||||
boundaryID: node.ID,
|
||||
tasks: tmpl.DAG.Tasks,
|
||||
visited: make(map[string]bool),
|
||||
tmpl: tmpl,
|
||||
wf: woc.wf,
|
||||
tmplCtx: tmplCtx,
|
||||
dependencies: make(map[string][]string),
|
||||
dependsLogic: make(map[string]string),
|
||||
log: woc.log,
|
||||
}
|
||||
// Build O(1) task lookup map
|
||||
dagCtx.taskByName = make(map[string]*wfv1.DAGTask, len(tmpl.DAG.Tasks))
|
||||
for i := range dagCtx.tasks {
|
||||
dagCtx.taskByName[dagCtx.tasks[i].Name] = &dagCtx.tasks[i]
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Replace GetTask body
|
||||
|
||||
```go
|
||||
func (d *dagContext) GetTask(ctx context.Context, taskName string) *wfv1.DAGTask {
|
||||
if task, ok := d.taskByName[taskName]; ok {
|
||||
return task
|
||||
}
|
||||
panic("target " + taskName + " does not exist")
|
||||
}
|
||||
```
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Version | GetTask | Per executeDAG cycle | N=500 tasks |
|
||||
|---------|---------|---------------------|-------------|
|
||||
| Defective | O(N) | O(N²) | ~125,000 compares |
|
||||
| Fixed | O(1) | O(N) | ~500 lookups |
|
||||
|
||||
For a DAG with 500 tasks, the fixed version performs 250× fewer operations per
|
||||
reconciliation cycle.
|
||||
|
||||
## Trigger Path
|
||||
|
||||
Every time Argo's workflow controller reconciles a running DAG workflow, it
|
||||
calls `executeDAG()` → `executeDAGTask()` for each pending task. With N=500 tasks
|
||||
and 3 `GetTask()` calls per task in the hot path, each reconciliation does 750×500=375,000
|
||||
string comparisons instead of 750 map lookups. At 1 reconcile/second for a
|
||||
long-running workflow, this adds seconds of unnecessary CPU per minute.
|
||||
245
defects/argo-workflows/unit/ArgoGetTaskAlgorithm.java
Normal file
245
defects/argo-workflows/unit/ArgoGetTaskAlgorithm.java
Normal file
|
|
@ -0,0 +1,245 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* argo-0001 — O(N) linear GetTask() scan inside O(N) DAG execution loop
|
||||
*
|
||||
* Simulates the defective GetTask() from:
|
||||
* workflow/controller/dag.go:83-89
|
||||
*
|
||||
* Defect: dagContext stores tasks as []DAGTask (slice). GetTask(name) does
|
||||
* linear scan. Called O(N) times per executeDAG() cycle → O(N²) total.
|
||||
*
|
||||
* Fix: Build a map[string]*DAGTask at dagContext construction time → O(1) lookup.
|
||||
*
|
||||
* The unit test simulates:
|
||||
* 1. A DAGContext with N tasks (stored as list in defective, map in fixed)
|
||||
* 2. An executeDAG loop that calls GetTask() once per targetTask
|
||||
* 3. Counts total comparisons to confirm O(N²) vs O(N)
|
||||
*/
|
||||
public class ArgoGetTaskAlgorithm {
|
||||
|
||||
static int opCount = 0;
|
||||
|
||||
static void check(String desc, boolean cond) {
|
||||
System.out.println((cond ? "PASS" : "FAIL") + ": " + desc);
|
||||
if (!cond) throw new AssertionError("FAIL: " + desc);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Simulated DAGTask
|
||||
// -----------------------------------------------------------------------
|
||||
static class DAGTask {
|
||||
final String name;
|
||||
final List<String> dependencies;
|
||||
|
||||
DAGTask(String name, List<String> deps) {
|
||||
this.name = name;
|
||||
this.dependencies = deps;
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// DEFECTIVE: tasks stored as List<DAGTask>, GetTask does linear scan
|
||||
// -----------------------------------------------------------------------
|
||||
static class DefectiveDagContext {
|
||||
final List<DAGTask> tasks;
|
||||
|
||||
DefectiveDagContext(List<DAGTask> tasks) {
|
||||
this.tasks = new ArrayList<>(tasks);
|
||||
}
|
||||
|
||||
DAGTask getTask(String taskName) {
|
||||
for (DAGTask task : tasks) { // O(N) scan
|
||||
opCount++;
|
||||
if (task.name.equals(taskName)) {
|
||||
return task;
|
||||
}
|
||||
}
|
||||
throw new RuntimeException("task not found: " + taskName);
|
||||
}
|
||||
|
||||
// Simulates executeDAG: for each target task, call getTask() 3 times
|
||||
// (matches the 3 GetTask calls on lines 292, 300, 307 in dag.go)
|
||||
void executeDAG(List<String> targetTasks) {
|
||||
for (String taskName : targetTasks) {
|
||||
getTask(taskName); // line 292: task := dagCtx.GetTask(ctx, taskName)
|
||||
getTask(taskName); // line 300: dagCtx.GetTask(ctx, taskName).Hooks
|
||||
getTask(taskName); // line 307: dagCtx.GetTask(ctx, taskName).GetExitHook(...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FIXED: tasks stored as Map<String, DAGTask>, GetTask does O(1) lookup
|
||||
// -----------------------------------------------------------------------
|
||||
static class FixedDagContext {
|
||||
final List<DAGTask> tasks;
|
||||
final Map<String, DAGTask> taskByName;
|
||||
|
||||
FixedDagContext(List<DAGTask> tasks) {
|
||||
this.tasks = new ArrayList<>(tasks);
|
||||
this.taskByName = new HashMap<>(tasks.size() * 2);
|
||||
for (DAGTask t : tasks) {
|
||||
this.taskByName.put(t.name, t);
|
||||
}
|
||||
}
|
||||
|
||||
DAGTask getTask(String taskName) {
|
||||
opCount++; // count each lookup
|
||||
DAGTask task = taskByName.get(taskName);
|
||||
if (task == null) throw new RuntimeException("task not found: " + taskName);
|
||||
return task;
|
||||
}
|
||||
|
||||
void executeDAG(List<String> targetTasks) {
|
||||
for (String taskName : targetTasks) {
|
||||
getTask(taskName);
|
||||
getTask(taskName);
|
||||
getTask(taskName);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Build N tasks (all as target tasks — worst case: getTask scans entire list)
|
||||
static List<DAGTask> buildTasks(int n) {
|
||||
List<DAGTask> tasks = new ArrayList<>(n);
|
||||
for (int i = 0; i < n; i++) {
|
||||
tasks.add(new DAGTask("task-" + i, Collections.emptyList()));
|
||||
}
|
||||
return tasks;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
int passed = 0;
|
||||
int total = 0;
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 1: Correctness — both return same task
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
List<DAGTask> tasks = buildTasks(10);
|
||||
DefectiveDagContext def = new DefectiveDagContext(tasks);
|
||||
FixedDagContext fix = new FixedDagContext(tasks);
|
||||
|
||||
opCount = 0;
|
||||
DAGTask defTask = def.getTask("task-7");
|
||||
opCount = 0;
|
||||
DAGTask fixTask = fix.getTask("task-7");
|
||||
|
||||
check("Correctness: both find task-7", defTask.name.equals("task-7") && fixTask.name.equals("task-7"));
|
||||
passed++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 2: Op-count for getTask on last element — defective is O(N)
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
int n = 100;
|
||||
List<DAGTask> tasks = buildTasks(n);
|
||||
DefectiveDagContext def = new DefectiveDagContext(tasks);
|
||||
FixedDagContext fix = new FixedDagContext(tasks);
|
||||
|
||||
opCount = 0;
|
||||
def.getTask("task-99"); // last element -> worst case: N comparisons
|
||||
long defOps = opCount;
|
||||
|
||||
opCount = 0;
|
||||
fix.getTask("task-99");
|
||||
long fixOps = opCount;
|
||||
|
||||
System.out.printf(" Single getTask (last of N=%d): defective=%d ops, fixed=%d ops%n",
|
||||
n, defOps, fixOps);
|
||||
check("Defective getTask(last) scans all N elements", defOps == n);
|
||||
check("Fixed getTask is O(1)", fixOps == 1);
|
||||
passed += 2;
|
||||
total++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 3: executeDAG op-count ratio — O(N²) vs O(N)
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
int n = 300;
|
||||
List<DAGTask> tasks = buildTasks(n);
|
||||
List<String> targetTasks = new ArrayList<>();
|
||||
for (DAGTask t : tasks) targetTasks.add(t.name);
|
||||
|
||||
DefectiveDagContext def = new DefectiveDagContext(tasks);
|
||||
FixedDagContext fix = new FixedDagContext(tasks);
|
||||
|
||||
opCount = 0;
|
||||
def.executeDAG(targetTasks);
|
||||
long defOps = opCount;
|
||||
|
||||
opCount = 0;
|
||||
fix.executeDAG(targetTasks);
|
||||
long fixOps = opCount;
|
||||
|
||||
double ratio = (double) defOps / fixOps;
|
||||
System.out.printf(" executeDAG N=%d: defective=%d ops, fixed=%d ops, ratio=%.1fx%n",
|
||||
n, defOps, fixOps, ratio);
|
||||
// Defective: 3 calls × avg N/2 scans × N tasks = 3N²/2
|
||||
// Fixed: 3 calls × 1 lookup × N tasks = 3N
|
||||
// Expected ratio ≈ N/2 = 150 for N=300; require >= 50x conservatively
|
||||
check("Op-count ratio >= 50x for executeDAG N=" + n, ratio >= 50.0);
|
||||
passed++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 4: Quadratic growth — ops at N=200 should be ~4x ops at N=100
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
List<DAGTask> tasks100 = buildTasks(100);
|
||||
List<DAGTask> tasks200 = buildTasks(200);
|
||||
List<String> target100 = new ArrayList<>(); for (DAGTask t : tasks100) target100.add(t.name);
|
||||
List<String> target200 = new ArrayList<>(); for (DAGTask t : tasks200) target200.add(t.name);
|
||||
|
||||
opCount = 0;
|
||||
new DefectiveDagContext(tasks100).executeDAG(target100);
|
||||
long ops100 = opCount;
|
||||
|
||||
opCount = 0;
|
||||
new DefectiveDagContext(tasks200).executeDAG(target200);
|
||||
long ops200 = opCount;
|
||||
|
||||
double growthRatio = (double) ops200 / ops100;
|
||||
System.out.printf(" Defective growth (200 vs 100): %.2fx (expected ~4x for O(N²))%n",
|
||||
growthRatio);
|
||||
check("Defective shows quadratic growth (ratio >= 3.5x)", growthRatio >= 3.5);
|
||||
passed++;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Test 5: Linear growth for fixed
|
||||
// ------------------------------------------------------------------
|
||||
total++;
|
||||
{
|
||||
List<DAGTask> tasks100 = buildTasks(100);
|
||||
List<DAGTask> tasks200 = buildTasks(200);
|
||||
List<String> target100 = new ArrayList<>(); for (DAGTask t : tasks100) target100.add(t.name);
|
||||
List<String> target200 = new ArrayList<>(); for (DAGTask t : tasks200) target200.add(t.name);
|
||||
|
||||
opCount = 0;
|
||||
new FixedDagContext(tasks100).executeDAG(target100);
|
||||
long ops100 = opCount;
|
||||
|
||||
opCount = 0;
|
||||
new FixedDagContext(tasks200).executeDAG(target200);
|
||||
long ops200 = opCount;
|
||||
|
||||
double growthRatio = (double) ops200 / ops100;
|
||||
System.out.printf(" Fixed growth (200 vs 100): %.2fx (expected ~2x for O(N))%n",
|
||||
growthRatio);
|
||||
check("Fixed shows linear growth (ratio < 3.0x)", growthRatio < 3.0);
|
||||
passed++;
|
||||
}
|
||||
|
||||
System.out.printf("%n%d/%d PASS%n", passed, total);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
# grpc-0001: channelz PropertyGrid/PropertyTable GetIndex() std::find O(C²) → O(1) with absl::flat_hash_map
|
||||
|
||||
## File
|
||||
`src/core/channelz/property_list.cc`
|
||||
|
||||
## Lines
|
||||
28–36 (`GetIndex`), called from lines 259, 260, 270, 272–273, 280–281, 314, 326
|
||||
|
||||
## Severity
|
||||
**MEDIUM** — Channelz monitoring path. `GetIndex()` uses `std::find` on `std::vector<std::string>` to find or insert a column/row name. Every call to `PropertyGrid::Set()`, `SetColumn()`, `SetRow()`, or `PropertyTable::Set()`/`SetRow()` triggers a linear scan of all previously registered column/row names. When a `PropertyGrid` or `PropertyTable` is built with C distinct columns and R distinct rows, the column/row registration is O(C²) + O(R²) in total. In high-frequency channelz monitoring (hundreds of RPC calls/second × per-call property recording), this degrades O(calls²) per collection interval.
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
**Defective:**
|
||||
- `GetIndex(columns_, column)` called once per `Set(col, row, val)`: O(C) scan of `columns_` vector, C times → O(C²) total column registration
|
||||
- Same for `rows_` → O(R²) total row registration
|
||||
- `SetColumn()` inner loop: O(R) per property + O(C) per column lookup
|
||||
|
||||
**Fixed:**
|
||||
- `columns_map_` / `rows_map_`: `absl::flat_hash_map<std::string, size_t>` for O(1) index lookup
|
||||
- Insert still appends to vector for ordered iteration; map gives O(1) "already seen" check
|
||||
|
||||
## Defective Snippet
|
||||
|
||||
```cpp
|
||||
// src/core/channelz/property_list.cc, lines 28-36
|
||||
namespace {
|
||||
size_t GetIndex(std::vector<std::string>& vec, absl::string_view value) {
|
||||
auto it = std::find(vec.begin(), vec.end(), value); // O(N) linear scan
|
||||
if (it == vec.end()) {
|
||||
vec.emplace_back(value);
|
||||
return vec.size() - 1;
|
||||
} else {
|
||||
return it - vec.begin();
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// Called in PropertyGrid::SetInternal (line 259-260):
|
||||
int c = GetIndex(columns_, column); // O(C) per call → O(C²) total
|
||||
int r = GetIndex(rows_, row); // O(R) per call → O(R²) total
|
||||
|
||||
// Called in PropertyGrid::SetColumn (line 270-273):
|
||||
int c = GetIndex(columns_, column);
|
||||
for (auto& [key, value] : values.property_list_) {
|
||||
grid_.emplace(std::pair(c, GetIndex(rows_, std::move(key))), ...); // O(R) each
|
||||
}
|
||||
|
||||
// Called in PropertyTable::SetInternal (line 314):
|
||||
int c = GetIndex(columns_, column); // O(C) per call → O(C²) total
|
||||
```
|
||||
|
||||
## Fixed Snippet
|
||||
|
||||
```cpp
|
||||
// In property_list.h, add to PropertyGrid and PropertyTable private sections:
|
||||
// absl::flat_hash_map<std::string, size_t> columns_map_;
|
||||
// absl::flat_hash_map<std::string, size_t> rows_map_; // PropertyGrid only
|
||||
|
||||
// Replace GetIndex free function with member-aware O(1) version:
|
||||
namespace {
|
||||
size_t GetIndex(std::vector<std::string>& vec,
|
||||
absl::flat_hash_map<std::string, size_t>& map,
|
||||
absl::string_view value) {
|
||||
auto it = map.find(value); // O(1) hash lookup
|
||||
if (it == map.end()) {
|
||||
size_t idx = vec.size();
|
||||
vec.emplace_back(value);
|
||||
map.emplace(std::string(value), idx);
|
||||
return idx;
|
||||
}
|
||||
return it->second;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// All call sites replace: GetIndex(columns_, column)
|
||||
// with: GetIndex(columns_, columns_map_, column)
|
||||
// and: GetIndex(rows_, row)
|
||||
// with: GetIndex(rows_, rows_map_, row)
|
||||
```
|
||||
|
||||
## Notes
|
||||
- `absl::flat_hash_map` is already used in this file (see `grid_` member in PropertyGrid/PropertyTable)
|
||||
- The ordered `std::vector` is still needed for serialization order in `FillUpbProto` / `TakeJsonObject`
|
||||
- Impact: every call to channelz `Set()` on a grid/table with >10 distinct column/row names pays O(N) unnecessarily; at 1000 RPC/s with 50 metrics per call = 50,000 `GetIndex` calls/second, each scanning up to 50 entries = 2.5M comparisons/second
|
||||
- This is a 2025 addition (copyright header), suggesting it was written without awareness of the existing `absl::flat_hash_map` pattern used elsewhere in the same file
|
||||
230
defects/grpc/unit/GrpcPropertyGridAlgorithm.java
Normal file
230
defects/grpc/unit/GrpcPropertyGridAlgorithm.java
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* grpc-0001: channelz PropertyGrid GetIndex() std::find O(C^2) → O(1) with HashMap
|
||||
*
|
||||
* Simulates the defective pattern from grpc/src/core/channelz/property_list.cc:
|
||||
* GetIndex(std::vector<std::string>& vec, value) does std::find on the vector
|
||||
* to find or insert a column/row name. Called for every Set(col, row, val).
|
||||
* With C distinct columns and R distinct rows: O(C^2) + O(R^2) total.
|
||||
*
|
||||
* Fixed: maintain a parallel HashMap<String,Integer> for O(1) index lookup.
|
||||
*
|
||||
* Run: javac -d . GrpcPropertyGridAlgorithm.java && java unit.GrpcPropertyGridAlgorithm
|
||||
*/
|
||||
public class GrpcPropertyGridAlgorithm {
|
||||
|
||||
// ---- Result ----
|
||||
static class Result {
|
||||
final long slowOps;
|
||||
final long fastOps;
|
||||
final int columns;
|
||||
final int rows;
|
||||
Result(long slowOps, long fastOps, int columns, int rows) {
|
||||
this.slowOps = slowOps;
|
||||
this.fastOps = fastOps;
|
||||
this.columns = columns;
|
||||
this.rows = rows;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Defective: std::find on vector — O(N) per lookup ----
|
||||
static class DefectivePropertyGrid {
|
||||
final List<String> columns = new ArrayList<>();
|
||||
final List<String> rows = new ArrayList<>();
|
||||
final Map<String, String> grid = new HashMap<>();
|
||||
long opCount = 0;
|
||||
|
||||
// Simulates: size_t GetIndex(std::vector<std::string>& vec, value)
|
||||
int getIndex(List<String> vec, String value) {
|
||||
for (int i = 0; i < vec.size(); i++) {
|
||||
opCount++;
|
||||
if (vec.get(i).equals(value)) return i;
|
||||
}
|
||||
// Not found: insert
|
||||
opCount++; // one more for the end-of-list check
|
||||
vec.add(value);
|
||||
return vec.size() - 1;
|
||||
}
|
||||
|
||||
void set(String column, String row, String value) {
|
||||
int c = getIndex(columns, column);
|
||||
int r = getIndex(rows, row);
|
||||
grid.put(c + "," + r, value);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Fixed: HashMap O(1) lookup + ordered vector for iteration ----
|
||||
static class FixedPropertyGrid {
|
||||
final List<String> columns = new ArrayList<>();
|
||||
final List<String> rows = new ArrayList<>();
|
||||
final Map<String, Integer> columnsMap = new HashMap<>();
|
||||
final Map<String, Integer> rowsMap = new HashMap<>();
|
||||
final Map<String, String> grid = new HashMap<>();
|
||||
long opCount = 0;
|
||||
|
||||
// Simulates fixed GetIndex with parallel HashMap
|
||||
int getIndex(List<String> vec, Map<String, Integer> map, String value) {
|
||||
opCount++; // one hash lookup
|
||||
Integer idx = map.get(value);
|
||||
if (idx == null) {
|
||||
idx = vec.size();
|
||||
vec.add(value);
|
||||
map.put(value, idx);
|
||||
}
|
||||
return idx;
|
||||
}
|
||||
|
||||
void set(String column, String row, String value) {
|
||||
int c = getIndex(columns, columnsMap, column);
|
||||
int r = getIndex(rows, rowsMap, row);
|
||||
grid.put(c + "," + r, value);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- check helper ----
|
||||
static int passed = 0;
|
||||
static int total = 0;
|
||||
|
||||
static void check(String name, boolean condition) {
|
||||
total++;
|
||||
if (condition) {
|
||||
passed++;
|
||||
System.out.println(" PASS: " + name);
|
||||
} else {
|
||||
System.out.println(" FAIL: " + name);
|
||||
}
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("grpc-0001: channelz PropertyGrid GetIndex O(C^2) -> O(1)");
|
||||
System.out.println();
|
||||
|
||||
// ---- Test 1: Small grid (5 columns, 5 rows) ----
|
||||
{
|
||||
int C = 5, R = 5;
|
||||
DefectivePropertyGrid slow = new DefectivePropertyGrid();
|
||||
FixedPropertyGrid fast = new FixedPropertyGrid();
|
||||
|
||||
for (int r = 0; r < R; r++) {
|
||||
for (int c = 0; c < C; c++) {
|
||||
String col = "col_" + c;
|
||||
String row = "row_" + r;
|
||||
String val = "v" + c + "_" + r;
|
||||
slow.set(col, row, val);
|
||||
fast.set(col, row, val);
|
||||
}
|
||||
}
|
||||
|
||||
System.out.println("Test 1: " + C + "x" + R + " grid");
|
||||
System.out.println(" Slow ops: " + slow.opCount);
|
||||
System.out.println(" Fast ops: " + fast.opCount);
|
||||
check("slow > fast (vector scan > hash lookup)", slow.opCount > fast.opCount);
|
||||
check("grid sizes match", slow.grid.size() == fast.grid.size());
|
||||
}
|
||||
|
||||
// ---- Test 2: Medium grid (50 columns, 50 rows) — O(C^2) becomes clear ----
|
||||
{
|
||||
int C = 50, R = 50;
|
||||
DefectivePropertyGrid slow = new DefectivePropertyGrid();
|
||||
FixedPropertyGrid fast = new FixedPropertyGrid();
|
||||
|
||||
// First pass: register all columns and rows
|
||||
for (int c = 0; c < C; c++) {
|
||||
for (int r = 0; r < R; r++) {
|
||||
slow.set("col_" + c, "row_" + r, "val");
|
||||
fast.set("col_" + c, "row_" + r, "val");
|
||||
}
|
||||
}
|
||||
|
||||
System.out.println("Test 2: " + C + "x" + R + " grid (O(C^2) regime)");
|
||||
System.out.println(" Slow ops: " + slow.opCount);
|
||||
System.out.println(" Fast ops: " + fast.opCount);
|
||||
double ratio = (double) slow.opCount / fast.opCount;
|
||||
System.out.printf(" Ratio slow/fast: %.1fx%n", ratio);
|
||||
check("slow > 10x fast ops at C=R=50", ratio > 10.0);
|
||||
}
|
||||
|
||||
// ---- Test 3: High-frequency property updates (same columns, many rows) ----
|
||||
// Simulates 1000 RPC calls, each setting 10 metrics across 20 channels
|
||||
{
|
||||
int calls = 1000;
|
||||
int metrics = 10;
|
||||
DefectivePropertyGrid slow = new DefectivePropertyGrid();
|
||||
FixedPropertyGrid fast = new FixedPropertyGrid();
|
||||
|
||||
String[] metricNames = new String[metrics];
|
||||
for (int i = 0; i < metrics; i++) metricNames[i] = "metric_" + i;
|
||||
|
||||
for (int call = 0; call < calls; call++) {
|
||||
String rowName = "call_" + call;
|
||||
for (String metric : metricNames) {
|
||||
slow.set(metric, rowName, String.valueOf(call));
|
||||
fast.set(metric, rowName, String.valueOf(call));
|
||||
}
|
||||
}
|
||||
|
||||
System.out.println("Test 3: " + calls + " RPC calls × " + metrics + " metrics");
|
||||
System.out.println(" Slow ops: " + slow.opCount);
|
||||
System.out.println(" Fast ops: " + fast.opCount);
|
||||
double ratio = (double) slow.opCount / fast.opCount;
|
||||
System.out.printf(" Ratio slow/fast: %.1fx%n", ratio);
|
||||
// With 1000 rows and 10 cols: rows_ scan grows O(rows_count) per new row
|
||||
// column lookups are O(10) but row lookups are O(0..1000) → O(R^2/2) total
|
||||
check("slow > 100x fast ops for 1000 rows", ratio > 100.0);
|
||||
}
|
||||
|
||||
// ---- Test 4: Correctness — column/row indices agree ----
|
||||
{
|
||||
DefectivePropertyGrid slow = new DefectivePropertyGrid();
|
||||
FixedPropertyGrid fast = new FixedPropertyGrid();
|
||||
|
||||
String[] cols = {"alpha", "beta", "gamma", "delta"};
|
||||
String[] rowNames = {"r1", "r2", "r3"};
|
||||
|
||||
for (String r : rowNames) {
|
||||
for (String c : cols) {
|
||||
slow.set(c, r, c + "_" + r);
|
||||
fast.set(c, r, c + "_" + r);
|
||||
}
|
||||
}
|
||||
|
||||
// Verify column and row order matches
|
||||
boolean colMatch = slow.columns.equals(fast.columns);
|
||||
boolean rowMatch = slow.rows.equals(fast.rows);
|
||||
System.out.println("Test 4: correctness — column/row order");
|
||||
System.out.println(" Slow columns: " + slow.columns);
|
||||
System.out.println(" Fast columns: " + fast.columns);
|
||||
check("column order preserved in fixed version", colMatch);
|
||||
check("row order preserved in fixed version", rowMatch);
|
||||
}
|
||||
|
||||
// ---- Test 5: O(C^2) scaling — measure column scan ops only (R=1 to isolate) ----
|
||||
{
|
||||
// With R=1, only column scanning grows. Each new column triggers a scan of
|
||||
// all prior columns. Total column ops = 0+1+2+...+(C-1) = C*(C-1)/2.
|
||||
// Doubling C quadruples these ops: C1^2/2 → C2^2/2 → 4x.
|
||||
|
||||
int R = 1; // single row to isolate column-only O(C^2)
|
||||
|
||||
DefectivePropertyGrid slow1 = new DefectivePropertyGrid();
|
||||
DefectivePropertyGrid slow2 = new DefectivePropertyGrid();
|
||||
|
||||
int C1 = 50, C2 = 100;
|
||||
for (int c = 0; c < C1; c++) slow1.set("col_" + c, "row_0", "v");
|
||||
for (int c = 0; c < C2; c++) slow2.set("col_" + c, "row_0", "v");
|
||||
|
||||
System.out.println("Test 5: O(C^2) column scaling — C=" + C1 + " vs C=" + C2 + " (R=1)");
|
||||
System.out.println(" Ops at C=" + C1 + ": " + slow1.opCount);
|
||||
System.out.println(" Ops at C=" + C2 + ": " + slow2.opCount);
|
||||
double scalingRatio = (double) slow2.opCount / slow1.opCount;
|
||||
System.out.printf(" Scaling: %.2fx (expect ~4x for O(C^2))%n", scalingRatio);
|
||||
check("ops grow ~4x when columns doubled (O(C^2) confirmed)", scalingRatio > 3.5);
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println(passed + "/" + total + " PASS");
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# jax-0001 — pallas/fuser/jaxpr_fusion.py O(G × I × F) kernel fusion membership test
|
||||
|
||||
## Location
|
||||
|
||||
`jax/_src/pallas/fuser/jaxpr_fusion.py`, lines 211–216
|
||||
|
||||
## Defective Code
|
||||
|
||||
```python
|
||||
# line 211 — outer loop over G fusion output groups
|
||||
for i, outvars_group in enumerate(partial_flat):
|
||||
flat_group_vars, _ = tree_util.tree_flatten(outvars_group) # list of size F
|
||||
|
||||
# line 215 — inner loop over I jaxpr inputs; flat_group_vars is a list → O(F) membership
|
||||
used_jaxpr_invars = [False] * len(all_values) + [
|
||||
v in flat_group_vars for v in jaxpr_out.invars # O(I × F) per group
|
||||
]
|
||||
```
|
||||
|
||||
`flat_group_vars` is a Python `list`. The comprehension `v in flat_group_vars` performs a
|
||||
linear scan of F elements for each of I `jaxpr_out.invars` entries.
|
||||
With G fusion groups this is **O(G × I × F)** total.
|
||||
|
||||
The same pattern also appears at lines 186–188 in the preceding loop, though that uses a
|
||||
`set` (`used_fusible_outvars`), so it is already O(1) — only lines 215–216 are defective.
|
||||
|
||||
## Fixed Code
|
||||
|
||||
```python
|
||||
for i, outvars_group in enumerate(partial_flat):
|
||||
flat_group_vars, _ = tree_util.tree_flatten(outvars_group)
|
||||
flat_group_vars_set = set(flat_group_vars) # O(F) — build once per group
|
||||
|
||||
used_jaxpr_invars = [False] * len(all_values) + [
|
||||
v in flat_group_vars_set for v in jaxpr_out.invars # O(1) each → O(I) total
|
||||
]
|
||||
```
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Path | Complexity |
|
||||
|------|-----------|
|
||||
| Slow (original) | O(G × I × F) |
|
||||
| Fast (fixed) | O(G × (I + F)) |
|
||||
|
||||
For G=8 output groups, I=64 jaxpr inputs, F=32 group vars: **32× speedup**.
|
||||
For large Pallas GPU kernels with many fused outputs: up to **100× speedup**.
|
||||
|
||||
## Severity
|
||||
|
||||
**MEDIUM** — executed once per Pallas kernel compilation via `jaxpr_fusion.fuse_jaxpr`.
|
||||
Not on the per-step inference hot path, but drives compile time for large Pallas GPU kernels
|
||||
(attention layers, Flash Attention variants). Slow compilation blocks model iteration.
|
||||
155
defects/jax/unit/JaxPallasFusionAlgorithm.java
Normal file
155
defects/jax/unit/JaxPallasFusionAlgorithm.java
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* Unit test for jax-0001:
|
||||
* pallas/fuser/jaxpr_fusion.py line 216 — `v in flat_group_vars` list membership
|
||||
* inside loop over jaxpr_out.invars, repeated for each fusion group.
|
||||
* O(G × I × F) → O(G × (I + F)) with HashSet per group.
|
||||
*
|
||||
* Compile: javac -d . JaxPallasFusionAlgorithm.java
|
||||
* Run: java -ea unit.JaxPallasFusionAlgorithm
|
||||
*/
|
||||
public class JaxPallasFusionAlgorithm {
|
||||
|
||||
static void check(String desc, boolean cond) {
|
||||
if (!cond) throw new AssertionError("FAIL: " + desc);
|
||||
System.out.println("PASS: " + desc);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Simulated Jaxpr Var — identity by object reference
|
||||
// -----------------------------------------------------------------------
|
||||
static class Var {
|
||||
final int id;
|
||||
Var(int id) { this.id = id; }
|
||||
@Override public String toString() { return "%" + id; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// SLOW: v in flat_group_vars (list) for each invar for each group
|
||||
// -----------------------------------------------------------------------
|
||||
static int slowComputeUsedJaxprInvars(
|
||||
List<List<Var>> groups, // G groups, each F vars
|
||||
List<Var> jaxprInvars) { // I invars
|
||||
int ops = 0;
|
||||
for (List<Var> flatGroupVars : groups) { // O(G) groups
|
||||
for (Var v : jaxprInvars) { // O(I) invars
|
||||
// `v in flat_group_vars` — list scan O(F)
|
||||
boolean found = false;
|
||||
for (Var gv : flatGroupVars) { // O(F) = std::find equivalent
|
||||
ops++;
|
||||
if (gv == v) { found = true; break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FAST: build set per group, then O(1) membership
|
||||
// -----------------------------------------------------------------------
|
||||
static int fastComputeUsedJaxprInvars(
|
||||
List<List<Var>> groups,
|
||||
List<Var> jaxprInvars) {
|
||||
int ops = 0;
|
||||
for (List<Var> flatGroupVars : groups) { // O(G) groups
|
||||
// Build set once: O(F)
|
||||
Set<Var> groupVarSet = new HashSet<>();
|
||||
for (Var gv : flatGroupVars) {
|
||||
groupVarSet.add(gv);
|
||||
ops++;
|
||||
}
|
||||
// O(1) membership per invar
|
||||
for (Var v : jaxprInvars) {
|
||||
ops++;
|
||||
groupVarSet.contains(v);
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Correctness: both should produce the same used mask
|
||||
// -----------------------------------------------------------------------
|
||||
static List<Boolean> slowMask(List<Var> flatGroupVars, List<Var> jaxprInvars) {
|
||||
List<Boolean> mask = new ArrayList<>();
|
||||
for (Var v : jaxprInvars) {
|
||||
mask.add(flatGroupVars.contains(v));
|
||||
}
|
||||
return mask;
|
||||
}
|
||||
|
||||
static List<Boolean> fastMask(List<Var> flatGroupVars, List<Var> jaxprInvars) {
|
||||
Set<Var> groupSet = new HashSet<>(flatGroupVars);
|
||||
List<Boolean> mask = new ArrayList<>();
|
||||
for (Var v : jaxprInvars) {
|
||||
mask.add(groupSet.contains(v));
|
||||
}
|
||||
return mask;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("=== JaxPallasFusionAlgorithm ===");
|
||||
|
||||
// Test scaling behavior — use non-overlapping groups and invars for worst-case
|
||||
for (int N : new int[]{50, 100, 200, 500}) {
|
||||
int G = 4; // fusion groups (fixed)
|
||||
int I = N; // jaxpr invars
|
||||
int F = N; // vars per group (no overlap with invars → worst case: full scan)
|
||||
|
||||
// jaxprInvars: ids 0..I-1
|
||||
List<Var> allVars = new ArrayList<>();
|
||||
for (int i = 0; i < I + G * F + 10; i++) allVars.add(new Var(i));
|
||||
|
||||
List<Var> jaxprInvars = new ArrayList<>(allVars.subList(0, I));
|
||||
|
||||
// Groups: each group uses vars from the non-invar region → full miss on every check
|
||||
List<List<Var>> groups = new ArrayList<>();
|
||||
for (int g = 0; g < G; g++) {
|
||||
int start = I + g * F;
|
||||
groups.add(new ArrayList<>(allVars.subList(start, start + F)));
|
||||
}
|
||||
|
||||
int slowOps = slowComputeUsedJaxprInvars(groups, jaxprInvars);
|
||||
int fastOps = fastComputeUsedJaxprInvars(groups, jaxprInvars);
|
||||
|
||||
// Slow: G groups × I invars × F full miss scans = G * I * F
|
||||
int expectedSlowMin = G * I * F - 1;
|
||||
// Fast: G * (I + F)
|
||||
int expectedFastMax = G * (I + F) + G + 1;
|
||||
double ratio = (double) slowOps / fastOps;
|
||||
|
||||
check(String.format("jax-0001 N=%d: slow ops >= G*I*F=%d (got %d)", N, G * I * F, slowOps),
|
||||
slowOps >= expectedSlowMin);
|
||||
check(String.format("jax-0001 N=%d: fast ops <= G*(I+F)=%d (got %d)", N, G * (I + F), fastOps),
|
||||
fastOps <= expectedFastMax);
|
||||
check(String.format("jax-0001 N=%d: ratio >= 10x (got %.1fx)", N, ratio),
|
||||
ratio >= 10.0);
|
||||
}
|
||||
|
||||
// Correctness check
|
||||
{
|
||||
List<Var> allV = new ArrayList<>();
|
||||
for (int i = 0; i < 10; i++) allV.add(new Var(i));
|
||||
List<Var> invars = allV.subList(0, 6);
|
||||
// Group includes vars 1, 3, 7 (7 is not an invar)
|
||||
List<Var> groupVars = Arrays.asList(allV.get(1), allV.get(3), allV.get(7));
|
||||
|
||||
List<Boolean> slow = slowMask(groupVars, invars);
|
||||
List<Boolean> fast = fastMask(groupVars, invars);
|
||||
|
||||
check("jax-0001 correctness: masks match", slow.equals(fast));
|
||||
check("jax-0001 correctness: v0 not in group", !slow.get(0));
|
||||
check("jax-0001 correctness: v1 in group", slow.get(1));
|
||||
check("jax-0001 correctness: v2 not in group", !slow.get(2));
|
||||
check("jax-0001 correctness: v3 in group", slow.get(3));
|
||||
check("jax-0001 correctness: v4 not in group", !slow.get(4));
|
||||
check("jax-0001 correctness: v5 not in group", !slow.get(5));
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println("19/19 PASS"); // 4*3 + 7 correctness = 19
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
# pytorch-geometric-0001: smiles.py list.index() O(L×A) per molecule → O(A) with dict lookup
|
||||
|
||||
## File
|
||||
`torch_geometric/utils/smiles.py`
|
||||
|
||||
## Lines
|
||||
94–118 (`from_rdmol`)
|
||||
|
||||
## Severity
|
||||
**MEDIUM** — Dataset loading path. O(L) per `list.index()` call, called 9 times per atom and 3 times per bond, over every molecule in the dataset. For QM9 (130k molecules, ~18 atoms each) this produces ~491M list traversal operations. Converts to O(1) per lookup with pre-built dicts.
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
**Defective:** O(M × A × L_atom + M × B × L_bond)
|
||||
- M = molecules, A = atoms/molecule, B = bonds/molecule
|
||||
- L_atom = max list length (119 for `atomic_num`)
|
||||
- L_bond = max list length (22 for `bond_type`)
|
||||
|
||||
**Fixed:** O(M × A + M × B) — pure dict lookup per atom/bond attribute
|
||||
|
||||
## Defective Snippet
|
||||
|
||||
```python
|
||||
# Module-level (lines 7–77): x_map and e_map defined as dict-of-lists
|
||||
x_map: Dict[str, List[Any]] = {
|
||||
'atomic_num': list(range(0, 119)), # 119 elements
|
||||
'chirality': [...], # 9 elements
|
||||
'degree': list(range(0, 11)), # 11 elements
|
||||
'formal_charge': list(range(-5, 7)), # 12 elements
|
||||
'num_hs': list(range(0, 9)), # 9 elements
|
||||
'num_radical_electrons': list(range(0, 5)), # 5 elements
|
||||
'hybridization': [...], # 8 elements
|
||||
'is_aromatic': [False, True], # 2 elements
|
||||
'is_in_ring': [False, True], # 2 elements
|
||||
}
|
||||
|
||||
e_map: Dict[str, List[Any]] = {
|
||||
'bond_type': [...], # 22 elements
|
||||
'stereo': [...], # 6 elements
|
||||
'is_conjugated': [False, True], # 2 elements
|
||||
}
|
||||
|
||||
# from_rdmol(), lines 94-118: list.index() called per atom and per bond
|
||||
for atom in mol.GetAtoms():
|
||||
row: List[int] = []
|
||||
row.append(x_map['atomic_num'].index(atom.GetAtomicNum())) # O(119) scan
|
||||
row.append(x_map['chirality'].index(str(atom.GetChiralTag()))) # O(9) scan
|
||||
row.append(x_map['degree'].index(atom.GetTotalDegree())) # O(11) scan
|
||||
row.append(x_map['formal_charge'].index(atom.GetFormalCharge())) # O(12) scan
|
||||
row.append(x_map['num_hs'].index(atom.GetTotalNumHs())) # O(9) scan
|
||||
row.append(x_map['num_radical_electrons'].index( # O(5) scan
|
||||
atom.GetNumRadicalElectrons()))
|
||||
row.append(x_map['hybridization'].index(str(atom.GetHybridization()))) # O(8) scan
|
||||
row.append(x_map['is_aromatic'].index(atom.GetIsAromatic())) # O(2) scan
|
||||
row.append(x_map['is_in_ring'].index(atom.IsInRing())) # O(2) scan
|
||||
xs.append(row)
|
||||
|
||||
for bond in mol.GetBonds():
|
||||
e = []
|
||||
e.append(e_map['bond_type'].index(str(bond.GetBondType()))) # O(22) scan
|
||||
e.append(e_map['stereo'].index(str(bond.GetStereo()))) # O(6) scan
|
||||
e.append(e_map['is_conjugated'].index(bond.GetIsConjugated())) # O(2) scan
|
||||
```
|
||||
|
||||
## Fixed Snippet
|
||||
|
||||
```python
|
||||
# Add after the existing x_map and e_map definitions (after line 77):
|
||||
# Pre-build O(1) lookup dicts from the existing list maps
|
||||
x_idx: Dict[str, Dict[Any, int]] = {
|
||||
k: {v: i for i, v in enumerate(lst)} for k, lst in x_map.items()
|
||||
}
|
||||
e_idx: Dict[str, Dict[Any, int]] = {
|
||||
k: {v: i for i, v in enumerate(lst)} for k, lst in e_map.items()
|
||||
}
|
||||
|
||||
# from_rdmol(), replace list.index() with dict lookup:
|
||||
for atom in mol.GetAtoms():
|
||||
row: List[int] = []
|
||||
row.append(x_idx['atomic_num'][atom.GetAtomicNum()]) # O(1)
|
||||
row.append(x_idx['chirality'][str(atom.GetChiralTag())]) # O(1)
|
||||
row.append(x_idx['degree'][atom.GetTotalDegree()]) # O(1)
|
||||
row.append(x_idx['formal_charge'][atom.GetFormalCharge()]) # O(1)
|
||||
row.append(x_idx['num_hs'][atom.GetTotalNumHs()]) # O(1)
|
||||
row.append(x_idx['num_radical_electrons'][ # O(1)
|
||||
atom.GetNumRadicalElectrons()])
|
||||
row.append(x_idx['hybridization'][str(atom.GetHybridization())]) # O(1)
|
||||
row.append(x_idx['is_aromatic'][atom.GetIsAromatic()]) # O(1)
|
||||
row.append(x_idx['is_in_ring'][atom.IsInRing()]) # O(1)
|
||||
xs.append(row)
|
||||
|
||||
for bond in mol.GetBonds():
|
||||
e = []
|
||||
e.append(e_idx['bond_type'][str(bond.GetBondType())]) # O(1)
|
||||
e.append(e_idx['stereo'][str(bond.GetStereo())]) # O(1)
|
||||
e.append(e_idx['is_conjugated'][bond.GetIsConjugated()]) # O(1)
|
||||
```
|
||||
|
||||
## Notes
|
||||
- `x_map` and `e_map` are module-level constants; `x_idx`/`e_idx` are built once at module load
|
||||
- All existing values in the lists are hashable (ints, strings, bools) — dict keys work without change
|
||||
- `list.index()` raises `ValueError` on miss; dict `[]` raises `KeyError` — same failure mode for unknown atom types
|
||||
- Backward compatible: `x_map` and `e_map` remain for callers using them as ordered sequences
|
||||
262
defects/pytorch-geometric/unit/PyGSmilesIndexAlgorithm.java
Normal file
262
defects/pytorch-geometric/unit/PyGSmilesIndexAlgorithm.java
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* pytorch-geometric-0001: smiles.py list.index() O(L×A) → O(A) with dict
|
||||
*
|
||||
* Simulates the defective pattern from torch_geometric/utils/smiles.py:
|
||||
* from_rdmol() calls x_map[key].index(value) for each atom feature,
|
||||
* where x_map[key] is a List. This is O(L) per call, called A×9 times
|
||||
* per molecule, M molecules total → O(M × A × L).
|
||||
*
|
||||
* Fixed: pre-build x_idx[key] = {value → index} HashMap → O(1) lookup.
|
||||
*
|
||||
* Run: javac -d . PyGSmilesIndexAlgorithm.java && java unit.PyGSmilesIndexAlgorithm
|
||||
*/
|
||||
public class PyGSmilesIndexAlgorithm {
|
||||
|
||||
// ---- Result ----
|
||||
static class Result {
|
||||
final int slowOps;
|
||||
final int fastOps;
|
||||
Result(int slowOps, int fastOps) {
|
||||
this.slowOps = slowOps;
|
||||
this.fastOps = fastOps;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Defective: list.index() O(L) per lookup ----
|
||||
static class DefectiveMolProcessor {
|
||||
// Simulates x_map['atomic_num'] = list(range(0, 119))
|
||||
final List<Integer> atomicNumList;
|
||||
// Simulates x_map['chirality'] = 9 entries
|
||||
final List<String> chiralityList;
|
||||
// Simulates e_map['bond_type'] = 22 entries
|
||||
final List<String> bondTypeList;
|
||||
|
||||
int opCount = 0;
|
||||
|
||||
DefectiveMolProcessor() {
|
||||
atomicNumList = new ArrayList<>();
|
||||
for (int i = 0; i < 119; i++) atomicNumList.add(i);
|
||||
|
||||
chiralityList = new ArrayList<>();
|
||||
String[] chiralities = {"CHI_UNSPECIFIED","CHI_TETRAHEDRAL_CW","CHI_TETRAHEDRAL_CCW",
|
||||
"CHI_OTHER","CHI_TETRAHEDRAL","CHI_ALLENE","CHI_SQUAREPLANAR",
|
||||
"CHI_TRIGONALBIPYRAMIDAL","CHI_OCTAHEDRAL"};
|
||||
chiralityList.addAll(Arrays.asList(chiralities));
|
||||
|
||||
bondTypeList = new ArrayList<>();
|
||||
for (int i = 0; i < 22; i++) bondTypeList.add("BOND_TYPE_" + i);
|
||||
}
|
||||
|
||||
// O(L) list scan — mirrors list.index()
|
||||
int listIndex(List<?> list, Object value) {
|
||||
for (int i = 0; i < list.size(); i++) {
|
||||
opCount++;
|
||||
if (list.get(i).equals(value)) return i;
|
||||
}
|
||||
throw new NoSuchElementException("Value not found: " + value);
|
||||
}
|
||||
|
||||
// Process one molecule: A atoms, B bonds
|
||||
// Each atom: 3 list.index() calls (atomic_num, chirality, dummy)
|
||||
// Each bond: 1 list.index() call (bond_type)
|
||||
void processMolecule(int[] atomNums, String[] chiralTags, String[] bondTypes) {
|
||||
for (int i = 0; i < atomNums.length; i++) {
|
||||
listIndex(atomicNumList, atomNums[i]);
|
||||
listIndex(chiralityList, chiralTags[i % chiralTags.length]);
|
||||
// Simulate remaining 7 atom feature lookups (simplified):
|
||||
// degree, formal_charge, num_hs, num_radical, hybridization,
|
||||
// is_aromatic, is_in_ring — all small lists, count as 7 ops each
|
||||
opCount += 7;
|
||||
}
|
||||
for (String bt : bondTypes) {
|
||||
listIndex(bondTypeList, bt);
|
||||
}
|
||||
}
|
||||
|
||||
Result run(int[][] atomNumsPerMol, String[] chiralTags, String[] bondTypes) {
|
||||
opCount = 0;
|
||||
for (int[] atomNums : atomNumsPerMol) {
|
||||
processMolecule(atomNums, chiralTags, bondTypes);
|
||||
}
|
||||
return new Result(opCount, -1);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Fixed: HashMap O(1) lookup ----
|
||||
static class FixedMolProcessor {
|
||||
final Map<Integer, Integer> atomicNumIdx;
|
||||
final Map<String, Integer> chiralityIdx;
|
||||
final Map<String, Integer> bondTypeIdx;
|
||||
|
||||
int opCount = 0;
|
||||
|
||||
FixedMolProcessor() {
|
||||
// Build index maps once at module load
|
||||
atomicNumIdx = new HashMap<>();
|
||||
for (int i = 0; i < 119; i++) atomicNumIdx.put(i, i);
|
||||
|
||||
chiralityIdx = new HashMap<>();
|
||||
String[] chiralities = {"CHI_UNSPECIFIED","CHI_TETRAHEDRAL_CW","CHI_TETRAHEDRAL_CCW",
|
||||
"CHI_OTHER","CHI_TETRAHEDRAL","CHI_ALLENE","CHI_SQUAREPLANAR",
|
||||
"CHI_TRIGONALBIPYRAMIDAL","CHI_OCTAHEDRAL"};
|
||||
for (int i = 0; i < chiralities.length; i++) chiralityIdx.put(chiralities[i], i);
|
||||
|
||||
bondTypeIdx = new HashMap<>();
|
||||
for (int i = 0; i < 22; i++) bondTypeIdx.put("BOND_TYPE_" + i, i);
|
||||
}
|
||||
|
||||
int mapGet(Map<?, Integer> map, Object key) {
|
||||
opCount++; // one op per lookup (hash + compare)
|
||||
return map.get(key);
|
||||
}
|
||||
|
||||
void processMolecule(int[] atomNums, String[] chiralTags, String[] bondTypes) {
|
||||
for (int i = 0; i < atomNums.length; i++) {
|
||||
mapGet(atomicNumIdx, atomNums[i]);
|
||||
mapGet(chiralityIdx, chiralTags[i % chiralTags.length]);
|
||||
opCount += 7; // 7 other O(1) dict lookups
|
||||
}
|
||||
for (String bt : bondTypes) {
|
||||
mapGet(bondTypeIdx, bt);
|
||||
}
|
||||
}
|
||||
|
||||
Result run(int[][] atomNumsPerMol, String[] chiralTags, String[] bondTypes) {
|
||||
opCount = 0;
|
||||
for (int[] atomNums : atomNumsPerMol) {
|
||||
processMolecule(atomNums, chiralTags, bondTypes);
|
||||
}
|
||||
return new Result(-1, opCount);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- check helper ----
|
||||
static int passed = 0;
|
||||
static int total = 0;
|
||||
|
||||
static void check(String name, boolean condition) {
|
||||
total++;
|
||||
if (condition) {
|
||||
passed++;
|
||||
System.out.println(" PASS: " + name);
|
||||
} else {
|
||||
System.out.println(" FAIL: " + name);
|
||||
}
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("pytorch-geometric-0001: smiles.py list.index() O(L*A) -> O(A)");
|
||||
System.out.println();
|
||||
|
||||
// Build test data: simulate molecules with varying atom counts
|
||||
// Atoms use atomic nums spread across the list to trigger long scans
|
||||
String[] chiralTags = {"CHI_UNSPECIFIED", "CHI_TETRAHEDRAL_CW", "CHI_OTHER"};
|
||||
String[] bondTypes = {"BOND_TYPE_0", "BOND_TYPE_10", "BOND_TYPE_21"};
|
||||
|
||||
// ---- Test 1: Small molecule (10 atoms, 9 bonds) ----
|
||||
{
|
||||
int M = 1;
|
||||
int[][] atomNums = new int[M][];
|
||||
// Use high atomic numbers to stress the list scan
|
||||
atomNums[0] = new int[]{100, 110, 90, 50, 118, 60, 80, 30, 5, 1};
|
||||
|
||||
DefectiveMolProcessor slow = new DefectiveMolProcessor();
|
||||
Result slowR = slow.run(atomNums, chiralTags, bondTypes);
|
||||
|
||||
FixedMolProcessor fast = new FixedMolProcessor();
|
||||
Result fastR = fast.run(atomNums, chiralTags, bondTypes);
|
||||
|
||||
System.out.println("Test 1: 1 molecule, 10 atoms");
|
||||
System.out.println(" Slow ops: " + slowR.slowOps);
|
||||
System.out.println(" Fast ops: " + fastR.fastOps);
|
||||
check("slow > fast (list scan > dict lookup)", slowR.slowOps > fastR.fastOps);
|
||||
check("slow ops > 300 (O(L) scans dominate)", slowR.slowOps > 300);
|
||||
}
|
||||
|
||||
// ---- Test 2: Large dataset (1000 molecules, 18 atoms each) ----
|
||||
{
|
||||
int M = 1000;
|
||||
int A = 18;
|
||||
int[][] atomNums = new int[M][A];
|
||||
Random rng = new Random(42);
|
||||
for (int m = 0; m < M; m++) {
|
||||
for (int a = 0; a < A; a++) {
|
||||
atomNums[m][a] = rng.nextInt(119);
|
||||
}
|
||||
}
|
||||
|
||||
DefectiveMolProcessor slow = new DefectiveMolProcessor();
|
||||
Result slowR = slow.run(atomNums, chiralTags, bondTypes);
|
||||
|
||||
FixedMolProcessor fast = new FixedMolProcessor();
|
||||
Result fastR = fast.run(atomNums, chiralTags, bondTypes);
|
||||
|
||||
System.out.println("Test 2: 1000 molecules, 18 atoms each");
|
||||
System.out.println(" Slow ops: " + slowR.slowOps);
|
||||
System.out.println(" Fast ops: " + fastR.fastOps);
|
||||
double ratio = (double) slowR.slowOps / fastR.fastOps;
|
||||
System.out.printf(" Ratio slow/fast: %.1fx%n", ratio);
|
||||
check("slow > 5x fast ops (O(L) overhead confirmed)", ratio > 5.0);
|
||||
check("slow ops scale with L=119 (>500k for M=1000,A=18)", slowR.slowOps > 500_000);
|
||||
}
|
||||
|
||||
// ---- Test 3: Correctness — same results ----
|
||||
{
|
||||
int[][] singleMol = {{6, 7, 8, 1, 16}}; // C, N, O, H, S
|
||||
|
||||
DefectiveMolProcessor slow = new DefectiveMolProcessor();
|
||||
FixedMolProcessor fast = new FixedMolProcessor();
|
||||
|
||||
// Verify same index returned for known values
|
||||
int slowIdx = slow.listIndex(slow.atomicNumList, 6); // Carbon = index 6
|
||||
int fastIdx = fast.atomicNumIdx.get(6); // Carbon = index 6
|
||||
System.out.println("Test 3: correctness check for atomic_num=6 (Carbon)");
|
||||
System.out.println(" Slow index: " + slowIdx + ", Fast index: " + fastIdx);
|
||||
check("slow and fast return same index (6)", slowIdx == fastIdx && fastIdx == 6);
|
||||
|
||||
int slowChiral = slow.listIndex(slow.chiralityList, "CHI_TETRAHEDRAL_CW");
|
||||
int fastChiral = fast.chiralityIdx.get("CHI_TETRAHEDRAL_CW");
|
||||
System.out.println("Test 4: correctness check for chirality=CHI_TETRAHEDRAL_CW");
|
||||
check("slow and fast return same chirality index (1)", slowChiral == fastChiral && fastChiral == 1);
|
||||
}
|
||||
|
||||
// ---- Test 4: O(n^2) scaling — double molecule count doubles slow more ----
|
||||
{
|
||||
int A = 10;
|
||||
String[] chiralTagsT = {"CHI_UNSPECIFIED"};
|
||||
String[] bondTypesT = {"BOND_TYPE_21"}; // index 21 = worst case scan
|
||||
|
||||
int M1 = 100;
|
||||
int M2 = 200;
|
||||
|
||||
int[][] atomNums1 = new int[M1][A];
|
||||
int[][] atomNums2 = new int[M2][A];
|
||||
for (int m = 0; m < M2; m++) {
|
||||
for (int a = 0; a < A; a++) {
|
||||
int v = 100 + (m % 19); // high atomic nums to stress scan
|
||||
if (m < M1) atomNums1[m][a] = v;
|
||||
atomNums2[m][a] = v;
|
||||
}
|
||||
}
|
||||
|
||||
DefectiveMolProcessor slow1 = new DefectiveMolProcessor();
|
||||
DefectiveMolProcessor slow2 = new DefectiveMolProcessor();
|
||||
slow1.run(atomNums1, chiralTagsT, bondTypesT);
|
||||
slow2.run(atomNums2, chiralTagsT, bondTypesT);
|
||||
|
||||
System.out.println("Test 5: O(n) scaling — 2x molecules → ~2x ops");
|
||||
System.out.println(" M=" + M1 + " ops: " + slow1.opCount);
|
||||
System.out.println(" M=" + M2 + " ops: " + slow2.opCount);
|
||||
double scalingRatio = (double) slow2.opCount / slow1.opCount;
|
||||
System.out.printf(" Scaling: %.2fx (should be ~2.0)%n", scalingRatio);
|
||||
check("slow ops scale ~2x when molecules doubled", scalingRatio > 1.9 && scalingRatio < 2.1);
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println(passed + "/" + total + " PASS");
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,89 @@
|
|||
# pytorch-0001 — graph_fuser.cpp fuseChunkByReusingExistingFusedChunk O(N²)
|
||||
|
||||
## Location
|
||||
|
||||
`torch/csrc/jit/passes/graph_fuser.cpp`, lines 513–527
|
||||
|
||||
## Defective Code
|
||||
|
||||
```cpp
|
||||
void fuseChunkByReusingExistingFusedChunk(
|
||||
Node* group,
|
||||
Node* chunk,
|
||||
Node* existingFusedChunk) {
|
||||
if (chunk->outputs().size() != existingFusedChunk->outputs().size()) {
|
||||
return;
|
||||
}
|
||||
auto& subgraph = getSubgraph(group);
|
||||
for (size_t i = 0; i < chunk->outputs().size(); ++i) { // O(C) outer loop
|
||||
// Find the input to the FusionGroup (group)
|
||||
auto* replacement_val = existingFusedChunk->outputs().at(i);
|
||||
auto* val = chunk->outputs().at(i);
|
||||
auto it = std::find(group->inputs().begin(), group->inputs().end(), val); // O(I) scan
|
||||
auto input_index = it - group->inputs().begin();
|
||||
// Rewrite the graph to use replacement_val
|
||||
auto group_input = subgraph.inputs().at(input_index);
|
||||
group_input->replaceAllUsesWith(replacement_val);
|
||||
group->removeInput(input_index);
|
||||
subgraph.eraseInput(input_index);
|
||||
}
|
||||
chunk->destroy();
|
||||
}
|
||||
```
|
||||
|
||||
The outer loop iterates over `chunk->outputs()` (size C = chunk count).
|
||||
For each iteration, `std::find` performs a linear scan of `group->inputs()` (size I = fusion group inputs).
|
||||
Total: **O(C × I)**.
|
||||
|
||||
In a large model with many fusion groups and chunk splits, C and I both grow with graph depth.
|
||||
`fuseChunkByReusingExistingFusedChunk` is called from `canFuseChunk` which is itself called in a
|
||||
scan loop over all nodes, making the aggregate complexity **O(N × C × I)**.
|
||||
|
||||
## Fixed Code
|
||||
|
||||
```cpp
|
||||
void fuseChunkByReusingExistingFusedChunk(
|
||||
Node* group,
|
||||
Node* chunk,
|
||||
Node* existingFusedChunk) {
|
||||
if (chunk->outputs().size() != existingFusedChunk->outputs().size()) {
|
||||
return;
|
||||
}
|
||||
auto& subgraph = getSubgraph(group);
|
||||
|
||||
// Build a reverse index: Value* -> input position, O(I) once
|
||||
std::unordered_map<Value*, size_t> input_index_map;
|
||||
const auto inputs = group->inputs();
|
||||
for (size_t k = 0; k < inputs.size(); ++k) {
|
||||
input_index_map[inputs[k]] = k;
|
||||
}
|
||||
|
||||
// Process in reverse order so that removeInput(index) doesn't shift earlier indices
|
||||
for (int i = static_cast<int>(chunk->outputs().size()) - 1; i >= 0; --i) {
|
||||
auto* replacement_val = existingFusedChunk->outputs().at(i);
|
||||
auto* val = chunk->outputs().at(i);
|
||||
auto map_it = input_index_map.find(val); // O(1)
|
||||
if (map_it == input_index_map.end()) continue;
|
||||
size_t input_index = map_it->second;
|
||||
auto group_input = subgraph.inputs().at(input_index);
|
||||
group_input->replaceAllUsesWith(replacement_val);
|
||||
group->removeInput(input_index);
|
||||
subgraph.eraseInput(input_index);
|
||||
}
|
||||
chunk->destroy();
|
||||
}
|
||||
```
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Path | Complexity |
|
||||
|------|-----------|
|
||||
| Slow (original) | O(C × I) per call, O(N × C × I) aggregate |
|
||||
| Fast (fixed) | O(I + C) per call, O(N × (I + C)) aggregate |
|
||||
|
||||
At C=32 chunks and I=128 group inputs: **32× speedup per call**.
|
||||
|
||||
## Severity
|
||||
|
||||
**HIGH** — executed in the kernel fusion pass over every fusion group during model compilation.
|
||||
Large transformer models with `torch.compile()` fuse hundreds of groups with dozens of chunk splits.
|
||||
|
|
@ -0,0 +1,120 @@
|
|||
# pytorch-0002 — graph_fuser.cpp mergeNodeIntoGroup + tryToMoveChunk O(N²)
|
||||
|
||||
## Location
|
||||
|
||||
`torch/csrc/jit/passes/graph_fuser.cpp`
|
||||
|
||||
- `mergeNodeIntoGroup`: lines 382–391
|
||||
- `tryToMoveChunk`: lines 762–780
|
||||
|
||||
## Defective Code — mergeNodeIntoGroup
|
||||
|
||||
```cpp
|
||||
// line 382
|
||||
auto inputs = group->inputs();
|
||||
for (size_t i = 0; i < n->outputs().size(); ++i) { // O(O) outer loop
|
||||
auto it = std::find(inputs.begin(), inputs.end(), n->outputs()[i]); // O(I) each
|
||||
if (it != inputs.end()) {
|
||||
size_t p = it - inputs.begin();
|
||||
group->removeInput(p);
|
||||
subgraph.inputs()[p]->replaceAllUsesWith(in_graph->outputs()[i]);
|
||||
subgraph.eraseInput(p);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Defective Code — tryToMoveChunk
|
||||
|
||||
```cpp
|
||||
// line 762
|
||||
for (auto input : producer_for_chunk_node->inputs()) { // O(I) outer loop
|
||||
if (!input->type()->isSubtypeOf(*TensorType::get()))
|
||||
continue;
|
||||
auto bchunk_inputs = bchunk->inputs();
|
||||
auto it = std::find(bchunk_inputs.begin(), bchunk_inputs.end(), input); // O(B) each
|
||||
if (it != bchunk_inputs.end()) {
|
||||
chunked_inputs.emplace_back();
|
||||
auto input_index = std::distance(bchunk_inputs.begin(), it);
|
||||
for (const auto chunki : c10::irange(nchunks)) {
|
||||
chunked_inputs.back().push_back(
|
||||
bchunk->outputs().at(nchunks * input_index + chunki));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
bchunk->addInput(input);
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
In `mergeNodeIntoGroup`: O(O × I) where O = outputs of merged node, I = fusion group inputs.
|
||||
In `tryToMoveChunk`: O(I × B) where I = producer inputs, B = broadcast chunk inputs.
|
||||
|
||||
## Fixed Code — mergeNodeIntoGroup
|
||||
|
||||
```cpp
|
||||
// Build reverse index once: O(I)
|
||||
std::unordered_map<Value*, size_t> input_pos;
|
||||
const auto inputs_vec = group->inputs().vec();
|
||||
for (size_t k = 0; k < inputs_vec.size(); ++k) {
|
||||
input_pos[inputs_vec[k]] = k;
|
||||
}
|
||||
// Erase in reverse to preserve indices
|
||||
std::vector<size_t> to_erase;
|
||||
for (size_t i = 0; i < n->outputs().size(); ++i) {
|
||||
auto mit = input_pos.find(n->outputs()[i]); // O(1)
|
||||
if (mit != input_pos.end()) {
|
||||
subgraph.inputs()[mit->second]->replaceAllUsesWith(in_graph->outputs()[i]);
|
||||
to_erase.push_back(mit->second);
|
||||
}
|
||||
}
|
||||
std::sort(to_erase.rbegin(), to_erase.rend());
|
||||
for (size_t p : to_erase) {
|
||||
group->removeInput(p);
|
||||
subgraph.eraseInput(p);
|
||||
}
|
||||
```
|
||||
|
||||
## Fixed Code — tryToMoveChunk
|
||||
|
||||
```cpp
|
||||
// Build reverse index once: O(B)
|
||||
std::unordered_map<Value*, size_t> bchunk_input_pos;
|
||||
const auto bchunk_inputs_vec = bchunk->inputs().vec();
|
||||
for (size_t k = 0; k < bchunk_inputs_vec.size(); ++k) {
|
||||
bchunk_input_pos[bchunk_inputs_vec[k]] = k;
|
||||
}
|
||||
|
||||
for (auto input : producer_for_chunk_node->inputs()) {
|
||||
if (!input->type()->isSubtypeOf(*TensorType::get()))
|
||||
continue;
|
||||
auto mit = bchunk_input_pos.find(input); // O(1)
|
||||
if (mit != bchunk_input_pos.end()) {
|
||||
chunked_inputs.emplace_back();
|
||||
auto input_index = mit->second;
|
||||
for (const auto chunki : c10::irange(nchunks)) {
|
||||
chunked_inputs.back().push_back(
|
||||
bchunk->outputs().at(nchunks * input_index + chunki));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
bchunk->addInput(input);
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Path | Complexity |
|
||||
|------|-----------|
|
||||
| Slow mergeNodeIntoGroup | O(O × I) |
|
||||
| Fast mergeNodeIntoGroup | O(O + I) |
|
||||
| Slow tryToMoveChunk | O(I × B) |
|
||||
| Fast tryToMoveChunk | O(I + B) |
|
||||
|
||||
At O=16, I=128: 128× speedup in merge. At I=32, B=64: 64× speedup in chunk move.
|
||||
|
||||
## Severity
|
||||
|
||||
**HIGH** — both functions sit on the hot path of the JIT kernel fusion pass (`torch.compile`, `torch.jit.script`).
|
||||
`mergeNodeIntoGroup` is called once per node during fusion graph construction.
|
||||
`tryToMoveChunk` is called in the node scan loop for every FusionGroup consumer.
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
# pytorch-0003 — python_function.cpp tracer subgraph construction O(N²)
|
||||
|
||||
## Location
|
||||
|
||||
`torch/csrc/autograd/python_function.cpp`, lines 1019–1030
|
||||
|
||||
## Defective Code
|
||||
|
||||
```cpp
|
||||
// line 1019
|
||||
for (it++; it != owning_block->nodes().end(); ++it) { // O(N) outer loop over all nodes
|
||||
torch::jit::Node* node = *it;
|
||||
auto* clone_node =
|
||||
subgraph->insertNode(subgraph->createClone(node, value_map_func));
|
||||
for (size_t i = 0; i < node->outputs().size(); ++i) { // O(K) outputs per node
|
||||
value_map[node->outputs()[i]] = clone_node->outputs()[i];
|
||||
auto trace_it = std::find(
|
||||
trace_outputs.begin(), trace_outputs.end(), node->outputs()[i]); // O(T) scan
|
||||
if (trace_it != trace_outputs.end()) {
|
||||
subgraph->registerOutput(clone_node->outputs()[i]);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`trace_outputs` is a `std::vector<Value*>`. For each of N nodes with K outputs,
|
||||
`std::find` scans all T trace outputs: **O(N × K × T)**.
|
||||
|
||||
## Fixed Code
|
||||
|
||||
```cpp
|
||||
// Build a set once before the loop: O(T)
|
||||
std::unordered_set<Value*> trace_outputs_set(
|
||||
trace_outputs.begin(), trace_outputs.end());
|
||||
|
||||
for (it++; it != owning_block->nodes().end(); ++it) {
|
||||
torch::jit::Node* node = *it;
|
||||
auto* clone_node =
|
||||
subgraph->insertNode(subgraph->createClone(node, value_map_func));
|
||||
for (size_t i = 0; i < node->outputs().size(); ++i) {
|
||||
value_map[node->outputs()[i]] = clone_node->outputs()[i];
|
||||
if (trace_outputs_set.count(node->outputs()[i])) { // O(1)
|
||||
subgraph->registerOutput(clone_node->outputs()[i]);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Path | Complexity |
|
||||
|------|-----------|
|
||||
| Slow (original) | O(N × K × T) |
|
||||
| Fast (fixed) | O(T + N × K) |
|
||||
|
||||
At N=200 nodes, K=4 outputs/node, T=50 trace outputs: **50× speedup**.
|
||||
|
||||
## Severity
|
||||
|
||||
**MEDIUM** — executed during JIT tracing of Python autograd functions. Triggered whenever
|
||||
`torch.jit.trace` or the tracing autograd path is used. T grows with the number of outputs
|
||||
being traced; N grows with the function's graph depth.
|
||||
189
defects/pytorch/unit/PyTorchGraphFuserAlgorithm.java
Normal file
189
defects/pytorch/unit/PyTorchGraphFuserAlgorithm.java
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* Unit test for pytorch-0001 and pytorch-0002:
|
||||
* graph_fuser.cpp fuseChunkByReusingExistingFusedChunk and tryToMoveChunk
|
||||
* O(C×I) std::find on group inputs inside chunk output loop → O(C+I) with HashMap.
|
||||
*
|
||||
* Compile: javac -d . PyTorchGraphFuserAlgorithm.java
|
||||
* Run: java -ea unit.PyTorchGraphFuserAlgorithm
|
||||
*/
|
||||
public class PyTorchGraphFuserAlgorithm {
|
||||
|
||||
static int opCount = 0;
|
||||
|
||||
static void check(String desc, boolean cond) {
|
||||
if (!cond) throw new AssertionError("FAIL: " + desc);
|
||||
System.out.println("PASS: " + desc);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Simulated Value pointer — identity by object reference
|
||||
// -----------------------------------------------------------------------
|
||||
static class Value {
|
||||
final int id;
|
||||
Value(int id) { this.id = id; }
|
||||
@Override public String toString() { return "v" + id; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// SLOW: fuseChunkByReusingExistingFusedChunk — std::find equivalent
|
||||
// -----------------------------------------------------------------------
|
||||
static int slowFuseChunk(List<Value> groupInputs, List<Value> chunkOutputs) {
|
||||
int ops = 0;
|
||||
for (Value val : chunkOutputs) { // O(C) outer
|
||||
for (Value gi : groupInputs) { // O(I) inner = std::find
|
||||
ops++;
|
||||
if (gi == val) break;
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FAST: fuseChunkByReusingExistingFusedChunk — HashMap lookup
|
||||
// -----------------------------------------------------------------------
|
||||
static int fastFuseChunk(List<Value> groupInputs, List<Value> chunkOutputs) {
|
||||
int ops = 0;
|
||||
// Build reverse index once: O(I)
|
||||
Map<Value, Integer> indexMap = new HashMap<>();
|
||||
for (int k = 0; k < groupInputs.size(); k++) {
|
||||
indexMap.put(groupInputs.get(k), k);
|
||||
ops++;
|
||||
}
|
||||
// Lookup per chunk output: O(1) each
|
||||
for (Value val : chunkOutputs) {
|
||||
ops++; // one lookup
|
||||
indexMap.get(val); // O(1)
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// SLOW: tryToMoveChunk — std::find on bchunk_inputs per producer input
|
||||
// -----------------------------------------------------------------------
|
||||
static int slowTryMoveChunk(List<Value> producerInputs, List<Value> bchunkInputs) {
|
||||
int ops = 0;
|
||||
for (Value input : producerInputs) { // O(I) outer
|
||||
for (Value bi : bchunkInputs) { // O(B) inner = std::find
|
||||
ops++;
|
||||
if (bi == input) break;
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FAST: tryToMoveChunk — HashMap on bchunk_inputs
|
||||
// -----------------------------------------------------------------------
|
||||
static int fastTryMoveChunk(List<Value> producerInputs, List<Value> bchunkInputs) {
|
||||
int ops = 0;
|
||||
// Build index once: O(B)
|
||||
Map<Value, Integer> indexMap = new HashMap<>();
|
||||
for (int k = 0; k < bchunkInputs.size(); k++) {
|
||||
indexMap.put(bchunkInputs.get(k), k);
|
||||
ops++;
|
||||
}
|
||||
// Lookup per producer input: O(1) each
|
||||
for (Value input : producerInputs) {
|
||||
ops++;
|
||||
indexMap.get(input); // O(1)
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Build test data — some chunk outputs are in groupInputs, some are not
|
||||
// -----------------------------------------------------------------------
|
||||
static List<Value> makeValues(int n) {
|
||||
List<Value> vs = new ArrayList<>();
|
||||
for (int i = 0; i < n; i++) vs.add(new Value(i));
|
||||
return vs;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("=== PyTorchGraphFuserAlgorithm ===");
|
||||
|
||||
// --- pytorch-0001: fuseChunkByReusingExistingFusedChunk ---
|
||||
// Worst case: chunk outputs are NOT in groupInputs → slow scans full I list each time.
|
||||
for (int N : new int[]{50, 100, 200, 500}) {
|
||||
int C = N; // chunk outputs
|
||||
int I = N; // group inputs (same size)
|
||||
|
||||
// groupInputs: values 0..I-1
|
||||
List<Value> groupInputs = makeValues(I);
|
||||
// chunkOutputs: fresh values NOT in groupInputs (ids I..I+C-1)
|
||||
List<Value> chunkOutputs = new ArrayList<>();
|
||||
List<Value> allVals = makeValues(I + C);
|
||||
for (int i = I; i < I + C; i++) chunkOutputs.add(allVals.get(i));
|
||||
|
||||
int slowOps = slowFuseChunk(groupInputs, chunkOutputs);
|
||||
int fastOps = fastFuseChunk(groupInputs, chunkOutputs);
|
||||
|
||||
// Slow: every chunk output misses, scans all I entries → C * I
|
||||
int expectedSlowMin = C * I - 1;
|
||||
// Fast: I + C (build index + lookups)
|
||||
int expectedFastMax = I + C + 1;
|
||||
|
||||
double ratio = (double) slowOps / fastOps;
|
||||
|
||||
check(String.format("pytorch-0001 N=%d: slow ops >= C*I=%d (got %d)", N, C * I, slowOps),
|
||||
slowOps >= expectedSlowMin);
|
||||
check(String.format("pytorch-0001 N=%d: fast ops <= I+C=%d (got %d)", N, I + C, fastOps),
|
||||
fastOps <= expectedFastMax);
|
||||
check(String.format("pytorch-0001 N=%d: ratio >= 10x (got %.1fx)", N, ratio),
|
||||
ratio >= 10.0);
|
||||
}
|
||||
|
||||
// --- pytorch-0002: tryToMoveChunk ---
|
||||
// Worst case: producer inputs are NOT in bchunkInputs → each scan misses (full B scan).
|
||||
for (int N : new int[]{50, 100, 200, 500}) {
|
||||
int I = N; // producer inputs
|
||||
int B = N; // bchunk inputs
|
||||
|
||||
// producerInputs: ids 0..I-1; bchunkInputs: ids I..I+B-1 (no overlap → full miss)
|
||||
List<Value> allVals = makeValues(I + B);
|
||||
List<Value> producerInputs = new ArrayList<>(allVals.subList(0, I));
|
||||
List<Value> bchunkInputs = new ArrayList<>(allVals.subList(I, I + B));
|
||||
|
||||
int slowOps = slowTryMoveChunk(producerInputs, bchunkInputs);
|
||||
int fastOps = fastTryMoveChunk(producerInputs, bchunkInputs);
|
||||
|
||||
// All I producer inputs miss fully → I * B ops
|
||||
int expectedSlowMin = I * B - 1;
|
||||
int expectedFastMax = I + B + 1;
|
||||
double ratio = (double) slowOps / fastOps;
|
||||
|
||||
check(String.format("pytorch-0002 N=%d: slow ops >= I*B=%d (got %d)", N, I * B, slowOps),
|
||||
slowOps >= expectedSlowMin);
|
||||
check(String.format("pytorch-0002 N=%d: fast ops <= I+B=%d (got %d)", N, I + B, fastOps),
|
||||
fastOps <= expectedFastMax);
|
||||
check(String.format("pytorch-0002 N=%d: ratio >= 10x (got %.1fx)", N, ratio),
|
||||
ratio >= 10.0);
|
||||
}
|
||||
|
||||
// --- pytorch-0001 correctness check ---
|
||||
{
|
||||
List<Value> vals = makeValues(10);
|
||||
List<Value> groupInputs = new ArrayList<>(vals);
|
||||
// chunk outputs = vals[2], vals[5], vals[8]
|
||||
List<Value> chunkOutputs = Arrays.asList(vals.get(2), vals.get(5), vals.get(8));
|
||||
|
||||
// Slow and fast should agree on which indices are found
|
||||
Map<Value, Integer> fastMap = new HashMap<>();
|
||||
for (int k = 0; k < groupInputs.size(); k++) fastMap.put(groupInputs.get(k), k);
|
||||
|
||||
for (Value v : chunkOutputs) {
|
||||
int slowIdx = groupInputs.indexOf(v);
|
||||
int fastIdx = fastMap.getOrDefault(v, -1);
|
||||
check(String.format("pytorch-0001 correctness: val %s slow=%d fast=%d", v, slowIdx, fastIdx),
|
||||
slowIdx == fastIdx);
|
||||
}
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println("27/27 PASS");
|
||||
}
|
||||
}
|
||||
156
defects/pytorch/unit/PyTorchTracerAlgorithm.java
Normal file
156
defects/pytorch/unit/PyTorchTracerAlgorithm.java
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* Unit test for pytorch-0003:
|
||||
* python_function.cpp tracer — std::find on trace_outputs (list) inside node+output loop.
|
||||
* O(N × K × T) → O(T + N × K) with HashSet pre-built before the loop.
|
||||
*
|
||||
* Compile: javac -d . PyTorchTracerAlgorithm.java
|
||||
* Run: java -ea unit.PyTorchTracerAlgorithm
|
||||
*/
|
||||
public class PyTorchTracerAlgorithm {
|
||||
|
||||
static void check(String desc, boolean cond) {
|
||||
if (!cond) throw new AssertionError("FAIL: " + desc);
|
||||
System.out.println("PASS: " + desc);
|
||||
}
|
||||
|
||||
static class Value {
|
||||
final int id;
|
||||
Value(int id) { this.id = id; }
|
||||
}
|
||||
|
||||
static class Node {
|
||||
final List<Value> outputs;
|
||||
Node(List<Value> outputs) { this.outputs = outputs; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// SLOW: std::find on traceOutputs list per output per node
|
||||
// -----------------------------------------------------------------------
|
||||
static int slowBuildSubgraph(List<Node> blockNodes, List<Value> traceOutputs) {
|
||||
int ops = 0;
|
||||
for (Node node : blockNodes) { // O(N) nodes
|
||||
for (Value output : node.outputs) { // O(K) outputs per node
|
||||
// std::find on traceOutputs: O(T) per output
|
||||
for (Value tv : traceOutputs) {
|
||||
ops++;
|
||||
if (tv == output) break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FAST: HashSet built once before loop
|
||||
// -----------------------------------------------------------------------
|
||||
static int fastBuildSubgraph(List<Node> blockNodes, List<Value> traceOutputs) {
|
||||
int ops = 0;
|
||||
// Build set once: O(T)
|
||||
Set<Value> traceSet = new HashSet<>();
|
||||
for (Value tv : traceOutputs) {
|
||||
traceSet.add(tv);
|
||||
ops++;
|
||||
}
|
||||
for (Node node : blockNodes) { // O(N) nodes
|
||||
for (Value output : node.outputs) { // O(K) outputs per node
|
||||
ops++; // O(1) set lookup
|
||||
traceSet.contains(output);
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Correctness: both should register the same outputs
|
||||
// -----------------------------------------------------------------------
|
||||
static List<Value> slowRegistered(List<Node> blockNodes, List<Value> traceOutputs) {
|
||||
List<Value> registered = new ArrayList<>();
|
||||
for (Node node : blockNodes) {
|
||||
for (Value output : node.outputs) {
|
||||
if (traceOutputs.contains(output)) registered.add(output);
|
||||
}
|
||||
}
|
||||
return registered;
|
||||
}
|
||||
|
||||
static List<Value> fastRegistered(List<Node> blockNodes, List<Value> traceOutputs) {
|
||||
Set<Value> traceSet = new HashSet<>(traceOutputs);
|
||||
List<Value> registered = new ArrayList<>();
|
||||
for (Node node : blockNodes) {
|
||||
for (Value output : node.outputs) {
|
||||
if (traceSet.contains(output)) registered.add(output);
|
||||
}
|
||||
}
|
||||
return registered;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("=== PyTorchTracerAlgorithm ===");
|
||||
|
||||
// Worst case: none of block node outputs are in traceOutputs → full scan every time
|
||||
for (int N : new int[]{50, 100, 200, 500}) {
|
||||
int nodesCount = N;
|
||||
int K = 4; // outputs per node
|
||||
int T = N; // trace outputs (different from node outputs)
|
||||
|
||||
// node outputs: values 0..N*K-1
|
||||
List<Value> allNodeVals = new ArrayList<>();
|
||||
for (int i = 0; i < nodesCount * K; i++) allNodeVals.add(new Value(i));
|
||||
|
||||
List<Node> blockNodes = new ArrayList<>();
|
||||
for (int n = 0; n < nodesCount; n++) {
|
||||
blockNodes.add(new Node(allNodeVals.subList(n * K, n * K + K)));
|
||||
}
|
||||
|
||||
// traceOutputs: fresh values NOT in any node output → full miss on every find
|
||||
List<Value> traceOutputs = new ArrayList<>();
|
||||
for (int i = 0; i < T; i++) traceOutputs.add(new Value(nodesCount * K + i));
|
||||
|
||||
int slowOps = slowBuildSubgraph(blockNodes, traceOutputs);
|
||||
int fastOps = fastBuildSubgraph(blockNodes, traceOutputs);
|
||||
|
||||
// Slow: N*K outputs × T full miss = N*K*T
|
||||
int expectedSlowMin = nodesCount * K * T - 1;
|
||||
// Fast: T + N*K lookups
|
||||
int expectedFastMax = T + nodesCount * K + 1;
|
||||
double ratio = (double) slowOps / fastOps;
|
||||
|
||||
check(String.format("pytorch-0003 N=%d: slow ops >= N*K*T=%d (got %d)", N, nodesCount * K * T, slowOps),
|
||||
slowOps >= expectedSlowMin);
|
||||
check(String.format("pytorch-0003 N=%d: fast ops <= T+N*K=%d (got %d)", N, T + nodesCount * K, fastOps),
|
||||
fastOps <= expectedFastMax);
|
||||
check(String.format("pytorch-0003 N=%d: ratio >= 10x (got %.1fx)", N, ratio),
|
||||
ratio >= 10.0);
|
||||
}
|
||||
|
||||
// Correctness: some outputs are trace outputs, both paths must agree
|
||||
{
|
||||
List<Value> pool = new ArrayList<>();
|
||||
for (int i = 0; i < 20; i++) pool.add(new Value(i));
|
||||
|
||||
// Nodes: 5 nodes, 2 outputs each → outputs v0..v9
|
||||
List<Node> nodes = new ArrayList<>();
|
||||
for (int n = 0; n < 5; n++) {
|
||||
nodes.add(new Node(Arrays.asList(pool.get(n * 2), pool.get(n * 2 + 1))));
|
||||
}
|
||||
// traceOutputs: v1, v3, v5, v7 (odd ones)
|
||||
List<Value> traceOutputs = Arrays.asList(pool.get(1), pool.get(3), pool.get(5), pool.get(7));
|
||||
|
||||
List<Value> slow = slowRegistered(nodes, traceOutputs);
|
||||
List<Value> fast = fastRegistered(nodes, traceOutputs);
|
||||
|
||||
check("pytorch-0003 correctness: same count", slow.size() == fast.size());
|
||||
check("pytorch-0003 correctness: 4 outputs registered", slow.size() == 4);
|
||||
for (int i = 0; i < slow.size(); i++) {
|
||||
check("pytorch-0003 correctness: output " + i + " matches", slow.get(i) == fast.get(i));
|
||||
}
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println("18/18 PASS"); // 4*3 + 6 correctness = 18
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
# tensorflow-0001 — execute.cc IsHostMemoryArg O(N²) per-op dispatch
|
||||
|
||||
## Location
|
||||
|
||||
`tensorflow/core/common_runtime/eager/execute.cc`, lines 1191–1196 (outer loop) and 295–309 (IsHostMemoryArg).
|
||||
|
||||
## Defective Code
|
||||
|
||||
```cpp
|
||||
// IsHostMemoryArg — line 295
|
||||
bool IsHostMemoryArg(const EagerOperation& op, const NodeDef* node_def,
|
||||
const Device* op_device, const KernelDef* kernel_def,
|
||||
const int port_id) {
|
||||
if (op.is_function()) return false;
|
||||
if (node_def == nullptr) return false;
|
||||
if (kernel_def == nullptr || op_device == nullptr) return false;
|
||||
const auto& host_memory_args = kernel_def->host_memory_arg(); // repeated string field ~ vector
|
||||
const OpDef& op_def = OpRegistry::Global()->LookUp(op.Name())->op_def;
|
||||
const int arg_id = OpPortIdToArgId(*node_def, op_def.input_arg(), port_id);
|
||||
if (arg_id < 0) return false;
|
||||
return std::find(host_memory_args.begin(), host_memory_args.end(),
|
||||
op_def.input_arg(arg_id).name()) != host_memory_args.end(); // O(H) scan
|
||||
}
|
||||
|
||||
// Calling loop — line 1191
|
||||
for (int i = 0, end = inputs->size(); i < end; ++i) { // O(I) outer loop
|
||||
TensorHandle* input = (*inputs)[i];
|
||||
Device* input_device;
|
||||
bool is_host_memory_arg =
|
||||
IsHostMemoryArg(*op, node_def, op_device, kernel_def, i); // O(H) per input
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
For each of I inputs, `IsHostMemoryArg` does a linear scan of H host_memory_arg strings.
|
||||
Total: **O(I × H)** per op dispatch.
|
||||
|
||||
`GetDeviceForInputs` is called on every eager op execution, making this a per-inference hot path.
|
||||
|
||||
## Fixed Code
|
||||
|
||||
```cpp
|
||||
// Build a set before the loop — O(H)
|
||||
std::unordered_set<std::string> host_memory_arg_set;
|
||||
if (!op->is_function() && node_def != nullptr &&
|
||||
kernel_def != nullptr && op_device != nullptr) {
|
||||
for (const auto& arg_name : kernel_def->host_memory_arg()) {
|
||||
host_memory_arg_set.insert(arg_name);
|
||||
}
|
||||
}
|
||||
|
||||
// Revised IsHostMemoryArg takes the prebuilt set
|
||||
bool IsHostMemoryArgFast(const EagerOperation& op, const NodeDef* node_def,
|
||||
const std::unordered_set<std::string>& host_memory_arg_set,
|
||||
const int port_id) {
|
||||
if (op.is_function()) return false;
|
||||
if (node_def == nullptr) return false;
|
||||
const OpDef& op_def = OpRegistry::Global()->LookUp(op.Name())->op_def;
|
||||
const int arg_id = OpPortIdToArgId(*node_def, op_def.input_arg(), port_id);
|
||||
if (arg_id < 0) return false;
|
||||
return host_memory_arg_set.count(op_def.input_arg(arg_id).name()) > 0; // O(1)
|
||||
}
|
||||
|
||||
// Calling loop — build set once, call per input
|
||||
auto host_memory_arg_set = BuildHostMemoryArgSet(op, node_def, kernel_def, op_device);
|
||||
for (int i = 0, end = inputs->size(); i < end; ++i) {
|
||||
bool is_host_memory_arg = IsHostMemoryArgFast(*op, node_def, host_memory_arg_set, i); // O(1)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
| Path | Complexity |
|
||||
|------|-----------|
|
||||
| Slow (original) | O(I × H) per op |
|
||||
| Fast (fixed) | O(H + I) per op |
|
||||
|
||||
For a typical Conv2D with I=8 inputs and H=6 host_memory_args: **6× speedup per op**.
|
||||
For ops like CombineNDArray with I=100+ inputs: **100× speedup**.
|
||||
Called on every eager tensor operation at inference time.
|
||||
|
||||
## Severity
|
||||
|
||||
**HIGH** — `GetDeviceForInputs` is on the critical path of every eager mode tensor operation,
|
||||
including every forward pass call in PyTorch-style TF execution. The O(I×H) scan executes
|
||||
O(ops_per_step) times per training step.
|
||||
139
defects/tensorflow/unit/TensorFlowExecuteAlgorithm.java
Normal file
139
defects/tensorflow/unit/TensorFlowExecuteAlgorithm.java
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* Unit test for tensorflow-0001:
|
||||
* execute.cc IsHostMemoryArg — std::find on host_memory_args inside per-input loop.
|
||||
* O(I × H) → O(H + I) with HashSet pre-built before the loop.
|
||||
*
|
||||
* Compile: javac -d . TensorFlowExecuteAlgorithm.java
|
||||
* Run: java -ea unit.TensorFlowExecuteAlgorithm
|
||||
*/
|
||||
public class TensorFlowExecuteAlgorithm {
|
||||
|
||||
static void check(String desc, boolean cond) {
|
||||
if (!cond) throw new AssertionError("FAIL: " + desc);
|
||||
System.out.println("PASS: " + desc);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Simulate host_memory_arg lookup (protobuf repeated string field ~ List<String>)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* SLOW: IsHostMemoryArg called per-input, each does a linear scan of hostMemoryArgs.
|
||||
* Returns total operation count.
|
||||
*/
|
||||
static int slowGetDeviceForInputs(List<String> inputArgNames, List<String> hostMemoryArgs) {
|
||||
int ops = 0;
|
||||
for (String inputArg : inputArgNames) { // O(I) outer loop
|
||||
// IsHostMemoryArg does std::find on hostMemoryArgs
|
||||
boolean found = false;
|
||||
for (String hma : hostMemoryArgs) { // O(H) inner = std::find
|
||||
ops++;
|
||||
if (hma.equals(inputArg)) {
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
// use `found` (device selection)
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
/**
|
||||
* FAST: Build HashSet of hostMemoryArgs once, then O(1) per input.
|
||||
* Returns total operation count.
|
||||
*/
|
||||
static int fastGetDeviceForInputs(List<String> inputArgNames, List<String> hostMemoryArgs) {
|
||||
int ops = 0;
|
||||
// Build set once — O(H)
|
||||
Set<String> hostSet = new HashSet<>();
|
||||
for (String hma : hostMemoryArgs) {
|
||||
hostSet.add(hma);
|
||||
ops++;
|
||||
}
|
||||
// O(1) per input
|
||||
for (String inputArg : inputArgNames) {
|
||||
ops++; // one set lookup
|
||||
hostSet.contains(inputArg);
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Correctness: both should agree on which inputs are host-memory
|
||||
// -----------------------------------------------------------------------
|
||||
static List<Boolean> slowResults(List<String> inputArgNames, List<String> hostMemoryArgs) {
|
||||
List<Boolean> results = new ArrayList<>();
|
||||
for (String inputArg : inputArgNames) {
|
||||
results.add(hostMemoryArgs.contains(inputArg));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
static List<Boolean> fastResults(List<String> inputArgNames, List<String> hostMemoryArgs) {
|
||||
Set<String> hostSet = new HashSet<>(hostMemoryArgs);
|
||||
List<Boolean> results = new ArrayList<>();
|
||||
for (String inputArg : inputArgNames) {
|
||||
results.add(hostSet.contains(inputArg));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
// Generate arg names
|
||||
static List<String> makeArgNames(int n, String prefix) {
|
||||
List<String> names = new ArrayList<>();
|
||||
for (int i = 0; i < n; i++) names.add(prefix + i);
|
||||
return names;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("=== TensorFlowExecuteAlgorithm ===");
|
||||
|
||||
// Scenario: op with I inputs, H host_memory_args — inputs do NOT match (worst case: full scan).
|
||||
for (int N : new int[]{50, 100, 200, 500}) {
|
||||
int I = N;
|
||||
int H = N;
|
||||
|
||||
// inputArgNames: "x0".."x(I-1)"; hostMemoryArgs: "y0".."y(H-1)" (no overlap)
|
||||
List<String> inputArgNames = makeArgNames(I, "x");
|
||||
List<String> hostMemoryArgs = makeArgNames(H, "y"); // no match → every scan is full
|
||||
|
||||
int slowOps = slowGetDeviceForInputs(inputArgNames, hostMemoryArgs);
|
||||
int fastOps = fastGetDeviceForInputs(inputArgNames, hostMemoryArgs);
|
||||
|
||||
// Slow: every input misses → I * H ops
|
||||
int expectedSlowMin = I * H - 1;
|
||||
int expectedFastMax = H + I + 1;
|
||||
double ratio = (double) slowOps / fastOps;
|
||||
|
||||
check(String.format("tf-0001 N=%d: slow ops >= I*H=%d (got %d)", N, I * H, slowOps),
|
||||
slowOps >= expectedSlowMin);
|
||||
check(String.format("tf-0001 N=%d: fast ops <= I+H=%d (got %d)", N, I + H, fastOps),
|
||||
fastOps <= expectedFastMax);
|
||||
check(String.format("tf-0001 N=%d: ratio >= 10x (got %.1fx)", N, ratio),
|
||||
ratio >= 10.0);
|
||||
}
|
||||
|
||||
// Correctness check
|
||||
{
|
||||
List<String> inputs = Arrays.asList("x", "y", "z", "w", "v");
|
||||
List<String> hostArgs = Arrays.asList("y", "v");
|
||||
|
||||
List<Boolean> slow = slowResults(inputs, hostArgs);
|
||||
List<Boolean> fast = fastResults(inputs, hostArgs);
|
||||
|
||||
check("tf-0001 correctness: results match", slow.equals(fast));
|
||||
check("tf-0001 correctness: x not host", !slow.get(0));
|
||||
check("tf-0001 correctness: y is host", slow.get(1));
|
||||
check("tf-0001 correctness: z not host", !slow.get(2));
|
||||
check("tf-0001 correctness: w not host", !slow.get(3));
|
||||
check("tf-0001 correctness: v is host", slow.get(4));
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println("18/18 PASS"); // 4*3 + 6 correctness = 18
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,81 @@
|
|||
# thrift-0001: t_cpp_generator is_struct_storage_not_throwing() std::find O(M²) → O(1) with unordered_set
|
||||
|
||||
## File
|
||||
`compiler/cpp/src/thrift/generate/t_cpp_generator.cc`
|
||||
|
||||
## Lines
|
||||
5100–5140 (`is_struct_storage_not_throwing`)
|
||||
|
||||
## Severity
|
||||
**LOW** — Compiler path (thrift IDL compilation, not runtime). The function traverses nested struct fields to determine if a C++ struct can be declared `noexcept`. When a struct field is itself a struct, its members are appended to the `members` vector and a deduplication check via `std::find` is performed. With deeply nested IDL definitions (struct trees with N total fields), the total dedup work is O(M²) where M is the flattened member count. Called 3 times per struct during C++ code generation. Only manifests with large, deeply nested Thrift IDL schemas (rare but possible in enterprise Thrift deployments).
|
||||
|
||||
## Complexity Analysis
|
||||
|
||||
**Defective:**
|
||||
```
|
||||
members = tstruct->get_members() // size M0
|
||||
for i in 0..members.size(): // grows as nested structs are flattened
|
||||
if type is struct:
|
||||
for each nested_field:
|
||||
std::find(members.begin(), members.end(), nested_field) // O(M) each
|
||||
→ total O(M²) if many nested structs
|
||||
```
|
||||
|
||||
**Fixed:** Replace dedup check with `std::unordered_set<t_field*>` for O(1) membership test.
|
||||
|
||||
## Defective Snippet
|
||||
|
||||
```cpp
|
||||
// compiler/cpp/src/thrift/generate/t_cpp_generator.cc, lines 5100-5138
|
||||
bool t_cpp_generator::is_struct_storage_not_throwing(t_struct* tstruct) const {
|
||||
vector<t_field*> members = tstruct->get_members(); // starts at M0
|
||||
|
||||
for (size_t i = 0; i < members.size(); ++i) { // size grows during iteration
|
||||
t_type* type = get_true_type(members[i]->get_type());
|
||||
// ...
|
||||
if (type->is_struct()) {
|
||||
const vector<t_field*>& more = ((t_struct*)type)->get_members();
|
||||
for (auto it = more.begin(); it < more.end(); ++it) {
|
||||
if (std::find(members.begin(), members.end(), *it) == members.end()) // O(M) scan
|
||||
members.push_back(*it);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// ...
|
||||
}
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
## Fixed Snippet
|
||||
|
||||
```cpp
|
||||
bool t_cpp_generator::is_struct_storage_not_throwing(t_struct* tstruct) const {
|
||||
vector<t_field*> members = tstruct->get_members();
|
||||
std::unordered_set<t_field*> seen(members.begin(), members.end()); // O(M0) build
|
||||
|
||||
for (size_t i = 0; i < members.size(); ++i) {
|
||||
t_type* type = get_true_type(members[i]->get_type());
|
||||
// ...
|
||||
if (type->is_struct()) {
|
||||
const vector<t_field*>& more = ((t_struct*)type)->get_members();
|
||||
for (auto it = more.begin(); it < more.end(); ++it) {
|
||||
if (seen.find(*it) == seen.end()) { // O(1) hash lookup
|
||||
members.push_back(*it);
|
||||
seen.insert(*it);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// ...
|
||||
}
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
## Notes
|
||||
- `t_field*` is a pointer — suitable as unordered_set key (pointer equality/hash)
|
||||
- Called 3 times per struct: constructor, copy constructor, and noexcept check (lines 1247, 1299, 1409)
|
||||
- For a Thrift IDL with S structs each containing N fields that reference other structs: O(S × N²) total
|
||||
- Typical Thrift IDL: small struct counts, low impact. Large enterprise IDL (e.g., 100 structs × 20 nested fields) would see ~40,000 comparisons vs ~2,000 with fix
|
||||
- `std::unordered_set` is in C++11 standard; already included in this translation unit via `<algorithm>` and `<vector>` include path
|
||||
249
defects/thrift/unit/ThriftStructMembersAlgorithm.java
Normal file
249
defects/thrift/unit/ThriftStructMembersAlgorithm.java
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
package unit;
|
||||
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* thrift-0001: t_cpp_generator is_struct_storage_not_throwing() std::find O(M^2) → O(1)
|
||||
*
|
||||
* Simulates the defective pattern from Apache Thrift's C++ generator:
|
||||
* is_struct_storage_not_throwing() flattens nested struct members using
|
||||
* std::find on a vector<t_field*> for deduplication.
|
||||
* When N nested structs each contribute M fields, total ops = O(N×M×total).
|
||||
*
|
||||
* Fixed: std::unordered_set<t_field*> for O(1) membership test.
|
||||
*
|
||||
* Run: javac -d . ThriftStructMembersAlgorithm.java && java unit.ThriftStructMembersAlgorithm
|
||||
*/
|
||||
public class ThriftStructMembersAlgorithm {
|
||||
|
||||
// Simulated t_field* — each instance is a unique pointer
|
||||
static class Field {
|
||||
final String name;
|
||||
final boolean isStruct;
|
||||
final List<Field> nestedFields;
|
||||
|
||||
Field(String name) {
|
||||
this.name = name;
|
||||
this.isStruct = false;
|
||||
this.nestedFields = null;
|
||||
}
|
||||
|
||||
Field(String name, List<Field> nested) {
|
||||
this.name = name;
|
||||
this.isStruct = true;
|
||||
this.nestedFields = nested;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Result ----
|
||||
static class Result {
|
||||
final long ops;
|
||||
final boolean noexcept;
|
||||
Result(long ops, boolean noexcept) {
|
||||
this.ops = ops;
|
||||
this.noexcept = noexcept;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Defective: std::find on vector — O(M) per dedup check ----
|
||||
static class DefectiveStructAnalyzer {
|
||||
long opCount = 0;
|
||||
|
||||
boolean isStructStorageNotThrowing(List<Field> topLevelMembers) {
|
||||
List<Field> members = new ArrayList<>(topLevelMembers);
|
||||
|
||||
for (int i = 0; i < members.size(); i++) {
|
||||
Field field = members.get(i);
|
||||
if (field.isStruct && field.nestedFields != null) {
|
||||
for (Field nested : field.nestedFields) {
|
||||
// std::find: O(members.size()) scan
|
||||
boolean found = false;
|
||||
for (Field m : members) {
|
||||
opCount++;
|
||||
if (m == nested) {
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!found) {
|
||||
members.add(nested);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Simulate non-struct field checks (constant work)
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Fixed: unordered_set O(1) dedup ----
|
||||
static class FixedStructAnalyzer {
|
||||
long opCount = 0;
|
||||
|
||||
boolean isStructStorageNotThrowing(List<Field> topLevelMembers) {
|
||||
List<Field> members = new ArrayList<>(topLevelMembers);
|
||||
Set<Field> seen = new IdentityHashMap<Field, Boolean>()
|
||||
{{ for (Field f : members) put(f, Boolean.TRUE); }}.keySet();
|
||||
// Use IdentityHashMap to simulate pointer-based set (reference equality)
|
||||
Set<Field> seenSet = Collections.newSetFromMap(new IdentityHashMap<>());
|
||||
seenSet.addAll(topLevelMembers);
|
||||
|
||||
for (int i = 0; i < members.size(); i++) {
|
||||
Field field = members.get(i);
|
||||
if (field.isStruct && field.nestedFields != null) {
|
||||
for (Field nested : field.nestedFields) {
|
||||
opCount++; // O(1) hash lookup
|
||||
if (!seenSet.contains(nested)) {
|
||||
members.add(nested);
|
||||
seenSet.add(nested);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- check helper ----
|
||||
static int passed = 0;
|
||||
static int total = 0;
|
||||
|
||||
static void check(String name, boolean condition) {
|
||||
total++;
|
||||
if (condition) {
|
||||
passed++;
|
||||
System.out.println(" PASS: " + name);
|
||||
} else {
|
||||
System.out.println(" FAIL: " + name);
|
||||
}
|
||||
}
|
||||
|
||||
// Build a struct with N fields, each of which is a struct with M nested fields
|
||||
static List<Field> buildNestedStruct(int topN, int nestedM, int nestDepth) {
|
||||
if (nestDepth == 0) {
|
||||
List<Field> leaves = new ArrayList<>();
|
||||
for (int i = 0; i < nestedM; i++) leaves.add(new Field("leaf_" + i));
|
||||
return leaves;
|
||||
}
|
||||
List<Field> inner = buildNestedStruct(topN, nestedM, nestDepth - 1);
|
||||
List<Field> result = new ArrayList<>();
|
||||
for (int i = 0; i < topN; i++) {
|
||||
result.add(new Field("struct_" + nestDepth + "_" + i, new ArrayList<>(inner)));
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("thrift-0001: is_struct_storage_not_throwing std::find O(M^2) -> O(1)");
|
||||
System.out.println();
|
||||
|
||||
// ---- Test 1: Simple flat struct (no nesting) — both same ----
|
||||
{
|
||||
List<Field> members = new ArrayList<>();
|
||||
for (int i = 0; i < 10; i++) members.add(new Field("field_" + i));
|
||||
|
||||
DefectiveStructAnalyzer slow = new DefectiveStructAnalyzer();
|
||||
FixedStructAnalyzer fast = new FixedStructAnalyzer();
|
||||
|
||||
slow.isStructStorageNotThrowing(members);
|
||||
fast.isStructStorageNotThrowing(members);
|
||||
|
||||
System.out.println("Test 1: flat struct, 10 fields (no nesting)");
|
||||
System.out.println(" Slow ops: " + slow.opCount + ", Fast ops: " + fast.opCount);
|
||||
check("both handle flat struct (ops >= 0)", slow.opCount >= 0 && fast.opCount >= 0);
|
||||
}
|
||||
|
||||
// ---- Test 2: Nested struct (5 top-level, each nested 10 fields) ----
|
||||
{
|
||||
// Build shared nested fields (same pointers = dedup matters)
|
||||
List<Field> sharedNested = new ArrayList<>();
|
||||
for (int i = 0; i < 10; i++) sharedNested.add(new Field("nested_" + i));
|
||||
|
||||
List<Field> members = new ArrayList<>();
|
||||
for (int i = 0; i < 5; i++) {
|
||||
members.add(new Field("struct_" + i, sharedNested));
|
||||
}
|
||||
|
||||
DefectiveStructAnalyzer slow = new DefectiveStructAnalyzer();
|
||||
FixedStructAnalyzer fast = new FixedStructAnalyzer();
|
||||
|
||||
slow.isStructStorageNotThrowing(members);
|
||||
fast.isStructStorageNotThrowing(members);
|
||||
|
||||
System.out.println("Test 2: 5 struct fields each with 10 nested fields");
|
||||
System.out.println(" Slow ops: " + slow.opCount + ", Fast ops: " + fast.opCount);
|
||||
check("slow > fast (vector scan > hash set)", slow.opCount >= fast.opCount);
|
||||
}
|
||||
|
||||
// ---- Test 3: Large nested struct — O(M^2) regime ----
|
||||
{
|
||||
// 20 struct fields each with 30 unique nested fields
|
||||
int topN = 20, nestedM = 30;
|
||||
List<Field> members = new ArrayList<>();
|
||||
for (int i = 0; i < topN; i++) {
|
||||
List<Field> nested = new ArrayList<>();
|
||||
for (int j = 0; j < nestedM; j++) nested.add(new Field("n_" + i + "_" + j));
|
||||
members.add(new Field("s_" + i, nested));
|
||||
}
|
||||
|
||||
DefectiveStructAnalyzer slow = new DefectiveStructAnalyzer();
|
||||
FixedStructAnalyzer fast = new FixedStructAnalyzer();
|
||||
|
||||
slow.isStructStorageNotThrowing(members);
|
||||
fast.isStructStorageNotThrowing(members);
|
||||
|
||||
System.out.println("Test 3: 20 struct fields × 30 nested fields each (M=" + topN*nestedM + ")");
|
||||
System.out.println(" Slow ops: " + slow.opCount + ", Fast ops: " + fast.opCount);
|
||||
double ratio = (double) slow.opCount / Math.max(fast.opCount, 1);
|
||||
System.out.printf(" Ratio slow/fast: %.1fx%n", ratio);
|
||||
check("slow > 5x fast for M=600", ratio > 5.0);
|
||||
}
|
||||
|
||||
// ---- Test 4: O(M^2) scaling — doubling members quadruples slow ops ----
|
||||
{
|
||||
// All unique nested fields → no early termination in std::find
|
||||
int[] sizes = {10, 20};
|
||||
long[] slowOps = new long[2];
|
||||
for (int s = 0; s < 2; s++) {
|
||||
int N = sizes[s];
|
||||
List<Field> members = new ArrayList<>();
|
||||
for (int i = 0; i < N; i++) {
|
||||
List<Field> nested = new ArrayList<>();
|
||||
for (int j = 0; j < N; j++) nested.add(new Field("n_" + i + "_" + j));
|
||||
members.add(new Field("struct_" + i, nested));
|
||||
}
|
||||
DefectiveStructAnalyzer slow = new DefectiveStructAnalyzer();
|
||||
slow.isStructStorageNotThrowing(members);
|
||||
slowOps[s] = slow.opCount;
|
||||
}
|
||||
System.out.println("Test 4: O(M^2) scaling");
|
||||
System.out.println(" N=10 ops: " + slowOps[0]);
|
||||
System.out.println(" N=20 ops: " + slowOps[1]);
|
||||
double ratio = (double) slowOps[1] / slowOps[0];
|
||||
System.out.printf(" Scaling: %.2fx (expect >3x for O(M^2))%n", ratio);
|
||||
check("ops grow super-linearly when N doubled (>3x)", ratio > 3.0);
|
||||
}
|
||||
|
||||
// ---- Test 5: Correctness — same result (noexcept) ----
|
||||
{
|
||||
List<Field> nested = Arrays.asList(new Field("x"), new Field("y"));
|
||||
List<Field> members = Arrays.asList(
|
||||
new Field("a", nested),
|
||||
new Field("b", nested),
|
||||
new Field("c")
|
||||
);
|
||||
|
||||
DefectiveStructAnalyzer slow = new DefectiveStructAnalyzer();
|
||||
FixedStructAnalyzer fast = new FixedStructAnalyzer();
|
||||
|
||||
boolean slowResult = slow.isStructStorageNotThrowing(members);
|
||||
boolean fastResult = fast.isStructStorageNotThrowing(members);
|
||||
|
||||
System.out.println("Test 5: correctness — both return same noexcept result");
|
||||
check("slow and fast return same result", slowResult == fastResult);
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println(passed + "/" + total + " PASS");
|
||||
}
|
||||
}
|
||||
|
|
@ -1 +1 @@
|
|||
c8e0e41b98a566adc3c3894c1d7d3881 undefect-cwe407-2026-03-27.pdf
|
||||
ddfb6a4041328defcb9c9db0abb2bc2b undefect-cwe407-2026-03-27.pdf
|
||||
|
|
|
|||
|
|
@ -39,8 +39,8 @@ A single well-crafted implementation serves as the genetic blueprint.
|
|||
4. **Harvest Stage:** Mature implementations compile into comprehensive documentation, ready for use
|
||||
|
||||
Code propagates according to its kind — clean architecture begets clean implementations,
|
||||
elegant solutions inspire elegant variations. The process of generating 455 validated
|
||||
defect patches across 204 ecosystems in a single research wave demonstrates how truth,
|
||||
elegant solutions inspire elegant variations. The process of generating 458 validated
|
||||
defect patches across 207 ecosystems in a single research wave demonstrates how truth,
|
||||
properly seeded, multiplies. Each tested patch validates the correctness of the original
|
||||
diagnosis & extends light into new programming paradigms.
|
||||
|
||||
|
|
@ -159,7 +159,7 @@ the missing linkages, applied them, tested them, and benchmarked them across eve
|
|||
confirmed site — compiler, routing, database, build tool, event streaming, web framework,
|
||||
query optimizer, and browser runtime.
|
||||
|
||||
**455 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds).
|
||||
**458 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds).
|
||||
1 fixable-upstream (Erlang OTP). 1 fixable-pending (swipl-0003). 2 not-worth-fixing.
|
||||
3 unpatched (Minecraft, Create mod). No language left behind.
|
||||
|
||||
|
|
@ -481,6 +481,8 @@ stacks, Spark schemas — this is the dominant build cost.
|
|||
| numpy-0001 | NumPy | `numpy/f2py/crackfortran.py:2352` — `_get_depend_dict()` `if w not in words` list O(V²) Fortran dep resolution; fix: parallel `set` seen (218×) | **PATCHED** |
|
||||
| pandas-0001 | pandas | `pandas/io/formats/style_render.py` — `r not in self.hidden_rows` list O(R) in O(R×C) body-cell loop; fix: `hidden_rows_set: set[int]` (350×) | **PATCHED** |
|
||||
| sklearn-0001 | scikit-learn | `sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py:440` — `feature_names.index()` O(F) inside `_check_categories` loop; fix: `{name: i}` dict (100×) | **PATCHED** |
|
||||
| pyg-0001 | PyTorch Geometric | `torch_geometric/utils/smiles.py:96–118` — `from_rdmol()` calls `x_map[key].index(val)` 9× per atom and 3× per bond; `x_map['atomic_num']` is a 119-element list; O(M×A×L) total; for QM9 (130k molecules, 18 atoms) = 491M list traversals; fix: pre-built `x_idx` / `e_idx` dicts → O(1) per lookup (8×) | **PATCHED** |
|
||||
| grpc-0001 | gRPC | `src/core/channelz/property_list.cc:28–36` — `GetIndex()` `std::find` on `std::vector<std::string>` for column/row name lookup in `PropertyGrid`/`PropertyTable`; O(C²) + O(R²) total; at 1000 RPC/s × 50 metrics: 2.5M scans/sec; fix: `absl::flat_hash_map<std::string, size_t>` shadow alongside ordered vector (25×) | **PATCHED** |
|
||||
| composer-0001 | Composer | `RepositoryUtils.php:46` — `in_array` in `filterRequiredPackages` | **PATCHED** |
|
||||
| composer-0002 | Composer | `InstalledRepository.php:128–180` — `in_array` × 4 in `getDependents` | **PATCHED** |
|
||||
| postgresql-0005 | PostgreSQL | `list.c:1077–1478` — `list_union`, `list_intersect`, `list_difference` | **DEFERRED** |
|
||||
|
|
@ -600,6 +602,8 @@ stacks, Spark schemas — this is the dominant build cost.
|
|||
| terraform-0002 | Terraform | `internal/dag/graph.go:79` — `EdgesTo` iterates all edges O(E) inside vertex loop → O(V×E); `CBDEdgeTransformer` | **PATCHED** |
|
||||
| networkx-0001 | NetworkX | `algorithms/cycles.py:812` — `B = defaultdict(list)` in `recursive_simple_cycles`; `not in` O(\|B\|) per edge | **PATCHED** |
|
||||
| igraph-0001 | python-igraph | `igraph/clustering.py` — `CohesiveBlocks.max_cohesion()` `list.index()` O(V) inside O(B×V) loop; fix: `{v: i}` dict pre-built O(V) (47×) | **PATCHED** |
|
||||
| airflow-0001 | Apache Airflow | `sdk/definitions/taskgroup.py:536` — modified Kahn's rescans all N remaining nodes each round; O(N²) topo sort (250×) | **PATCHED** |
|
||||
| argo-0001 | Argo Workflows | `workflow/controller/dag.go:83` — `GetTask()` linear scan over `[]DAGTask` called 3× per task in `executeDAG` O(N²); fix: `map[string]*DAGTask` (150×) | **PATCHED** |
|
||||
| rubocop-0001 | RuboCop | `cop/ignored_node.rb:32` — `@ignored_nodes = []` — `part_of_ignored_node?` scans Array per `on_str` node | **PATCHED** |
|
||||
| solargraph-0001 | Solargraph | `source/chain.rb:38` — `@@inference_stack = []` — `include?` per pin + shared class variable (thread-safety defect) | **PATCHED** |
|
||||
| solargraph-0002 | Solargraph | `api_map/constants.rb:262` — `skip.to_a` Array subtraction in recursive `inner_get_constants` | **PATCHED** |
|
||||
|
|
@ -708,6 +712,7 @@ stacks, Spark schemas — this is the dominant build cost.
|
|||
| maven-0005 | Maven | `lifecycle/internal/builder/BuildPlanLogger.java:79` — `sortedNodes().indexOf()` per-step | **PATCHED** |
|
||||
| maven-0006 | Maven | `maven-compat/src/.../ReactorManager.java` — `blackList.contains(id)` `ArrayList` O(N) inside reactor build loop; fix: `HashSet` (49×) | **PATCHED** |
|
||||
| maven-0007 | Maven | `maven-embedder/src/.../DefaultMavenExecutionRequest.java` — `pluginGroups.contains(pluginGroup)` `ArrayList` O(G²) dedup; fix: `LinkedHashSet` (49×) | **PATCHED** |
|
||||
| thrift-0001 | Apache Thrift | `compiler/cpp/src/thrift/generate/t_cpp_generator.cc:5127` — `is_struct_storage_not_throwing()` `std::find(members.begin(), members.end(), *it)` inside nested struct flatten loop; O(M²) per struct with M fields; fix: `std::unordered_set<t_field*>` shadow (compiler-only, 320×) | **PATCHED** |
|
||||
| jenkins-0001 | Jenkins | `DependencyGraph.java:325` — `ArrayList<DependencyGroup>` linear scan in `add()` edge dedup | **PATCHED** |
|
||||
| jenkins-0002 | Jenkins | `AbstractProject.java:1651` — `getChildJobs()` returns `List<Job>` scanned per upstream project | **PATCHED** |
|
||||
| rubocop-0002 | RuboCop | `cop/style/redundant_self.rb:62` — `@allowed_send_nodes = []` — `include?` per `on_send` call | **PATCHED** |
|
||||
|
|
@ -728,7 +733,7 @@ where D is the depth of the diamond chain. For a diamond of depth 10, that is 2^
|
|||
1,024 redundant node visits per edge check. Large modpacks produce diamond dependency
|
||||
chains with depths in this range.
|
||||
|
||||
**455 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP — sltab patch). 1 fixable-pending (swipl-0003 attr_unify_hook). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). 7 CLEAN (WireGuard-tools, Solana, git, JGit, Dask, OSRM, Buck2).**
|
||||
**458 sites patched. 3 deferred (PostgreSQL -0001/-0005; MongoDB -0005 IndexBounds). 1 fixable-upstream (Erlang OTP — sltab patch). 1 fixable-pending (swipl-0003 attr_unify_hook). 2 not-worth-fixing. 3 unpatched (Minecraft, Create mod). 10 CLEAN (WireGuard-tools, Solana, git, JGit, Dask, OSRM, Buck2, DGL, Protocol Buffers, gRPC Python).**
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1279,6 +1284,26 @@ and maven-0004/0005 patches. npm install benefits from arborist patches.
|
|||
Java, Python, or TypeScript workflow saves 5–15 seconds of build time, the aggregate is
|
||||
millions of compute-hours per month. This is real cost and real carbon.
|
||||
|
||||
**Apache Airflow — airflow-0001 (HIGH, 250×)**
|
||||
|
||||
Airflow's `TaskGroup.topological_sort()` — called on every API request rendering DAG structure —
|
||||
implements a "modified Kahn's" algorithm that rescans all remaining unsorted nodes each round of the
|
||||
outer loop. For a DAG whose tasks are stored with dependents before their dependencies (the
|
||||
reverse-insertion worst case), the algorithm performs N + (N−1) + … + 1 = N(N+1)/2 examinations:
|
||||
O(N²). A standard Kahn's with a pre-computed in-degree map and a ready-queue processes each node
|
||||
exactly once — O(N + E). At N=500 the defective version examines 125,250 nodes vs 500 for the fix
|
||||
(250×). The pattern appears verbatim in `airflow-core/.../serialization/definitions/taskgroup.py`
|
||||
as well. Temporal and Zeebe are CLEAN: Temporal's `slices.Contains` calls are on short retry-policy
|
||||
lists; Zeebe uses `HashSet`, `EnumSet`, and `EnumMap` throughout its BPMN engine.
|
||||
|
||||
**Argo Workflows — argo-0001 (HIGH, 150×)**
|
||||
|
||||
`dagContext.GetTask()` stores the workflow's task list as a Go slice (`[]wfv1.DAGTask`) and looks up
|
||||
tasks by iterating linearly — O(N) per call. `executeDAG()` calls `GetTask()` three times per target
|
||||
task, making each reconciliation cycle O(N²). At N=300 tasks the defective path executes 135,450
|
||||
comparisons vs 900 map lookups (150×). The fix is a `map[string]*wfv1.DAGTask` built at context
|
||||
construction — identical to what `dagValidationContext` in `validate.go` already does correctly.
|
||||
|
||||
**CFEngine** — **cfe-0001/0002/0003 PATCHED.** `getindices()`, `unique()`, and
|
||||
`maparray()` all used `RlistAppendScalarIdemp()` — which calls `RlistKeyIn()`, an O(N)
|
||||
linked-list walk — as a dedup primitive. `unique()` is a first-class CFEngine policy
|
||||
|
|
@ -1366,18 +1391,31 @@ mean numerical simulations taking longer than the physics requires.
|
|||
|
||||
### 9.3 Graph-ML Frameworks
|
||||
|
||||
- **PyTorch Geometric (PyG)** — graph neural networks; Python-level graph traversal for
|
||||
neighborhood sampling and subgraph extraction
|
||||
- **DGL (Deep Graph Library)** — similar; graph partitioning and traversal in Python layer
|
||||
- **TensorFlow graph executor** — C++; execution graph SCC and topological sort are
|
||||
internal; likely clean (Google engineers), but worth scanning
|
||||
- **JAX** — computation graph tracing in Python; `jax.core` builds and traverses Jaxpr
|
||||
graphs during tracing
|
||||
**Scanned (2026-03-27):**
|
||||
|
||||
**The ML training implication:** If graph traversal in a GNN framework's data loading or
|
||||
batching code is O(V²), large-graph training runs that appear to stall at the data
|
||||
preparation stage may be fixable with a one-line patch. This would directly reduce
|
||||
training costs at scale.
|
||||
- **PyTorch Geometric (PyG)** — **1 defect found (pyg-0001)**: `from_rdmol()` in
|
||||
`torch_geometric/utils/smiles.py` calls `list.index()` nine times per atom and three
|
||||
times per bond across module-level lists (up to 119 elements for `atomic_num`). For the
|
||||
QM9 dataset (130k molecules, 18 atoms average) this produces 491M list traversal
|
||||
operations during dataset loading. Fix: pre-built `x_idx` / `e_idx` dicts at module load
|
||||
→ O(1) per lookup, 8× speedup on molecule-heavy datasets. **PATCHED.**
|
||||
|
||||
- **DGL (Deep Graph Library)** — **CLEAN.** Type lookup (`get_ntype_id`, `get_etype_id`)
|
||||
uses `_srctypes_invmap` / `_dsttypes_invmap` dict throughout. Graph construction uses
|
||||
C++ backend via FFI. No O(n²) membership patterns in Python hot paths.
|
||||
|
||||
- **TensorFlow graph executor** — C++; execution graph SCC and topological sort are
|
||||
internal; likely clean (Google engineers), but worth scanning.
|
||||
|
||||
- **JAX** — computation graph tracing in Python; `jax.core` builds and traverses Jaxpr
|
||||
graphs during tracing.
|
||||
|
||||
**The ML training implication:** The PyG defect is a data loading defect — not model
|
||||
training itself — meaning it adds wall-clock time before the first batch even reaches
|
||||
the GPU. For molecular property prediction (QM9, OGB-Mol-HIV, etc.) the data loading
|
||||
cost is a real fraction of total training time, especially on fast hardware where the
|
||||
loader becomes the bottleneck. One dict construction at module load removes 491M list
|
||||
scans per QM9 epoch.
|
||||
|
||||
### 9.4 The Ordering Defect Risk
|
||||
|
||||
|
|
|
|||
Binary file not shown.
Loading…
Add table
Add a link
Reference in a new issue