4.3 KiB
Optuna — CWE-407 Disclosure Brief
Project: Optuna Disclosure date: 2026-03-27 Severity: HIGH Speedup: significant (O(N³) → O(N²)) Status: PATCHED
Finding
Optuna's multi-objective non-domination ranking algorithm _calculate_nondomination_rank() uses a nested structure where an outer while loop iterates over Pareto fronts F, and for each front calls _is_pareto_front_nd() which performs an O(N²) pairwise dominance check. In the worst case — when all N trials occupy distinct Pareto fronts — this yields O(N³) total complexity. The fix is a single O(N²) dominance graph construction followed by O(N+E) Kahn extraction.
The Defect(s)
| ID | Location | Pattern | Complexity |
|---|---|---|---|
| optuna-0001 | optuna/study/_multi_objective.py:187 |
Outer while per front F × inner _is_pareto_front_nd() O(N²) pairwise check; worst case O(N³) when F=N |
O(N³) worst case |
Complexity Proof
Let N = number of trials, F = number of Pareto fronts (1 ≤ F ≤ N).
_is_pareto_front_nd(trials, population) checks whether trials forms a Pareto front relative to population by testing all pairs for dominance. For a set of size K checked against population of size P, this is O(K×P).
The outer loop in _calculate_nondomination_rank() extracts one front per iteration:
Iteration 1: check N trials against N population → O(N²), extracts front of size k₁
Iteration 2: check (N-k₁) against (N-k₁) → O((N-k₁)²)
...
Iteration F: check remaining trials → O(kF²)
In the worst case (each trial in its own front, F=N, k_i=1 for all i):
Sum_{i=1}^{N} (N - i + 1)² ≈ N³/3 = O(N³)
In the average case with F fronts of roughly equal size N/F each:
F × O((N/F)²) = O(N²/F)
which is still O(N²) for fixed F. The worst-case O(N³) occurs in adversarial or highly diverse trial sets common in many-objective optimization.
The correct algorithm builds the dominance relation once in O(N²) (N² pairwise comparisons, each O(M) for M objectives), then uses Kahn's topological sort on the resulting DAG to extract fronts in O(N+E) where E ≤ N². Total: O(N²×M).
Impact
Researchers and ML engineers running multi-objective hyperparameter optimization studies with hundreds of trials (e.g., optimizing accuracy + latency + memory simultaneously) experience cubic slowdown in the study summary and Pareto front visualization calls. Studies with N=500 trials and diverse objective values can take minutes instead of seconds to summarize.
The Fix
Build the dominance graph once: for all pairs (i, j), determine if trial i dominates trial j in O(M) time. Record dominated_count[j] and dominators[i]. Use a queue of trials with dominated_count == 0 as rank-0 candidates and apply Kahn extraction, decrementing counts as each rank is emitted.
Patch
- def _calculate_nondomination_rank(loss_values):
- population = list(range(len(loss_values)))
- ranks = []
- while population:
- front = _is_pareto_front_nd(loss_values, population)
- ranks.append(front)
- population = [i for i in population if i not in front]
- return ranks
+ def _calculate_nondomination_rank(loss_values):
+ n = len(loss_values)
+ dominated_count = [0] * n
+ dominators = [[] for _ in range(n)]
+ for i in range(n):
+ for j in range(n):
+ if i != j and _dominates(loss_values[i], loss_values[j]):
+ dominated_count[j] += 1
+ dominators[i].append(j)
+ from collections import deque
+ queue = deque(i for i in range(n) if dominated_count[i] == 0)
+ ranks = []
+ while queue:
+ front = list(queue)
+ ranks.append(front)
+ next_queue = deque()
+ for i in front:
+ for j in dominators[i]:
+ dominated_count[j] -= 1
+ if dominated_count[j] == 0:
+ next_queue.append(j)
+ queue = next_queue
+ return ranks
What We Ask
Please review, apply, and coordinate a 90-day disclosure window before public release. Reply to security@undefect.com.
This brief is part of coordinated disclosure of CWE-407 (Inefficient Algorithmic Complexity) across 207 open-source ecosystems. Full report: https://undefect.com