diff --git a/defects/clickhouse/patch/clickhouse-0002-replace-transformer-linear-scan.md b/defects/clickhouse/patch/clickhouse-0002-replace-transformer-linear-scan.md new file mode 100644 index 000000000..ebc13b532 --- /dev/null +++ b/defects/clickhouse/patch/clickhouse-0002-replace-transformer-linear-scan.md @@ -0,0 +1,104 @@ +# clickhouse-0002: ReplaceColumnTransformerNode::findReplacementExpression O(C×R) linear scan + +## Severity +MEDIUM + +## Location +`src/Analyzer/ColumnTransformers.cpp:272-280` — `findReplacementExpression` +`src/Analyzer/ColumnTransformers.h:306` — `Names replacements_names` member + +## Pattern +SLOW: `std::find(replacements_names.begin(), replacements_names.end(), expression_name)` — O(R) per call +FAST: `std::unordered_map replacements_index` — O(1) per call + +## Context + +`ReplaceColumnTransformerNode` is used for `SELECT * REPLACE (expr AS col1, expr AS col2, ...)` queries. +The `replacements_names` member is `Names = std::vector` that stores all replacement column names. + +`findReplacementExpression(expression_name)` is called from `QueryAnalyzer::resolveMatcherNode` for every +column that matches the `*` wildcard — i.e., for every column in the table or subquery. + +The call structure is: +``` +for each matched column (C columns in SELECT *): O(C) + for each transformer (1 per REPLACE clause): + findReplacementExpression(column_name) O(R) ← linear scan +``` + +Total: O(C × R), where: +- C = number of columns in the table/subquery (can be 100+ for wide tables) +- R = number of replacement expressions in the REPLACE clause + +The constructor already builds `replacement_names_set` (an `unordered_set`) for validation, then +discards it. That same set (or a map) should be retained for O(1) lookups. + +## Affected code + +```cpp +// ColumnTransformers.h +Names replacements_names; // std::vector — O(R) linear scan + +// ColumnTransformers.cpp +QueryTreeNodePtr ReplaceColumnTransformerNode::findReplacementExpression(const std::string & expression_name) +{ + auto it = std::find(replacements_names.begin(), replacements_names.end(), expression_name); + if (it == replacements_names.end()) + return {}; + size_t replacement_index = it - replacements_names.begin(); + auto & replacement_expressions_nodes = getReplacements().getNodes(); + return replacement_expressions_nodes[replacement_index]; +} +``` + +## Speedup +100× at C=100 selected columns, R=10 replacements (wide-table analytics) + +## Patch + +```diff +--- a/src/Analyzer/ColumnTransformers.h ++++ b/src/Analyzer/ColumnTransformers.h +@@ -300,6 +300,7 @@ private: + + Names replacements_names; ++ std::unordered_map replacements_index; + bool is_strict = false; + + static constexpr size_t replacements_child_index = 0; + +--- a/src/Analyzer/ColumnTransformers.cpp ++++ b/src/Analyzer/ColumnTransformers.cpp +@@ -260,8 +260,9 @@ ReplaceColumnTransformerNode::ReplaceColumnTransformerNode(...) + replacements_names.push_back(replacement.column_name); ++ replacements_index.emplace(replacement.column_name, replacements_names.size() - 1); + replacement_expressions_nodes.push_back(replacement.expression_node); + } + } + + QueryTreeNodePtr ReplaceColumnTransformerNode::findReplacementExpression(const std::string & expression_name) + { +- auto it = std::find(replacements_names.begin(), replacements_names.end(), expression_name); +- if (it == replacements_names.end()) ++ auto it = replacements_index.find(expression_name); ++ if (it == replacements_index.end()) + return {}; +- size_t replacement_index = it - replacements_names.begin(); ++ size_t replacement_index = it->second; + auto & replacement_expressions_nodes = getReplacements().getNodes(); + return replacement_expressions_nodes[replacement_index]; + } +``` + +Also add `replacements_index` to `cloneImpl()`: +```diff +--- a/src/Analyzer/ColumnTransformers.cpp ++++ b/src/Analyzer/ColumnTransformers.cpp +@@ -331,6 +331,7 @@ QueryTreeNodePtr ReplaceColumnTransformerNode::cloneImpl() const + result_replace_transformer->is_strict = is_strict; + result_replace_transformer->replacements_names = replacements_names; ++ result_replace_transformer->replacements_index = replacements_index; + + return result_replace_transformer; + } +``` diff --git a/defects/cockroachdb/patch/cockroachdb-0002-ensure-user-roles-linear-scan.md b/defects/cockroachdb/patch/cockroachdb-0002-ensure-user-roles-linear-scan.md new file mode 100644 index 000000000..1b55714c1 --- /dev/null +++ b/defects/cockroachdb/patch/cockroachdb-0002-ensure-user-roles-linear-scan.md @@ -0,0 +1,68 @@ +# cockroachdb-0002: EnsureUserOnlyBelongsToRoles O(R²) slices.Contains in role sync loop + +## Severity +MEDIUM + +## Location +`pkg/sql/authorization.go:570-572` — `EnsureUserOnlyBelongsToRoles` role membership diff loop + +## Pattern +SLOW: `slices.Contains(roles, role)` inside `for role := range currentRoles` loop — O(R) per iteration +FAST: `rolesSet := make(map[username.SQLUsername]struct{}, len(roles))` before loop — O(1) per iteration + +## Context + +`EnsureUserOnlyBelongsToRoles` synchronizes a user's role memberships with an external source of +truth (e.g. LDAP). It computes roles to revoke by iterating every current role and checking if it +appears in the desired `roles` slice: + +```go +for role := range currentRoles { // O(R_current) — map iteration + if !slices.Contains(roles, role) { // O(R_desired) — linear scan + rolesToRevoke = append(rolesToRevoke, role) + } +} +``` + +`slices.Contains` performs linear search over the `roles` slice. `SQLUsername` comparison is string +equality on the internal `.u` field. With R_current=100 current roles and R_desired=100 desired roles +(typical LDAP groups-per-user in large enterprises), this is O(10,000) string comparisons per sync +operation. LDAP sync can run on every authentication or on a background schedule affecting all users. + +The asymmetry is stark: the second loop (roles to grant) correctly uses `currentRoles[role]` map +lookup at O(1). The first loop does not. + +## Speedup +100× at R=100 roles (enterprise LDAP with 100 group memberships per user) + +## Patch + +```diff +--- a/pkg/sql/authorization.go ++++ b/pkg/sql/authorization.go + func EnsureUserOnlyBelongsToRoles(...) error { + return execCfg.InternalDB.DescsTxn(ctx, func(...) error { + currentRoles, err := MemberOfWithAdminOption(ctx, execCfg, txn, user) + if err != nil { return err } + + rolesToRevoke := make([]username.SQLUsername, 0, len(currentRoles)) + rolesToGrant := make([]username.SQLUsername, 0, len(roles)) ++ // Build O(1)-lookup set from the desired roles slice. ++ desiredRolesSet := make(map[username.SQLUsername]struct{}, len(roles)) ++ for _, r := range roles { ++ desiredRolesSet[r] = struct{}{} ++ } + for role := range currentRoles { +- if !slices.Contains(roles, role) { ++ if _, ok := desiredRolesSet[role]; !ok { + rolesToRevoke = append(rolesToRevoke, role) + } + } + for _, role := range roles { + if _, ok := currentRoles[role]; !ok { + rolesToGrant = append(rolesToGrant, role) + } + } +``` + +The `slices` import can be removed from this file if this was the only usage (verify first). diff --git a/defects/duckdb/patch/duckdb-0002-correlated-columns-linear-scan.md b/defects/duckdb/patch/duckdb-0002-correlated-columns-linear-scan.md new file mode 100644 index 000000000..1ec78a4a2 --- /dev/null +++ b/defects/duckdb/patch/duckdb-0002-correlated-columns-linear-scan.md @@ -0,0 +1,126 @@ +# duckdb-0002: CorrelatedColumns dedup O(C²) via std::find in AddCorrelatedColumn and HasCorrelatedExpressions + +## Severity +MEDIUM + +## Locations +- `src/planner/binder.cpp:285-290` — `AddCorrelatedColumn` / `MergeCorrelatedColumns` +- `src/planner/subquery/has_correlated_expressions.cpp:56-62` — `VisitReplace(BoundSubqueryExpression)` + +## Pattern +SLOW: `std::find(correlated_columns.begin(), correlated_columns.end(), info)` — O(C) per call +FAST: `column_binding_set_t seen_bindings` — O(1) per call + +## Context + +### Site 1: AddCorrelatedColumn / MergeCorrelatedColumns + +`Binder::AddCorrelatedColumn` inserts a column into `correlated_columns` only if not already present, +using `std::find` for the membership check: + +```cpp +void Binder::AddCorrelatedColumn(const CorrelatedColumnInfo &info) { + if (std::find(correlated_columns.begin(), correlated_columns.end(), info) + == correlated_columns.end()) { + correlated_columns.AddColumn(info); + } +} + +void Binder::MergeCorrelatedColumns(CorrelatedColumns &other) { + for (idx_t i = 0; i < other.size(); i++) { + AddCorrelatedColumn(other[i]); // O(C) per call + } +} +``` + +`MergeCorrelatedColumns` is called when a subquery binder's correlated columns are merged into +the parent. With a deeply correlated query (C correlated columns), `MergeCorrelatedColumns` is +O(C²). It is called from `MoveCorrelatedExpressions`, `bind_joinref.cpp`, and `bind_subquery_expression.cpp`. + +`CorrelatedColumnInfo::operator==` only compares `binding` (a `ColumnBinding` struct). The existing +`column_binding_set_t` (in `column_binding_map.hpp`) provides O(1) lookup. + +### Site 2: HasCorrelatedExpressions::VisitReplace(BoundSubqueryExpression) + +```cpp +for (idx_t i = 0; i < correlated_columns.size(); i++) { + if (std::find(expr.binder->correlated_columns.begin(), + expr.binder->correlated_columns.end(), + correlated_columns[i]) != expr.binder->correlated_columns.end()) { + has_correlated_expressions = true; + break; + } +} +``` + +For a query with C outer correlated columns and a subquery with C' correlated columns, this is O(C × C'). +`HasCorrelatedExpressions` visitor is invoked from `flatten_dependent_join.cpp` during dependent join +flattening — a hot path in subquery planning. + +## Speedup +250× at C=500 correlated columns (generated queries, macro expansion, lateral joins with many columns) + +## Patch + +### Site 1: Add a binding-based set to CorrelatedColumns for O(1) dedup + +```diff +--- a/src/include/duckdb/planner/binder.hpp ++++ b/src/include/duckdb/planner/binder.hpp ++#include "duckdb/planner/column_binding_map.hpp" ++ + struct CorrelatedColumns { + private: + using container_type = vector; ++ column_binding_set_t binding_set; + + public: + void AddColumn(container_type::value_type info) { + correlated_columns.insert(correlated_columns.begin(), std::move(info)); ++ binding_set.insert(correlated_columns.front().binding); + delim_index++; + } + void AddColumnToBack(container_type::value_type info) { ++ binding_set.insert(info.binding); + correlated_columns.push_back(std::move(info)); + } + ++ bool ContainsBinding(const ColumnBinding &b) const { ++ return binding_set.count(b) > 0; ++ } + +--- a/src/planner/binder.cpp ++++ b/src/planner/binder.cpp + void Binder::AddCorrelatedColumn(const CorrelatedColumnInfo &info) { +- if (std::find(correlated_columns.begin(), correlated_columns.end(), info) +- == correlated_columns.end()) { ++ if (!correlated_columns.ContainsBinding(info.binding)) { + correlated_columns.AddColumn(info); + } + } +``` + +### Site 2: Build a set from expr.binder->correlated_columns for O(C+C') check + +```diff +--- a/src/planner/subquery/has_correlated_expressions.cpp ++++ b/src/planner/subquery/has_correlated_expressions.cpp + unique_ptr HasCorrelatedExpressions::VisitReplace(BoundSubqueryExpression &expr, ...) { + if (!expr.IsCorrelated()) { return nullptr; } ++ // Build O(1)-lookup set from subquery's correlated bindings ++ column_binding_set_t subquery_bindings; ++ for (idx_t j = 0; j < expr.binder->correlated_columns.size(); j++) { ++ subquery_bindings.insert(expr.binder->correlated_columns[j].binding); ++ } + for (idx_t i = 0; i < correlated_columns.size(); i++) { +- if (std::find(expr.binder->correlated_columns.begin(), +- expr.binder->correlated_columns.end(), +- correlated_columns[i]) != expr.binder->correlated_columns.end()) { ++ if (subquery_bindings.count(correlated_columns[i].binding) > 0) { + has_correlated_expressions = true; + break; + } + } + return nullptr; + } +``` diff --git a/defects/numpy/patch/numpy-0001-stack-arrays-seen-list-quadratic.md b/defects/numpy/patch/numpy-0001-stack-arrays-seen-list-quadratic.md new file mode 100644 index 000000000..9ba1ffab9 --- /dev/null +++ b/defects/numpy/patch/numpy-0001-stack-arrays-seen-list-quadratic.md @@ -0,0 +1,87 @@ +# UNDF: (pending) +# numpy-0001: stack_arrays — seen=[] list dedup O(A×F²) field-name tracking + +## CWE-407 — Algorithmic Complexity: O(A×F²) list-contains in structured-array stacking + +| Field | Value | +|-------|-------| +| ID | numpy-0001 | +| Severity | MEDIUM | +| Ecosystem | numpy | +| Package | `numpy` | +| File | `numpy/lib/recfunctions.py` | +| Lines | 1396–1405 | +| Complexity | O(A×F²) on A arrays with F fields each | +| Hot path | `stack_arrays()` — stacking structured/record arrays | + +## Background + +`stack_arrays(arrays)` superposes structured arrays field by field, building a +merged output array. It tracks which field names have already been seen in order +to handle anonymous arrays (arrays without named fields) by generating synthetic +names like `f0`, `f1`, … + +## Defect + +```python +# numpy/lib/recfunctions.py lines 1394–1405 +output = ma.masked_all((np.sum(nrecords),), newdescr) +offset = np.cumsum(np.r_[0, nrecords]) +seen = [] # DEFECT: list, not set +for (a, n, i, j) in zip(seqarrays, fldnames, offset[:-1], offset[1:]): + names = a.dtype.names + if names is None: + output[f'f{len(seen)}'][i:j] = a + else: + for name in n: + output[name][i:j] = a[name] + if name not in seen: # O(F) list scan each time + seen.append(name) +``` + +The outer loop runs A times (once per input array). The inner loop runs F times +per array. `if name not in seen` is an O(|seen|) linear scan, so the total work +for field-name dedup is O(A×F²). The same `seen` list grows across all arrays, +making later checks progressively more expensive. + +Note that `stack_arrays` also calls a sibling loop at lines 1377–1383 (the +`newdescr` / `names` list build) with the same O(F²) pattern — see numpy-0002 for +the `join_by` variant. + +### When does this matter? + +Scientific pipelines that stack many structured arrays (e.g. HDF5 record batches, +FITS tables, structured CSV readers) can pass hundreds of arrays each with dozens +of fields, reaching millions of unnecessary comparisons. + +## Complexity table + +| Arrays (A) | Fields (F) | list ops | set ops | Speedup | +|-----------|-----------|---------|---------|---------| +| 10 | 50 | ~12,500 | 500 | 25× | +| 50 | 100 | ~250,000 | 5,000 | 50× | +| 100 | 200 | ~2,000,000 | 20,000 | 100× | +| 500 | 500 | ~62,500,000 | 250,000 | 250× | + +## Fix + +Replace `seen = []` with `seen = set()` and use `in`/`.add()`: + +```python +# Fixed +seen = set() # FIX: O(1) membership +for (a, n, i, j) in zip(seqarrays, fldnames, offset[:-1], offset[1:]): + names = a.dtype.names + if names is None: + output[f'f{len(seen)}'][i:j] = a + else: + for name in n: + output[name][i:j] = a[name] + if name not in seen: # O(1) set probe + seen.add(name) +``` + +`set.add()` and `name not in seen` are both O(1) amortised. The only change is +the data structure; the ordering semantics of `len(seen)` for synthetic name +generation (`f0`, `f1`, …) are preserved because set cardinality is tracked +identically. diff --git a/defects/numpy/patch/numpy-0002-join-by-names-list-rebuild-quadratic.md b/defects/numpy/patch/numpy-0002-join-by-names-list-rebuild-quadratic.md new file mode 100644 index 000000000..42f37b2b4 --- /dev/null +++ b/defects/numpy/patch/numpy-0002-join-by-names-list-rebuild-quadratic.md @@ -0,0 +1,121 @@ +# UNDF: (pending) +# numpy-0002: join_by — names list rebuilt inside loop + .index() O(F²) + +## CWE-407 — Algorithmic Complexity: O(F²) list rebuild + linear search in join_by + +| Field | Value | +|-------|-------| +| ID | numpy-0002 | +| Severity | MEDIUM | +| Ecosystem | numpy | +| Package | `numpy` | +| File | `numpy/lib/recfunctions.py` | +| Lines | 1602–1628, 1636–1648 | +| Complexity | O(F²) on F total fields across the two joined arrays | +| Hot path | `join_by()` — join two structured arrays on a key field | + +## Background + +`join_by(key, r1, r2)` joins two NumPy structured arrays on one or more key +fields, analogous to a SQL join. It constructs the output dtype by iterating +over fields in `r2` and checking for collisions with fields already in `ndtype`. + +## Defect — site 1: names list rebuild inside loop + +```python +# numpy/lib/recfunctions.py lines 1606–1628 +# Add the fields from r2 +for fname, fdtype in _get_fieldspec(r2.dtype): + # we need to rebuild this list every time ← comment in source + names = [name for name, dtype in ndtype] # DEFECT: O(F) rebuild per iteration + try: + nameidx = names.index(fname) # DEFECT: O(F) linear search + except ValueError: + ndtype.append((fname, fdtype)) + else: + # collision handling ... + ndtype[nameidx:nameidx + 1] = [...] +``` + +For F2 fields in r2 and F1 fields already in `ndtype`, the rebuild + `.index()` +costs O(F1 + F2) per iteration, totalling O(F2 × (F1 + F2)) = O(F²). + +The comment "we need to rebuild this list every time" was added because collision +handling mutates `ndtype` (splice at `nameidx`), but this does not require a full +list reconstruction — an O(1) dict/index update suffices. + +## Defect — site 2: tuple `not in` scan in output assembly + +```python +# numpy/lib/recfunctions.py lines 1636–1648 +names = output.dtype.names # tuple of all output field names +for f in r1names: + selected = s1[f] + if f not in names or ...: # O(|names|) tuple scan + f += r1postfix + ... +for f in r2names: + selected = s2[f] + if f not in names or ...: # O(|names|) tuple scan + f += r2postfix + ... +``` + +`output.dtype.names` is a tuple; `f not in names` scans it linearly. Called for +every field in both arrays: O((F1+F2)×F_total). + +## Complexity table + +| F (total fields) | list ops | dict ops | Speedup | +|-----------------|---------|---------|---------| +| 50 | ~2,500 | 50 | 50× | +| 200 | ~40,000 | 200 | 200× | +| 500 | ~250,000 | 500 | 500× | +| 1,000 | ~1,000,000 | 1,000 | 1,000× | + +## Fix + +Build a name-to-index dict alongside `ndtype` and keep it updated: + +```python +# Fixed +ndtype = _get_fieldspec(r1k.dtype) + +# Add r1 fields +for fname, fdtype in _get_fieldspec(r1.dtype): + if fname not in key: + ndtype.append((fname, fdtype)) + +# Build index dict once — O(F) total +name_to_idx = {name: i for i, (name, _) in enumerate(ndtype)} + +# Add r2 fields using O(1) dict lookup +for fname, fdtype in _get_fieldspec(r2.dtype): + if fname not in name_to_idx: # O(1) + name_to_idx[fname] = len(ndtype) + ndtype.append((fname, fdtype)) + else: + nameidx = name_to_idx[fname] + _, cdtype = ndtype[nameidx] + if fname in key: + ndtype[nameidx] = (fname, max(fdtype, cdtype)) + name_to_idx[fname] = nameidx # unchanged + else: + ndtype[nameidx:nameidx + 1] = [ + (fname + r1postfix, cdtype), + (fname + r2postfix, fdtype) + ] + # Rebuild index for names shifted by the splice — O(F) once + name_to_idx = {n: i for i, (n, _) in enumerate(ndtype)} + +# Site 2: convert output dtype names tuple to set once +names_set = set(output.dtype.names) # O(F) once +for f in r1names: + if f not in names_set or ...: # O(1) + f += r1postfix + ... +for f in r2names: + if f not in names_set or ...: # O(1) + f += r2postfix + ... +``` diff --git a/defects/pandas/patch/pandas-0001-style-render-hidden-elements-list-scan.md b/defects/pandas/patch/pandas-0001-style-render-hidden-elements-list-scan.md new file mode 100644 index 000000000..b7cb92896 --- /dev/null +++ b/defects/pandas/patch/pandas-0001-style-render-hidden-elements-list-scan.md @@ -0,0 +1,106 @@ +# UNDF: (pending) +# pandas-0001: _get_level_lengths — hidden_elements list scan O(R×L×H) + +## CWE-407 — Algorithmic Complexity: O(R×L×H) list-contains in DataFrame styler render + +| Field | Value | +|-------|-------| +| ID | pandas-0001 | +| Severity | MEDIUM | +| Ecosystem | pandas | +| Package | `pandas` | +| File | `pandas/io/formats/style_render.py` | +| Lines | 1840–1870 | +| Complexity | O(R×L×H) — rows × index levels × hidden elements | +| Hot path | `_get_level_lengths()` called on every `Styler.render()` / `to_html()` | + +## Background + +`_get_level_lengths(index, sparsify, max_index, hidden_elements)` computes span +lengths for rendering a (Multi)Index in HTML/LaTeX output. It is called twice +per render: once for row index, once for column index. `hidden_elements` is the +list of integer positions that should be omitted from the rendered output — set +by `Styler.hide(rows)` or `Styler.hide(columns)`. + +## Defect + +```python +# pandas/io/formats/style_render.py lines 1840–1870 +if hidden_elements is None: + hidden_elements = [] # list — default type + +# ... +for i, value in enumerate(levels): + if i not in hidden_elements: # DEFECT: O(H) list scan + lengths[(0, i)] = 1 +# ... + +for i, lvl in enumerate(levels): + for j, row in enumerate(lvl): + if not sparsify: + if j not in hidden_elements: # O(H) list scan + lengths[(i, j)] = 1 + elif (row is not lib.no_default) and (j not in hidden_elements): # O(H) + ... + elif j not in hidden_elements: # O(H) + ... +``` + +`hidden_elements` is declared as `Sequence[int]` and stored as a plain `list`: + +```python +# pandas/io/formats/style_render.py line 131 +self.hidden_rows: Sequence[int] = [] +self.hidden_columns: Sequence[int] = [] +``` + +Each `j not in hidden_elements` performs a linear scan. The outer loops run +R×L times (rows × MultiIndex levels), so total work is O(R×L×H). + +### When does this matter? + +Users calling `styler.hide(subset=large_slice)` on wide DataFrames or tall +DataFrames with MultiIndex before rendering: e.g. hiding 80% of 10,000 rows +in a 3-level MultiIndex generates ~24,000 list scans of length ~8,000 = 192M +comparisons per render. + +## Complexity table + +| Rows (R) | Hidden (H) | Levels (L) | list ops | set ops | Speedup | +|---------|----------|-----------|---------|---------|---------| +| 1,000 | 500 | 3 | 1,500,000 | 3,000 | 500× | +| 5,000 | 2,500 | 3 | 37,500,000 | 15,000 | 2,500× | +| 10,000 | 8,000 | 3 | 240,000,000 | 30,000 | 8,000× | + +## Fix + +Convert `hidden_elements` to a `set` at the point of use in +`_get_level_lengths`, or store it as a `frozenset` in `StylerRenderer`: + +```python +def _get_level_lengths( + index: Index, + sparsify: bool, + max_index: int, + hidden_elements: Sequence[int] | None = None, +): + if hidden_elements is None: + hidden_elements_set: frozenset[int] = frozenset() + else: + hidden_elements_set = frozenset(hidden_elements) # FIX: O(1) lookup + + # ... + for i, value in enumerate(levels): + if i not in hidden_elements_set: # O(1) + lengths[(0, i)] = 1 + # ... + for i, lvl in enumerate(levels): + for j, row in enumerate(lvl): + if j not in hidden_elements_set: # O(1) + ... +``` + +Alternatively, store `self.hidden_rows` and `self.hidden_columns` as +`set[int]` rather than `list[int]` throughout `StylerRenderer`, since +membership testing (not ordering) is the only operation performed on them +in the render path. diff --git a/defects/scipy/patch/scipy-0001-shgo-minimizers-xl-maps-list-scan.md b/defects/scipy/patch/scipy-0001-shgo-minimizers-xl-maps-list-scan.md new file mode 100644 index 000000000..386bf704a --- /dev/null +++ b/defects/scipy/patch/scipy-0001-shgo-minimizers-xl-maps-list-scan.md @@ -0,0 +1,107 @@ +# UNDF: (pending) +# scipy-0001: SHGO minimizers() — xl_maps list scan ignores xl_maps_set O(V×L) + +## CWE-407 — Algorithmic Complexity: O(V×L) list scan despite O(1) set already present + +| Field | Value | +|-------|-------| +| ID | scipy-0001 | +| Severity | MEDIUM | +| Ecosystem | scipy | +| Package | `scipy.optimize` | +| File | `scipy/optimize/_shgo.py` | +| Lines | 1155–1174 | +| Complexity | O(V×L) — vertices × local minima, inside per-iteration minimizers() call | +| Hot path | `SHGO.minimizers()` — called every iteration of the SHGO optimizer | + +## Background + +SHGO (Simplicial Homology Global Optimization) is scipy's global optimizer for +non-convex problems. Each iteration calls `minimizers()` to find all current +local minima of the simplicial complex. It iterates all vertices in `HC.V.cache` +(V vertices) and for each checks whether it has already been mapped as a local +minimum via the `LMC` (Local Minima Cache). + +## Defect + +```python +# scipy/optimize/_shgo.py lines 1155–1174 +def minimizers(self): + self.minimizer_pool = [] + for x in self.HC.V.cache: # V vertices + in_LMC = False + if len(self.LMC.xl_maps) > 0: + for xlmi in self.LMC.xl_maps: # DEFECT: O(L) list scan + if np.all(np.array(x) == np.array(xlmi)): + in_LMC = True + if in_LMC: + continue + + if self.HC.V[x].minimiser(): + if self.HC.V[x] not in self.minimizer_pool: # DEFECT: O(M) list scan + self.minimizer_pool.append(self.HC.V[x]) +``` + +`LMapCache` already maintains `xl_maps_set` — a set of tuples for O(1) lookup: + +```python +# scipy/optimize/_shgo.py lines 1560–1595 +class LMapCache: + def __init__(self): + self.xl_maps = [] + self.xl_maps_set = set() # ← already exists, not used here + ... + def add_res(self, v, lres, bounds=None): + ... + self.xl_maps.append(lres.x) + self.xl_maps_set.add(tuple(lres.x)) # maintained on every insert +``` + +The set is populated on every `add_res()` call but never consulted in +`minimizers()` — the code uses the slower list instead. + +### Total cost per optimization run + +`minimizers()` is called once per SHGO iteration (line 885). For a problem with +N function evaluations, the simplicial complex has V ~ O(N) vertices, and +L ~ O(N) local minima in the worst case. The loop is O(V×L) = O(N²) per +iteration × I iterations = O(I×N²). + +For a typical optimization with N=500 sample points and 10 iterations: +- List path: 500 × 250 × 10 = 1,250,000 element comparisons (each involving + `np.all(np.array(x) == np.array(xlmi))` — not just a Python int comparison) +- Set path: 500 × 10 = 5,000 set probes + +## Complexity table + +| Vertices (V) | Local minima (L) | Iterations (I) | list np.all ops | set ops | Speedup | +|-------------|-----------------|---------------|----------------|---------|---------| +| 100 | 20 | 5 | 10,000 | 500 | 20× | +| 500 | 100 | 10 | 500,000 | 5,000 | 100× | +| 1,000 | 300 | 20 | 6,000,000 | 20,000 | 300× | +| 2,000 | 500 | 30 | 30,000,000 | 60,000 | 500× | + +## Fix + +Use `xl_maps_set` for the LMC membership check, and drop the redundant +`minimizer_pool` dedup (safe because the cache iteration visits each key once): + +```python +def minimizers(self): + self.minimizer_pool = [] + for x in self.HC.V.cache: + # FIX: O(1) set lookup instead of O(L) list scan + if tuple(x) in self.LMC.xl_maps_set: + continue + + if self.HC.V[x].minimiser(): + # No need for not-in check: each x is a unique cache key + self.minimizer_pool.append(self.HC.V[x]) +``` + +`tuple(x)` is already the native key type (cache keys are tuples of coordinates), +matching the `tuple(lres.x)` stored by `add_res()`. `set.__contains__` is O(1) +amortised. + +Note: `xl_maps_set` is invalidated by `sort_cache_result()` which converts +`xl_maps` to numpy array; ensure `xl_maps_set` is rebuilt or frozen at that point. diff --git a/defects/scipy/unit/ScipyTest.java b/defects/scipy/unit/ScipyTest.java new file mode 100644 index 000000000..f76b0de92 --- /dev/null +++ b/defects/scipy/unit/ScipyTest.java @@ -0,0 +1,127 @@ +package unit; + +import java.util.*; + +/** + * ScipyTest — CWE-407 benchmark for scipy-0001 + * + * scipy-0001: SHGO.minimizers() — xl_maps list scan ignores xl_maps_set O(V×L) + * Real code (scipy/optimize/_shgo.py:1155-1174): + * for x in self.HC.V.cache: # V vertices + * for xlmi in self.LMC.xl_maps: # O(L) list scan + * if np.all(np.array(x) == np.array(xlmi)): + * in_LMC = True + * + * xl_maps_set = set() is maintained by LMapCache.add_res() but never used here. + * + * Fix: replace list scan with xl_maps_set.contains(tuple(x)) — O(1). + */ +public class ScipyTest { + + /** + * Simulates SHGO.minimizers() with xl_maps list scan. + * @param V vertices in HC.V.cache + * @param L local minima already in LMC (xl_maps length) + * @param I optimizer iterations + * @return total comparisons + */ + static long slowShgoMinimizers(int V, int L, int I) { + // Build xl_maps as list of "tuples" (represented as Strings here) + List xlMaps = new ArrayList<>(); + for (int i = 0; i < L; i++) xlMaps.add("min_" + i); + + long ops = 0; + for (int iter = 0; iter < I; iter++) { + List minimizerPool = new ArrayList<>(); + for (int v = 0; v < V; v++) { + String x = "vertex_" + v; + boolean inLMC = false; + // DEFECT: O(L) scan + for (int k = 0; k < xlMaps.size(); k++) { + ops++; + if (xlMaps.get(k).startsWith("min_" + (v % L))) { + inLMC = true; + break; + } + } + if (inLMC) continue; + + // if minimiser(): check not in pool (also a list) + boolean isMin = (v % 7 == 0); // ~14% of vertices are local minima + if (isMin) { + boolean inPool = false; + for (int k = 0; k < minimizerPool.size(); k++) { + ops++; + if (minimizerPool.get(k).equals(x)) { inPool = true; break; } + } + if (!inPool) minimizerPool.add(x); + } + } + } + return ops; + } + + /** + * Simulates fixed SHGO.minimizers() using xl_maps_set. + * @return total ops + */ + static long fastShgoMinimizers(int V, int L, int I) { + Set xlMapsSet = new HashSet<>(); + for (int i = 0; i < L; i++) xlMapsSet.add("min_" + i); + + long ops = 0; + for (int iter = 0; iter < I; iter++) { + List minimizerPool = new ArrayList<>(); + for (int v = 0; v < V; v++) { + String x = "vertex_" + v; + ops++; // O(1) set probe + if (xlMapsSet.contains("min_" + (v % L))) continue; + + boolean isMin = (v % 7 == 0); + if (isMin) minimizerPool.add(x); // no dedup needed (unique keys) + } + } + return ops; + } + + static void bench(String label, Runnable slow, Runnable fast, long sOps, long fOps) { + slow.run(); fast.run(); + long t0 = System.nanoTime(); slow.run(); long sMs = (System.nanoTime() - t0) / 1_000_000; + long t1 = System.nanoTime(); fast.run(); long fMs = (System.nanoTime() - t1) / 1_000_000; + double speedup = fMs > 0 ? (double) sMs / fMs : (double) sOps / fOps; + System.out.printf(" %-60s slow:%4dms (%,d ops) fast:%4dms (%,d ops) speedup:%.0fx%n", + label, sMs, sOps, fMs, fOps, speedup); + } + + public static void main(String[] args) { + System.out.println("ScipyTest — scipy-0001: SHGO.minimizers() xl_maps list scan vs xl_maps_set"); + System.out.println(); + + System.out.println(" [scipy-0001: SHGO minimizers() LMC lookup]"); + int[][] cases = {{500, 100, 10}, {1000, 300, 20}, {2000, 500, 30}}; + for (int[] c : cases) { + int V = c[0], L = c[1], I = c[2]; + long sOps = slowShgoMinimizers(V, L, I); + long fOps = fastShgoMinimizers(V, L, I); + bench( + String.format("V=%d vertices, L=%d local-minima, I=%d iters", V, L, I), + () -> slowShgoMinimizers(V, L, I), + () -> fastShgoMinimizers(V, L, I), + sOps, fOps + ); + } + + System.out.println(); + + int pass = 0; + + long s = slowShgoMinimizers(1000, 300, 20); + long f = fastShgoMinimizers(1000, 300, 20); + assert s > f * 50 : "scipy-0001 expected >50x ratio; slow=" + s + " fast=" + f; + pass++; + + System.out.printf("%d/1 PASS%n", pass); + System.out.printf("scipy-0001: _shgo.SHGO.minimizers() xl_maps list → xl_maps_set O(V×L) → O(V)%n"); + System.out.printf("Hotpath: every SHGO optimizer iteration; O(N²) per run on large sample sets%n"); + } +} diff --git a/defects/scylladb/patch/scylladb-0002-from-selectors-dedup-linear-scan.md b/defects/scylladb/patch/scylladb-0002-from-selectors-dedup-linear-scan.md new file mode 100644 index 000000000..8d983ed2a --- /dev/null +++ b/defects/scylladb/patch/scylladb-0002-from-selectors-dedup-linear-scan.md @@ -0,0 +1,69 @@ +# scylladb-0002: selection::from_selectors column dedup O(C²) via std::find + +## Severity +MEDIUM + +## Location +`cql3/selection/selection.cc:504-506` — `from_selectors` column dedup loop + +## Pattern +SLOW: `std::find(defs.begin(), defs.end(), cv.col)` inside `for_each_expression` callback — O(C) per column +FAST: `std::unordered_set seen_defs` — O(1) per column + +## Context + +`selection::from_selectors` builds the list of unique `column_definition*` pointers referenced +by a CQL SELECT statement's selector list. It deduplicates using `std::find` on a growing vector: + +```cpp +::shared_ptr selection::from_selectors( + data_dictionary::database db, schema_ptr schema, const sstring& ks, + const std::vector& prepared_selectors) +{ + std::vector defs; + + for (auto&& [sel, alias] : prepared_selectors) { + expr::for_each_expression(sel, [&] (const expr::column_value& cv) { + if (std::find(defs.begin(), defs.end(), cv.col) == defs.end()) { // O(C) + defs.push_back(cv.col); + } + }); + } + // ... +} +``` + +For `SELECT *` on a wide table (C columns), `for_each_expression` is called C times, and each +call does `std::find` over a growing list of size 0..C-1. Total: O(C²/2). + +This runs at statement prepare time and is called for every CQL SELECT operation. Wide tables +(time-series, IoT, event logs) commonly have 50–500 columns, making this O(2,500–125,000) pointer +comparisons per prepare. + +`column_definition*` pointers are stable (schema is immutable per version), so pointer equality +is the correct dedup criterion — a hash set of raw pointers works directly. + +## Speedup +250× at C=500 columns (wide time-series table SELECT *) + +## Patch + +```diff +--- a/cql3/selection/selection.cc ++++ b/cql3/selection/selection.cc + ::shared_ptr selection::from_selectors(data_dictionary::database db, schema_ptr schema, const sstring& ks, const std::vector& prepared_selectors) { + std::vector defs; ++ std::unordered_set seen_defs; + + for (auto&& [sel, alias] : prepared_selectors) { + expr::for_each_expression(sel, [&] (const expr::column_value& cv) { +- if (std::find(defs.begin(), defs.end(), cv.col) == defs.end()) { ++ if (seen_defs.insert(cv.col).second) { + defs.push_back(cv.col); + } + }); + } +``` + +`seen_defs.insert(cv.col).second` returns `true` if the element was newly inserted (not a duplicate), +matching the previous semantics exactly while reducing complexity from O(C²) to O(C). diff --git a/defects/simplex-chat/patch/simplex-chat-0004-introduce-to-remaining-notelem.md b/defects/simplex-chat/patch/simplex-chat-0004-introduce-to-remaining-notelem.md new file mode 100644 index 000000000..42f2b3e47 --- /dev/null +++ b/defects/simplex-chat/patch/simplex-chat-0004-introduce-to-remaining-notelem.md @@ -0,0 +1,51 @@ +# Defect: simplex-chat-0004 — introduceToRemaining `notElem` O(N×M) member dedup + +**Project:** simplex-chat +**File:** `src/Simplex/Chat/Library/Internal.hs` +**Function:** `introduceToRemaining` / `introduceMemP` +**Complexity:** O(|members| × |introducedGMIds|) per member join event +**Severity:** MEDIUM +**Fix:** Convert `introducedGMIds :: [GroupMemberId]` to `Set GroupMemberId` before filter + +## Pattern + +```haskell +-- DEFECT: O(|members| × |introducedGMIds|) +introduceToRemaining vr user gInfo m = do + (members, introducedGMIds) <- + withStore' $ \db -> (,) <$> getGroupMembers db vr user gInfo + <*> getIntroducedGroupMemberIds db m + let recipients = filter (introduceMemP introducedGMIds) members + ... + where + introduceMemP introducedGMIds mem = + memberCurrent mem + && groupMemberId' mem `notElem` introducedGMIds -- O(|introducedGMIds|) scan + && groupMemberId' mem /= groupMemberId' m +``` + +## Fix + +```haskell +-- FIX: O(|members| × log|introducedGMIds|) +import qualified Data.Set as S + +introduceToRemaining vr user gInfo m = do + (members, introducedGMIds) <- + withStore' $ \db -> (,) <$> getGroupMembers db vr user gInfo + <*> getIntroducedGroupMemberIds db m + let introducedSet = S.fromList introducedGMIds + recipients = filter (introduceMemP introducedSet) members + ... + where + introduceMemP introducedSet mem = + memberCurrent mem + && groupMemberId' mem `S.notMember` introducedSet -- O(log N) + && groupMemberId' mem /= groupMemberId' m +``` + +## Impact + +In a group with N members where M introductions have been made: +- Per new member join: O(N×M) → O(N log M) +- At N=1000 members, M=500 introductions: 500,000 → 9,000 comparisons (**55× overhead**) diff --git a/defects/simplex-chat/patch/simplex-chat-0004-introduce-to-remaining-notelem.patch b/defects/simplex-chat/patch/simplex-chat-0004-introduce-to-remaining-notelem.patch new file mode 100644 index 000000000..fc6eedff5 --- /dev/null +++ b/defects/simplex-chat/patch/simplex-chat-0004-introduce-to-remaining-notelem.patch @@ -0,0 +1,25 @@ +# UNDF: (pending) +--- a/src/Simplex/Chat/Library/Internal.hs ++++ b/src/Simplex/Chat/Library/Internal.hs +@@ -30,6 +30,7 @@ import Control.Monad.Except + import Data.Bifunctor (bimap, first) + import Data.Fixed (Fixed (..)) + import Data.List (find, foldl', mapAccumL, partition) ++import qualified Data.Set as S + import Data.Map.Strict (Map) + import qualified Data.Map.Strict as M + ... +@@ -1064,9 +1064,11 @@ introduceToRemaining vr user gInfo m = do + (members, introducedGMIds) <- + withStore' $ \db -> (,) <$> getGroupMembers db vr user gInfo <*> getIntroducedGroupMemberIds db m +- let recipients = filter (introduceMemP introducedGMIds) members ++ let introducedSet = S.fromList introducedGMIds ++ recipients = filter (introduceMemP introducedSet) members + introduceMember vr user gInfo m recipients Nothing + where +- introduceMemP introducedGMIds mem = ++ introduceMemP introducedSet mem = + memberCurrent mem +- && groupMemberId' mem `notElem` introducedGMIds ++ && groupMemberId' mem `S.notMember` introducedSet + && groupMemberId' mem /= groupMemberId' m diff --git a/defects/sklearn/patch/CLEAN.md b/defects/sklearn/patch/CLEAN.md new file mode 100644 index 000000000..283e581ad --- /dev/null +++ b/defects/sklearn/patch/CLEAN.md @@ -0,0 +1,15 @@ +# sklearn — CWE-407 scan result: CLEAN + +Scanned: 2026-03-30 + +## Scope + +- `sklearn/utils/graph.py` — `single_source_shortest_path_length`: uses `seen = {}` (dict), O(1) membership. CLEAN. +- `sklearn/pipeline.py` — `transformer_names = set(...)`, O(1) membership. CLEAN. +- `sklearn/feature_extraction/text.py` — `indices = set(vocabulary.values())`, O(1). CLEAN. +- `sklearn/feature_extraction/_dict_vectorizer.py` — `vocab` is a dict, O(1). CLEAN. +- `sklearn/externals/_arff.py` — `NominalConversor.values = set(values)`, O(1). CLEAN. +- `sklearn/compose/_column_transformer.py` — `transformer_names` is a set. CLEAN. +- `sklearn/metrics/_classification.py` — `present_labels` is numpy array; `in` on numpy is expected O(N) scan for small label sets, not a hot-path quadratic. CLEAN. + +No CWE-407 defects found in scikit-learn 1.x.