simplex-chat-0004: introduceToRemaining notElem O(N×M) member dedup; fix: Set.notMember O(log N)
This commit is contained in:
parent
eb534f0944
commit
d9ca5b236f
12 changed files with 1006 additions and 0 deletions
|
|
@ -0,0 +1,87 @@
|
|||
# UNDF: (pending)
|
||||
# numpy-0001: stack_arrays — seen=[] list dedup O(A×F²) field-name tracking
|
||||
|
||||
## CWE-407 — Algorithmic Complexity: O(A×F²) list-contains in structured-array stacking
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | numpy-0001 |
|
||||
| Severity | MEDIUM |
|
||||
| Ecosystem | numpy |
|
||||
| Package | `numpy` |
|
||||
| File | `numpy/lib/recfunctions.py` |
|
||||
| Lines | 1396–1405 |
|
||||
| Complexity | O(A×F²) on A arrays with F fields each |
|
||||
| Hot path | `stack_arrays()` — stacking structured/record arrays |
|
||||
|
||||
## Background
|
||||
|
||||
`stack_arrays(arrays)` superposes structured arrays field by field, building a
|
||||
merged output array. It tracks which field names have already been seen in order
|
||||
to handle anonymous arrays (arrays without named fields) by generating synthetic
|
||||
names like `f0`, `f1`, …
|
||||
|
||||
## Defect
|
||||
|
||||
```python
|
||||
# numpy/lib/recfunctions.py lines 1394–1405
|
||||
output = ma.masked_all((np.sum(nrecords),), newdescr)
|
||||
offset = np.cumsum(np.r_[0, nrecords])
|
||||
seen = [] # DEFECT: list, not set
|
||||
for (a, n, i, j) in zip(seqarrays, fldnames, offset[:-1], offset[1:]):
|
||||
names = a.dtype.names
|
||||
if names is None:
|
||||
output[f'f{len(seen)}'][i:j] = a
|
||||
else:
|
||||
for name in n:
|
||||
output[name][i:j] = a[name]
|
||||
if name not in seen: # O(F) list scan each time
|
||||
seen.append(name)
|
||||
```
|
||||
|
||||
The outer loop runs A times (once per input array). The inner loop runs F times
|
||||
per array. `if name not in seen` is an O(|seen|) linear scan, so the total work
|
||||
for field-name dedup is O(A×F²). The same `seen` list grows across all arrays,
|
||||
making later checks progressively more expensive.
|
||||
|
||||
Note that `stack_arrays` also calls a sibling loop at lines 1377–1383 (the
|
||||
`newdescr` / `names` list build) with the same O(F²) pattern — see numpy-0002 for
|
||||
the `join_by` variant.
|
||||
|
||||
### When does this matter?
|
||||
|
||||
Scientific pipelines that stack many structured arrays (e.g. HDF5 record batches,
|
||||
FITS tables, structured CSV readers) can pass hundreds of arrays each with dozens
|
||||
of fields, reaching millions of unnecessary comparisons.
|
||||
|
||||
## Complexity table
|
||||
|
||||
| Arrays (A) | Fields (F) | list ops | set ops | Speedup |
|
||||
|-----------|-----------|---------|---------|---------|
|
||||
| 10 | 50 | ~12,500 | 500 | 25× |
|
||||
| 50 | 100 | ~250,000 | 5,000 | 50× |
|
||||
| 100 | 200 | ~2,000,000 | 20,000 | 100× |
|
||||
| 500 | 500 | ~62,500,000 | 250,000 | 250× |
|
||||
|
||||
## Fix
|
||||
|
||||
Replace `seen = []` with `seen = set()` and use `in`/`.add()`:
|
||||
|
||||
```python
|
||||
# Fixed
|
||||
seen = set() # FIX: O(1) membership
|
||||
for (a, n, i, j) in zip(seqarrays, fldnames, offset[:-1], offset[1:]):
|
||||
names = a.dtype.names
|
||||
if names is None:
|
||||
output[f'f{len(seen)}'][i:j] = a
|
||||
else:
|
||||
for name in n:
|
||||
output[name][i:j] = a[name]
|
||||
if name not in seen: # O(1) set probe
|
||||
seen.add(name)
|
||||
```
|
||||
|
||||
`set.add()` and `name not in seen` are both O(1) amortised. The only change is
|
||||
the data structure; the ordering semantics of `len(seen)` for synthetic name
|
||||
generation (`f0`, `f1`, …) are preserved because set cardinality is tracked
|
||||
identically.
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
# UNDF: (pending)
|
||||
# numpy-0002: join_by — names list rebuilt inside loop + .index() O(F²)
|
||||
|
||||
## CWE-407 — Algorithmic Complexity: O(F²) list rebuild + linear search in join_by
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | numpy-0002 |
|
||||
| Severity | MEDIUM |
|
||||
| Ecosystem | numpy |
|
||||
| Package | `numpy` |
|
||||
| File | `numpy/lib/recfunctions.py` |
|
||||
| Lines | 1602–1628, 1636–1648 |
|
||||
| Complexity | O(F²) on F total fields across the two joined arrays |
|
||||
| Hot path | `join_by()` — join two structured arrays on a key field |
|
||||
|
||||
## Background
|
||||
|
||||
`join_by(key, r1, r2)` joins two NumPy structured arrays on one or more key
|
||||
fields, analogous to a SQL join. It constructs the output dtype by iterating
|
||||
over fields in `r2` and checking for collisions with fields already in `ndtype`.
|
||||
|
||||
## Defect — site 1: names list rebuild inside loop
|
||||
|
||||
```python
|
||||
# numpy/lib/recfunctions.py lines 1606–1628
|
||||
# Add the fields from r2
|
||||
for fname, fdtype in _get_fieldspec(r2.dtype):
|
||||
# we need to rebuild this list every time ← comment in source
|
||||
names = [name for name, dtype in ndtype] # DEFECT: O(F) rebuild per iteration
|
||||
try:
|
||||
nameidx = names.index(fname) # DEFECT: O(F) linear search
|
||||
except ValueError:
|
||||
ndtype.append((fname, fdtype))
|
||||
else:
|
||||
# collision handling ...
|
||||
ndtype[nameidx:nameidx + 1] = [...]
|
||||
```
|
||||
|
||||
For F2 fields in r2 and F1 fields already in `ndtype`, the rebuild + `.index()`
|
||||
costs O(F1 + F2) per iteration, totalling O(F2 × (F1 + F2)) = O(F²).
|
||||
|
||||
The comment "we need to rebuild this list every time" was added because collision
|
||||
handling mutates `ndtype` (splice at `nameidx`), but this does not require a full
|
||||
list reconstruction — an O(1) dict/index update suffices.
|
||||
|
||||
## Defect — site 2: tuple `not in` scan in output assembly
|
||||
|
||||
```python
|
||||
# numpy/lib/recfunctions.py lines 1636–1648
|
||||
names = output.dtype.names # tuple of all output field names
|
||||
for f in r1names:
|
||||
selected = s1[f]
|
||||
if f not in names or ...: # O(|names|) tuple scan
|
||||
f += r1postfix
|
||||
...
|
||||
for f in r2names:
|
||||
selected = s2[f]
|
||||
if f not in names or ...: # O(|names|) tuple scan
|
||||
f += r2postfix
|
||||
...
|
||||
```
|
||||
|
||||
`output.dtype.names` is a tuple; `f not in names` scans it linearly. Called for
|
||||
every field in both arrays: O((F1+F2)×F_total).
|
||||
|
||||
## Complexity table
|
||||
|
||||
| F (total fields) | list ops | dict ops | Speedup |
|
||||
|-----------------|---------|---------|---------|
|
||||
| 50 | ~2,500 | 50 | 50× |
|
||||
| 200 | ~40,000 | 200 | 200× |
|
||||
| 500 | ~250,000 | 500 | 500× |
|
||||
| 1,000 | ~1,000,000 | 1,000 | 1,000× |
|
||||
|
||||
## Fix
|
||||
|
||||
Build a name-to-index dict alongside `ndtype` and keep it updated:
|
||||
|
||||
```python
|
||||
# Fixed
|
||||
ndtype = _get_fieldspec(r1k.dtype)
|
||||
|
||||
# Add r1 fields
|
||||
for fname, fdtype in _get_fieldspec(r1.dtype):
|
||||
if fname not in key:
|
||||
ndtype.append((fname, fdtype))
|
||||
|
||||
# Build index dict once — O(F) total
|
||||
name_to_idx = {name: i for i, (name, _) in enumerate(ndtype)}
|
||||
|
||||
# Add r2 fields using O(1) dict lookup
|
||||
for fname, fdtype in _get_fieldspec(r2.dtype):
|
||||
if fname not in name_to_idx: # O(1)
|
||||
name_to_idx[fname] = len(ndtype)
|
||||
ndtype.append((fname, fdtype))
|
||||
else:
|
||||
nameidx = name_to_idx[fname]
|
||||
_, cdtype = ndtype[nameidx]
|
||||
if fname in key:
|
||||
ndtype[nameidx] = (fname, max(fdtype, cdtype))
|
||||
name_to_idx[fname] = nameidx # unchanged
|
||||
else:
|
||||
ndtype[nameidx:nameidx + 1] = [
|
||||
(fname + r1postfix, cdtype),
|
||||
(fname + r2postfix, fdtype)
|
||||
]
|
||||
# Rebuild index for names shifted by the splice — O(F) once
|
||||
name_to_idx = {n: i for i, (n, _) in enumerate(ndtype)}
|
||||
|
||||
# Site 2: convert output dtype names tuple to set once
|
||||
names_set = set(output.dtype.names) # O(F) once
|
||||
for f in r1names:
|
||||
if f not in names_set or ...: # O(1)
|
||||
f += r1postfix
|
||||
...
|
||||
for f in r2names:
|
||||
if f not in names_set or ...: # O(1)
|
||||
f += r2postfix
|
||||
...
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue