diamond-scan: cpython/django O(2^D) hierarchy traversal; flask/rails/ruby/phoenix CLEAN
cpython-0001: pydoc.allmethods unconditional __bases__ recursion, no visited guard cpython-0002: turtle.__methodDict unconditional __bases__ recursion, fires at import cpython-0003: idlelib.rpc._getmethods unconditional __bases__ recursion, IDLE debug path django-0006: migrations.state.flatten_bases abstract model diamond traversal (comment in source acknowledges duplicates) flask/rails/ruby/phoenix: CLEAN
This commit is contained in:
parent
77a5410eee
commit
4d4c256673
8 changed files with 405 additions and 0 deletions
|
|
@ -0,0 +1,84 @@
|
|||
# UNDF: (pending)
|
||||
# cpython-0001: pydoc.allmethods — O(2^D) diamond base traversal
|
||||
|
||||
## CWE-407 — Algorithmic Complexity: O(2^D) diamond mixin/base traversal
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | cpython-0001 |
|
||||
| Severity | MEDIUM |
|
||||
| Ecosystem | cpython |
|
||||
| Package | pydoc |
|
||||
| File | `Lib/pydoc.py` |
|
||||
| Lines | 232–240 |
|
||||
| Complexity | O(2^D) on diamond inheritance hierarchies |
|
||||
| Hot path | `pydoc.allmethods(cl)` — collect all methods across inheritance tree |
|
||||
|
||||
## Defect
|
||||
|
||||
```python
|
||||
# BEFORE (DEFECT) — O(2^D): no visited guard, unconditional recursion
|
||||
def allmethods(cl):
|
||||
methods = {}
|
||||
for key, value in inspect.getmembers(cl, inspect.isroutine):
|
||||
methods[key] = 1
|
||||
for base in cl.__bases__:
|
||||
methods.update(allmethods(base)) # all your base are belong to us
|
||||
for key in methods.keys():
|
||||
methods[key] = getattr(cl, key)
|
||||
return methods
|
||||
```
|
||||
|
||||
With a diamond hierarchy `D → B, C → A` (A shared), `allmethods(D)` recurses into both
|
||||
`B` and `C`, each of which recurses into `A`, visiting it twice. At depth D the traversal
|
||||
visits `2^D` nodes instead of the O(D) unique classes.
|
||||
|
||||
## Fix
|
||||
|
||||
```python
|
||||
# AFTER — O(N): guard at entry prevents re-visiting shared ancestors
|
||||
def allmethods(cl, _visited=None):
|
||||
if _visited is None:
|
||||
_visited = set()
|
||||
if cl in _visited:
|
||||
return {}
|
||||
_visited.add(cl)
|
||||
methods = {}
|
||||
for key, value in inspect.getmembers(cl, inspect.isroutine):
|
||||
methods[key] = 1
|
||||
for base in cl.__bases__:
|
||||
methods.update(allmethods(base, _visited))
|
||||
for key in methods.keys():
|
||||
methods[key] = getattr(cl, key)
|
||||
return methods
|
||||
```
|
||||
|
||||
Note: `inspect.getmro(cl)` already returns the linearized MRO and could replace the
|
||||
recursive traversal entirely:
|
||||
|
||||
```python
|
||||
# SIMPLER FIX: use MRO directly (already handles diamond)
|
||||
def allmethods(cl):
|
||||
methods = {}
|
||||
for klass in reversed(inspect.getmro(cl)):
|
||||
for key, value in inspect.getmembers(klass, inspect.isroutine):
|
||||
methods[key] = 1
|
||||
for key in methods.keys():
|
||||
methods[key] = getattr(cl, key)
|
||||
return methods
|
||||
```
|
||||
|
||||
## Speedup
|
||||
|
||||
| Diamond depth (D) | Nodes visited (before) | Nodes visited (after) | Speedup |
|
||||
|------------------|----------------------|----------------------|---------|
|
||||
| 5 | 31 | 6 | 5× |
|
||||
| 10 | 1,023 | 11 | 93× |
|
||||
| 15 | 32,767 | 16 | 2,048× |
|
||||
| 20 | 1,048,575 | 21 | 49,932× |
|
||||
|
||||
## Notes
|
||||
|
||||
`allmethods` is defined in `pydoc.py` but is no longer called from the main doc-rendering
|
||||
paths (which use `__mro__` directly). However, it remains exported as part of the module's
|
||||
public API and is available for external callers. The defect is present and latent.
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
# UNDF: (pending)
|
||||
# cpython-0002: turtle.__methodDict — O(2^D) diamond base traversal
|
||||
|
||||
## CWE-407 — Algorithmic Complexity: O(2^D) diamond mixin/base traversal
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | cpython-0002 |
|
||||
| Severity | MEDIUM |
|
||||
| Ecosystem | cpython |
|
||||
| Package | turtle |
|
||||
| File | `Lib/turtle.py` |
|
||||
| Lines | 286–293 |
|
||||
| Complexity | O(2^D) on diamond inheritance hierarchies |
|
||||
| Hot path | Called at module import time via `__forwardmethods(ScrolledCanvas, TK.Canvas, '_canvas')` |
|
||||
|
||||
## Defect
|
||||
|
||||
```python
|
||||
# BEFORE (DEFECT) — O(2^D): no visited guard, unconditional recursion into __bases__
|
||||
def __methodDict(cls, _dict):
|
||||
"""helper function for Scrolled Canvas"""
|
||||
baseList = list(cls.__bases__)
|
||||
baseList.reverse()
|
||||
for _super in baseList:
|
||||
__methodDict(_super, _dict) # unconditional — diamond re-traversal
|
||||
for key, value in cls.__dict__.items():
|
||||
if type(value) == types.FunctionType:
|
||||
_dict[key] = value
|
||||
```
|
||||
|
||||
Called at module import time (line 426):
|
||||
```python
|
||||
__forwardmethods(ScrolledCanvas, TK.Canvas, '_canvas')
|
||||
```
|
||||
|
||||
`__forwardmethods` calls `__methodDict(toClass, _dict_1)` where `toClass` is `TK.Canvas`.
|
||||
If `TK.Canvas` participates in a diamond MRO (common in Tkinter/ttk widget hierarchies),
|
||||
the traversal visits shared ancestors exponentially many times.
|
||||
|
||||
## Fix
|
||||
|
||||
```python
|
||||
# AFTER — O(N): pass visited set to prevent re-traversal of shared ancestors
|
||||
def __methodDict(cls, _dict, _visited=None):
|
||||
"""helper function for Scrolled Canvas"""
|
||||
if _visited is None:
|
||||
_visited = set()
|
||||
if cls in _visited:
|
||||
return
|
||||
_visited.add(cls)
|
||||
baseList = list(cls.__bases__)
|
||||
baseList.reverse()
|
||||
for _super in baseList:
|
||||
__methodDict(_super, _dict, _visited)
|
||||
for key, value in cls.__dict__.items():
|
||||
if type(value) == types.FunctionType:
|
||||
_dict[key] = value
|
||||
```
|
||||
|
||||
Alternatively, use `cls.__mro__` directly:
|
||||
|
||||
```python
|
||||
# SIMPLER FIX: iterate MRO (linearized, no duplicates)
|
||||
def __methodDict(cls, _dict):
|
||||
"""helper function for Scrolled Canvas"""
|
||||
for klass in reversed(cls.__mro__):
|
||||
for key, value in klass.__dict__.items():
|
||||
if type(value) == types.FunctionType:
|
||||
_dict[key] = value
|
||||
```
|
||||
|
||||
## Speedup
|
||||
|
||||
| Diamond depth (D) | Nodes visited (before) | Nodes visited (after) | Speedup |
|
||||
|------------------|----------------------|----------------------|---------|
|
||||
| 5 | 31 | 6 | 5× |
|
||||
| 10 | 1,023 | 11 | 93× |
|
||||
| 15 | 32,767 | 16 | 2,048× |
|
||||
|
||||
## Notes
|
||||
|
||||
This fires at every `import turtle` since `__forwardmethods` is called unconditionally at
|
||||
module level. For typical Tkinter class hierarchies the depth is bounded, limiting practical
|
||||
impact — but ttk widgets add multiple inheritance layers that could trigger exponential behavior.
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
# UNDF: (pending)
|
||||
# cpython-0003: idlelib.rpc._getmethods — O(2^D) diamond base traversal
|
||||
|
||||
## CWE-407 — Algorithmic Complexity: O(2^D) diamond mixin/base traversal
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | cpython-0003 |
|
||||
| Severity | MEDIUM |
|
||||
| Ecosystem | cpython |
|
||||
| Package | idlelib.rpc |
|
||||
| File | `Lib/idlelib/rpc.py` |
|
||||
| Lines | 581–590 |
|
||||
| Complexity | O(2^D) on diamond inheritance hierarchies |
|
||||
| Hot path | Triggered when IDLE debugger calls `__methods__` on a remote object (every debug session inspect) |
|
||||
|
||||
## Defect
|
||||
|
||||
```python
|
||||
# BEFORE (DEFECT) — O(2^D): no visited guard, unconditional recursion into __bases__
|
||||
def _getmethods(obj, methods):
|
||||
# Helper to get a list of methods from an object
|
||||
# Adds names to dictionary argument 'methods'
|
||||
for name in dir(obj):
|
||||
attr = getattr(obj, name)
|
||||
if callable(attr):
|
||||
methods[name] = 1
|
||||
if isinstance(obj, type):
|
||||
for super in obj.__bases__:
|
||||
_getmethods(super, methods) # unconditional — diamond re-traversal
|
||||
```
|
||||
|
||||
Called via IDLE RPC when someone requests `__methods__` (line 179–181):
|
||||
```python
|
||||
if methodname == "__methods__":
|
||||
methods = {}
|
||||
_getmethods(obj, methods)
|
||||
```
|
||||
|
||||
With a diamond hierarchy, each shared ancestor is visited 2^D times instead of once.
|
||||
`dir(obj)` is called on each visit, making the actual cost O(2^D × M) where M is the
|
||||
number of methods per class.
|
||||
|
||||
## Fix
|
||||
|
||||
```python
|
||||
# AFTER — O(N): pass visited set to prevent re-traversal of shared ancestors
|
||||
def _getmethods(obj, methods, _visited=None):
|
||||
if _visited is None:
|
||||
_visited = set()
|
||||
if obj in _visited:
|
||||
return
|
||||
_visited.add(obj)
|
||||
for name in dir(obj):
|
||||
attr = getattr(obj, name)
|
||||
if callable(attr):
|
||||
methods[name] = 1
|
||||
if isinstance(obj, type):
|
||||
for super in obj.__bases__:
|
||||
_getmethods(super, methods, _visited)
|
||||
```
|
||||
|
||||
## Speedup
|
||||
|
||||
| Diamond depth (D) | `dir()` calls (before) | `dir()` calls (after) | Speedup |
|
||||
|------------------|----------------------|----------------------|---------|
|
||||
| 5 | 31 | 6 | 5× |
|
||||
| 10 | 1,023 | 11 | 93× |
|
||||
| 15 | 32,767 | 16 | 2,048× |
|
||||
|
||||
## Notes
|
||||
|
||||
`dir()` is an expensive call (it consults `__dict__`, `__dir__`, and all descriptor slots).
|
||||
In a deep diamond hierarchy under the IDLE debugger, inspecting a complex object could
|
||||
cause the IDLE UI to stall for seconds. The `methods` dict deduplication only prevents
|
||||
incorrect output — it does not prevent the exponential traversal cost.
|
||||
Loading…
Add table
Add a link
Reference in a new issue