russell@unturf.com
0a580b313d
Authors: russell@unturf.com · brackishbert@gmail.com · foxhop.net · TimeHexOn.com Patches, unit tests, benchmarks, whitepaper, and outreach briefs. Public domain — no copyright claimed. Use freely.
127 lines
3.5 KiB
ReStructuredText
127 lines
3.5 KiB
ReStructuredText
CPython PEG Parser Generator — CWE-407 Analysis
|
|
================================================
|
|
|
|
.. contents:: :local:
|
|
|
|
Overview
|
|
--------
|
|
|
|
CPython's PEG parser generator (``peg_generator``) generates the parser used by CPython itself
|
|
since Python 3.9. It includes a utility module ``sccutils.py`` that computes strongly connected
|
|
components of the grammar rule dependency graph. One CWE-407 defect site was found in the SCC
|
|
implementation. It has been patched.
|
|
|
|
Defect Sites
|
|
------------
|
|
|
|
cpython-0001
|
|
~~~~~~~~~~~~
|
|
|
|
**File:** ``Tools/peg_generator/peg_parser/sccutils.py:73``
|
|
|
|
**Pattern:**
|
|
|
|
.. code-block:: python
|
|
|
|
# Tarjan SCC — 'node in path' where path is a list
|
|
def strongly_connected_components(graph):
|
|
identified = set()
|
|
stack = []
|
|
lowlinks = {}
|
|
index = {}
|
|
|
|
def dfs(v):
|
|
index[v] = lowlinks[v] = len(index)
|
|
stack.append(v)
|
|
for w in graph.get(v, ()):
|
|
if w not in index:
|
|
dfs(w)
|
|
lowlinks[v] = min(lowlinks[v], lowlinks[w])
|
|
elif w not in identified:
|
|
if w in stack: # O(V) — 'in' on list is linear scan
|
|
lowlinks[v] = min(lowlinks[v], index[w])
|
|
if lowlinks[v] == index[v]:
|
|
scc = set()
|
|
while True:
|
|
w = stack.pop()
|
|
identified.add(w)
|
|
scc.add(w)
|
|
if w == v:
|
|
break
|
|
yield scc
|
|
|
|
**Why this is O(n):** ``w in stack`` where ``stack`` is a Python list; ``in`` on a list is
|
|
O(|stack|) = O(V).
|
|
|
|
**Complexity:** ``O(V²)`` where V = number of grammar rules
|
|
|
|
**Patch:**
|
|
|
|
.. code-block:: python
|
|
|
|
def strongly_connected_components(graph):
|
|
identified = set()
|
|
stack = []
|
|
on_stack = set() # O(1) membership test
|
|
lowlinks = {}
|
|
index = {}
|
|
|
|
def dfs(v):
|
|
index[v] = lowlinks[v] = len(index)
|
|
stack.append(v)
|
|
on_stack.add(v)
|
|
for w in graph.get(v, ()):
|
|
if w not in index:
|
|
dfs(w)
|
|
lowlinks[v] = min(lowlinks[v], lowlinks[w])
|
|
elif w not in identified:
|
|
if w in on_stack: # O(1)
|
|
lowlinks[v] = min(lowlinks[v], index[w])
|
|
if lowlinks[v] == index[v]:
|
|
scc = set()
|
|
while True:
|
|
w = stack.pop()
|
|
on_stack.discard(w)
|
|
identified.add(w)
|
|
scc.add(w)
|
|
if w == v:
|
|
break
|
|
yield scc
|
|
|
|
**Data structure change:** ``list`` + ``in`` → ``set`` + ``in`` (separate ``on_stack`` set)
|
|
|
|
**Status:** Patched
|
|
|
|
Benchmark Results
|
|
-----------------
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
* - V (grammar rules)
|
|
- Before (ops)
|
|
- After (ops)
|
|
- Ratio
|
|
* - 100
|
|
- ~5,050
|
|
- ~100
|
|
- ~50x
|
|
* - 500
|
|
- ~125,250
|
|
- ~500
|
|
- ~250x
|
|
|
|
Complexity Proof
|
|
----------------
|
|
|
|
Let V = number of grammar rule nodes. ``w in stack`` where ``stack`` is a list of up to V
|
|
elements costs O(V). Called once per edge in DFS: total O(V²) across all edges (in a dense
|
|
graph). The patched code maintains a companion ``on_stack`` set — ``w in on_stack`` is O(1).
|
|
Total: O(V + E) = O(V) for sparse grammars. QED.
|
|
|
|
References
|
|
----------
|
|
|
|
* Defect ticket: ``tools/tickets/defects/cpython-0001.md``
|
|
* Patch: ``defects/cpython/patch/``
|
|
* Tests: ``defects/cpython/unit/``, ``defects/cpython/integration/``
|