russell@unturf.com
0dbb699821
openai-python, httpx, pydantic, tiktoken, openai-node — no CWE-407 defects. All targets use proper dict/set/frozenset for membership tests in hot paths.
26 lines
895 B
Markdown
26 lines
895 B
Markdown
# tiktoken: CLEAN
|
|
|
|
CWE-407 scan: 2026-03-30
|
|
|
|
Source: https://github.com/openai/tiktoken (depth=1)
|
|
|
|
## Scan scope
|
|
|
|
- `tiktoken/` — 6 Python files, `src/` — 2 Rust files
|
|
- Focus: encoding registry membership, special token dedup, BPE merge list scanning
|
|
- Keywords: `in list`, linear scans
|
|
|
|
## Findings
|
|
|
|
No CWE-407 defects found.
|
|
|
|
- `registry.py` — all lookups use `dict` (ENCODINGS, ENCODING_CONSTRUCTORS) — O(1)
|
|
- `core.py` — `special_tokens_set` is a `set[str]` (cached property) — O(1) membership
|
|
- `core.py:116-119` — `disallowed_special` converted to `frozenset` before use — O(1) membership
|
|
- BPE core is implemented in Rust (`src/lib.rs`) using `HashMap` — correct
|
|
- No list-based membership tests anywhere in the Python layer
|
|
|
|
## Verdict
|
|
|
|
CLEAN — minimal, well-designed codebase. All membership tests use dict/set/frozenset.
|
|
Core tokenizer logic in Rust with HashMap.
|