java-topology/defects/tiktoken/CLEAN.md

tiktoken: CLEAN

CWE-407 scan: 2026-03-30

Source: https://github.com/openai/tiktoken (depth=1)

Scan scope

• tiktoken/ — 6 Python files, src/ — 2 Rust files
• Focus: encoding registry membership, special token dedup, BPE merge list scanning
• Keywords: in list, linear scans

Findings

No CWE-407 defects found.

• registry.py — all lookups use dict (ENCODINGS, ENCODING_CONSTRUCTORS) — O(1)
• core.py — special_tokens_set is a set[str] (cached property) — O(1) membership
• core.py:116-119 — disallowed_special converted to frozenset before use — O(1) membership
• BPE core is implemented in Rust (src/lib.rs) using HashMap — correct
• No list-based membership tests anywhere in the Python layer

Verdict

CLEAN — minimal, well-designed codebase. All membership tests use dict/set/frozenset. Core tokenizer logic in Rust with HashMap.