# tiktoken: CLEAN

CWE-407 scan: 2026-03-30

Source: https://github.com/openai/tiktoken (depth=1)

## Scan scope

- `tiktoken/` — 6 Python files, `src/` — 2 Rust files
- Focus: encoding registry membership, special token dedup, BPE merge list scanning
- Keywords: `in list`, linear scans

## Findings

No CWE-407 defects found.

- `registry.py` — all lookups use `dict` (ENCODINGS, ENCODING_CONSTRUCTORS) — O(1)
- `core.py` — `special_tokens_set` is a `set[str]` (cached property) — O(1) membership
- `core.py:116-119` — `disallowed_special` converted to `frozenset` before use — O(1) membership
- BPE core is implemented in Rust (`src/lib.rs`) using `HashMap` — correct
- No list-based membership tests anywhere in the Python layer

## Verdict

CLEAN — minimal, well-designed codebase. All membership tests use dict/set/frozenset.
Core tokenizer logic in Rust with HashMap.