java-topology/whitepaper/outreach/transformers.md

Hugging Face Transformers — CWE-407 Disclosure Brief (transformers-0001)

2026-04-13 · Patch available — awaiting upstream merge

Finding

One O(T×S) defect in Transformers' base tokenizer. convert_ids_to_tokens() accesses the all_special_ids property inside a per-token loop, rebuilding the special token list on every iteration. Patched.

The Defects

transformers-0001 (PATCHED — MEDIUM): src/transformers/tokenization_python.py

# In PreTrainedTokenizer.convert_ids_to_tokens() — fires per decode call:
for index in ids:
    index = int(index)
    if skip_special_tokens and index in self.all_special_ids:  # @property rebuilds list each call
        continue

self.all_special_ids is a @property that rebuilds a list via convert_tokens_to_ids(self.all_special_tokens) on every access. The in membership test on this list is O(S) per token. With T tokens and S special tokens, total cost is O(T × S) plus T list reconstructions.

Complexity Proof

transformers-0001: At T=4,096 tokens, S=20 special tokens:

• Defective: 4,096 list reconstructions + 4,096 × 20 = 81,920 comparisons
• Fixed: 1 set construction + 4,096 O(1) lookups
• ~20× speedup at T=4,096. Fires on every decode call.

Impact

Hugging Face Transformers is the dominant ML framework for natural language processing, used in production by thousands of companies. convert_ids_to_tokens() fires on every model output decode. Modern LLM pipelines produce sequences of 4,096+ tokens. The property rebuild cost (not just the membership test) dominates, making this a per-request overhead across all tokenizer-based inference.

The Fix

transformers-0001: Cache all_special_ids as a set before the loop:

# Before — O(T×S) + T property rebuilds
if skip_special_tokens and index in self.all_special_ids:

# After — O(T)
special_ids = set(self.all_special_ids) if skip_special_tokens else None
if special_ids is not None and index in special_ids:

Patch

Fix available: defects/transformers/patch/transformers-0001.patch

Single-file patch in src/transformers/tokenization_python.py.

What We Ask

A patch is ready for review.

1. Confirm receipt and assign a GitHub issue reference (huggingface/transformers).
2. Assess severity — fires on every decode call in all tokenizer pipelines.
3. Coordinate a disclosure date — we are targeting 90 days from first contact.
4. We will credit the Hugging Face team in the public disclosure. Preferred acknowledgment format welcome.

Contact: see cover email. This brief is confidential until coordinated disclosure.