java-topology/whitepaper/outreach/transformers-0003.md

Hugging Face Transformers — CWE-407 Disclosure Brief (transformers-0003)

2026-04-13 · Patch available — awaiting upstream merge

Finding

Five O(T×S) defects across multiple tokenizers in Transformers. convert_tokens_to_string() checks token in self.all_special_tokens inside a per-token loop, where all_special_tokens is a @property returning a list. Patched across all affected tokenizers.

The Defects

transformers-0003 (PATCHED — MEDIUM): Multiple tokenizer files

All affected tokenizers share the same pattern:

# In convert_tokens_to_string() — fires per decode call:
for token in tokens:
    if token in self.all_special_tokens:  # @property returns list — O(S) per token
        out_string += sp_model.decode(current_sub_tokens) + token + " "

self.all_special_tokens returns a list[str] — membership test is O(S) per token. For M2M-100 translation models, S includes 100+ language-code tokens (e.g., __af__, __am__, ...), making S approximately 108.

Affected files (same pattern, same fix):

• src/transformers/models/marian/tokenization_marian.py
• src/transformers/models/m2m_100/tokenization_m2m_100.py
• src/transformers/models/speech_to_text/tokenization_speech_to_text.py
• src/transformers/models/siglip/tokenization_siglip.py
• src/transformers/models/gpt_sw3/tokenization_gpt_sw3.py

Complexity Proof

transformers-0003: At T=512 tokens, S=108 special tokens (M2M-100):

• Defective: 512 × 108 = 55,296 string comparisons per decode
• Fixed: 512 × 1 = 512 hash lookups
• ~100× speedup. Fires on every translation/transcription output.

Impact

These tokenizers serve Marian (machine translation), M2M-100 (multilingual translation), Speech2Text (ASR), SigLIP (vision-language), and GPT-SW3 (Swedish GPT). The M2M-100 tokenizer with 100+ language tokens is worst-case. Decode fires once per batch result across all inference requests for these model families.

The Fix

transformers-0003: Cache set(self.all_special_tokens) before the loop:

# Before — O(T×S)
for token in tokens:
    if token in self.all_special_tokens: ...

# After — O(T)
special_tokens_set = set(self.all_special_tokens)
for token in tokens:
    if token in special_tokens_set: ...

Patch

Fix available: defects/transformers-0003/patch/transformers-0003-convert-tokens-all-special-list-scan.patch

Multi-file patch across five tokenizer implementations.

What We Ask

A patch is ready for review.

1. Confirm receipt and assign a GitHub issue reference (huggingface/transformers).
2. Assess severity — fires on every decode call across five model families.
3. Coordinate a disclosure date — we are targeting 90 days from first contact.
4. We will credit the Hugging Face team in the public disclosure. Preferred acknowledgment format welcome.

Contact: see cover email. This brief is confidential until coordinated disclosure.