russell@unturf.com a89cc53fed transformers+vllm: 5-MOAD rescan; transformers-0004 CWE-407 all_special_ids per-loop rebuild in wav2vec2/esm

Rescan both targets against all 5 MOADs (2026-04-03).

New defect:
- transformers-0004: wav2vec2, wav2vec2_phoneme, esm tokenizers call
  self.all_special_ids/@property inside per-token decode loops, rebuilding
  list every iteration. O(T) -> O(1) fix: cache set before loop.
  wav2vec2_phoneme also has type mismatch (str vs list[int]), making
  the check always False, leaking special tokens.
  9/9 unit tests PASS.

Existing defects confirmed still present (not re-filed):
- transformers-0001/0002/0003: unchanged from 2026-03-31 scan.
- vllm-0001/0002: unchanged from 2026-03-31 scan.

MOAD-0002/0003/0004/0005: CLEAN on both targets (see SCAN-2026-04-03.md).

SCAN-TODO.md: marked transformers and vllm as complete with full summary.

Also includes UNDF stamps on jicofo-0001, jicofo-0002, langchain-0002 patches
from prior generate_undf.py run.

2026-04-03 15:32:37 -04:00

2.1 KiB

Raw Permalink Blame History

transformers-0004 — wav2vec2, wav2vec2_phoneme, esm: all_special_ids property rebuilt per loop iteration

Project: huggingface/transformers MOAD: 0001 (CWE-407 — Algorithmic Complexity) Severity: MEDIUM Status: PATCHED + TESTED (9/9 PASS)

Summary

Three tokenizers call self.all_special_ids or self.all_special_tokens inside a per-token loop or list comprehension. Both are @property methods that rebuild a list on every access (all_special_tokens iterates SPECIAL_TOKENS_ATTRIBUTES then extra_special_tokens; all_special_ids additionally calls convert_tokens_to_ids). Python evaluates the property once per loop iteration, resulting in O(T x S) work.

Affected Sites

File	Line	Pattern	Method
`src/transformers/models/wav2vec2/tokenization_wav2vec2.py`	286	`index in self.all_special_ids`	`convert_ids_to_tokens()`
`src/transformers/models/wav2vec2/tokenization_wav2vec2.py`	430	`token in self.all_special_tokens`	`_decode()`
`src/transformers/models/wav2vec2_phoneme/tokenization_wav2vec2_phoneme.py`	417	`token in self.all_special_ids`	`_decode()` (also type mismatch: str vs list[int], always False)
`src/transformers/models/esm/tokenization_esm.py`	128	`token in self.all_special_ids`	`get_special_tokens_mask()` list comprehension

Fix

Cache set(self.all_special_ids) (or set(self.all_special_tokens)) once before the loop. For the skip_special_tokens=False case, skip the set construction entirely.

For wav2vec2_phoneme: additionally fix the type mismatch by using all_special_tokens (list of strings) rather than all_special_ids (list of ints) when comparing string tokens.

Speedup

wav2vec2: S~7-15 special tokens, T=512 output tokens -> ~7680 wasted property rebuilds per decode call eliminated.

esm: S=5, T=1024 residues -> 5120 wasted property rebuild calls per get_special_tokens_mask call eliminated.

Ratio: O(T) -> O(1) property accesses per decode/mask call.

Patch

patch/transformers-0004-wav2vec2-esm-all-special-ids-loop-scan.patch

Test

test/Transformers0004Wav2Vec2EsmSpecialIdsTest.py (9 tests, 0 model downloads)

2.1 KiB Raw Permalink Blame History