transformers-0002: MOAD-0004 (CWE-312) regnet convert script logs HF_TOKEN verbatim transformers-0003: MOAD-0001 (CWE-407) convert_tokens_to_string O(T×S) list scan - marian, m2m_100, speech_to_text, siglip, gpt_sw3 all affected - all_special_tokens is list[str]; fix: cache set() before loop; 5x speedup vllm-0002: MOAD-0001 (CWE-407) Grok2Tokenizer O(N×V) dict.values() scan - decode() and convert_ids_to_tokens() use .values() view per token - sibling Mistral tokenizer already uses frozenset correctly - fix: add _special_token_ids frozenset at __init__; 10x speedup at N=2048, V=200 MOADs 0002/0003/0005 CLEAN for both repos |
||
|---|---|---|
| .. | ||
| patch | ||
| test | ||