moad-sweep: 9 initial findings — MOAD-0005 (Hungry Regex) x1, MOAD-0006 (Thundering Herd) x8
MOAD-0005 candidates (ReDoS): - bleach-MOADX-0001: sanitize_css O(2^N) — 71-char input causes 12s hang HIGH MOAD-0006 candidates (Thundering Herd / cache stampede): - hibernate-orm-MOADX-0001: QueryInterpretationCacheStandardImpl HQL plan cache HIGH - elasticsearch-MOADX-0001: EnrichCache "intentionally non-locking" enrich search HIGH - hadoop-MOADX-0001: FederationJCache/FederationCaffeineCache YARN default config HIGH - celery-MOADX-0001: BaseBackend.get_task_meta chord fan-in Redis stampede HIGH - traefik-MOADX-0001: CNAMEFlatten data race + N*30s DNS stampede HIGH - kafka-MOADX-0001: CachedConnectors.lookup classloader scan MEDIUM - spring-MOADX-0001: AbstractFallbackCacheOperationSource @Cacheable cold start MEDIUM - django-MOADX-0001: cached.Loader.get_template template compile MEDIUM CLEAN: Consul (singleflight), Kubernetes (RWMutex), Prometheus (per-scraper), Pulsar (AsyncLoadingCache), Cassandra (LoadingCache), Flink (CAS loop), ZooKeeper (synchronized), Airflow (hand-rolled singleflight)
This commit is contained in:
parent
1f48d1b89c
commit
9325c98470
9 changed files with 1111 additions and 0 deletions
|
|
@ -0,0 +1,144 @@
|
|||
# UNDF: (pending)
|
||||
# django-MOADX-0001: cached.Loader.get_template — stampede O(N) concurrent template compilations
|
||||
|
||||
## MOAD-0006 Candidate — The Thundering Herd
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | django-MOADX-0001 |
|
||||
| Severity | MEDIUM |
|
||||
| Ecosystem | Django |
|
||||
| Language | Python |
|
||||
| File | `django/template/loaders/cached.py` |
|
||||
| Lines | 28–65 |
|
||||
| Pattern | dict.get → None check → expensive compile → dict[key]=template (no lock) |
|
||||
| Trigger | N concurrent WSGI threads all request the same template during startup / cache clear |
|
||||
| Compute cost | File system read + template lexing + compilation to AST nodes |
|
||||
|
||||
## Defect
|
||||
|
||||
Django's `cached.Loader` wraps other template loaders and caches compiled `Template` objects in `self.get_template_cache` — a plain Python `dict`. The `Loader` instance is created once per engine alias (`EngineHandler.__getitem__`) and shared across all WSGI worker threads in the process.
|
||||
|
||||
```python
|
||||
# django/template/loaders/cached.py:28-65
|
||||
def get_template(self, template_name, skip=None):
|
||||
key = self.cache_key(template_name, skip)
|
||||
cached = self.get_template_cache.get(key) # <-- dict lookup, no lock
|
||||
if cached:
|
||||
# ... return or raise from cache
|
||||
return cached
|
||||
|
||||
# NO LOCK AROUND THIS BLOCK:
|
||||
try:
|
||||
template = super().get_template(template_name, skip) # <-- file I/O + compile
|
||||
except TemplateDoesNotExist as e:
|
||||
self.get_template_cache[key] = (
|
||||
copy_exception(e) if self.engine.debug else TemplateDoesNotExist
|
||||
)
|
||||
raise
|
||||
else:
|
||||
self.get_template_cache[key] = template # <-- dict write
|
||||
|
||||
return template
|
||||
```
|
||||
|
||||
`super().get_template()` walks each configured loader, opens the template file from disk, and compiles the template string through Django's lexer and parser into a `Template` object tree. This is orders of magnitude more expensive than a cached dict lookup.
|
||||
|
||||
**CPython GIL consideration:** Individual `dict.get` and `dict[key] = value` operations are atomic under CPython's GIL. However, the three-step sequence (check → file load + compile → write) is not atomic. Threads can and do interleave between the miss check and the subsequent write:
|
||||
|
||||
- Thread A: `cache.get(key)` → `None`
|
||||
- Thread B: `cache.get(key)` → `None` (before A has written)
|
||||
- Thread A: `super().get_template(...)` → opens file, parses template
|
||||
- Thread B: `super().get_template(...)` → opens file, parses template (duplicate work)
|
||||
- Thread A: `cache[key] = template`
|
||||
- Thread B: `cache[key] = template` (redundant write)
|
||||
|
||||
**Grower signal:** Harmless at low concurrency. Triggers on:
|
||||
1. Application startup: all templates cold-cached, first request burst loads all at once.
|
||||
2. `loader.reset()` called during configuration reload: cache cleared, all threads stampede on first access.
|
||||
3. Templates with `skip` variants generate distinct cache keys, multiplying stampede surface.
|
||||
|
||||
For a Django app with 50 templates and 32 WSGI workers, a `loader.reset()` followed by a traffic burst can produce 1600 redundant file reads and parses (32 workers × 50 templates).
|
||||
|
||||
## Fix
|
||||
|
||||
**Option A — `threading.Lock` per template key (fine-grained):**
|
||||
|
||||
```python
|
||||
import threading
|
||||
|
||||
class Loader(BaseLoader):
|
||||
def __init__(self, engine, loaders):
|
||||
self.get_template_cache = {}
|
||||
self._cache_locks = {}
|
||||
self._meta_lock = threading.Lock()
|
||||
self.loaders = engine.get_template_loaders(loaders)
|
||||
super().__init__(engine)
|
||||
|
||||
def _get_or_create_key_lock(self, key):
|
||||
with self._meta_lock:
|
||||
if key not in self._cache_locks:
|
||||
self._cache_locks[key] = threading.Lock()
|
||||
return self._cache_locks[key]
|
||||
|
||||
def get_template(self, template_name, skip=None):
|
||||
key = self.cache_key(template_name, skip)
|
||||
cached = self.get_template_cache.get(key)
|
||||
if cached:
|
||||
if isinstance(cached, type) and issubclass(cached, TemplateDoesNotExist):
|
||||
raise cached(template_name)
|
||||
elif isinstance(cached, TemplateDoesNotExist):
|
||||
raise copy_exception(cached)
|
||||
return cached
|
||||
|
||||
key_lock = self._get_or_create_key_lock(key)
|
||||
with key_lock:
|
||||
# Double-check after acquiring lock
|
||||
cached = self.get_template_cache.get(key)
|
||||
if cached:
|
||||
if isinstance(cached, type) and issubclass(cached, TemplateDoesNotExist):
|
||||
raise cached(template_name)
|
||||
elif isinstance(cached, TemplateDoesNotExist):
|
||||
raise copy_exception(cached)
|
||||
return cached
|
||||
|
||||
try:
|
||||
template = super().get_template(template_name, skip)
|
||||
except TemplateDoesNotExist as e:
|
||||
self.get_template_cache[key] = (
|
||||
copy_exception(e) if self.engine.debug else TemplateDoesNotExist
|
||||
)
|
||||
raise
|
||||
else:
|
||||
self.get_template_cache[key] = template
|
||||
return template
|
||||
```
|
||||
|
||||
**Option B — coarse `threading.RLock` (simpler, acceptable given templates are rarely loaded):**
|
||||
|
||||
```python
|
||||
self._lock = threading.RLock()
|
||||
|
||||
def get_template(self, template_name, skip=None):
|
||||
key = self.cache_key(template_name, skip)
|
||||
# Fast path without lock
|
||||
cached = self.get_template_cache.get(key)
|
||||
if cached:
|
||||
...
|
||||
with self._lock:
|
||||
cached = self.get_template_cache.get(key)
|
||||
if cached:
|
||||
...
|
||||
# compile under lock — acceptable since templates load once per key
|
||||
...
|
||||
```
|
||||
|
||||
**Option C — `functools.cached_property` pattern or `threading.local` per-thread cache:** Less correct since it defeats the purpose of a shared cache.
|
||||
|
||||
Option A is preferred: per-key locking means different templates compile concurrently, only the same template key serializes.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Redundant work:** N threads × file read + template parse per cache miss burst.
|
||||
- **Resource pressure:** Multiple simultaneous file handles to the same template file; parser AST allocations multiplied by N.
|
||||
- **Fix:** After fix, first miss for each key compiles once; subsequent requests wait and receive the cached result. Net: 1 compile per unique (template_name, skip) key regardless of concurrency.
|
||||
Loading…
Add table
Add a link
Reference in a new issue