moad-sweep: 9 initial findings — MOAD-0005 (Hungry Regex) x1, MOAD-0006 (Thundering Herd) x8
MOAD-0005 candidates (ReDoS): - bleach-MOADX-0001: sanitize_css O(2^N) — 71-char input causes 12s hang HIGH MOAD-0006 candidates (Thundering Herd / cache stampede): - hibernate-orm-MOADX-0001: QueryInterpretationCacheStandardImpl HQL plan cache HIGH - elasticsearch-MOADX-0001: EnrichCache "intentionally non-locking" enrich search HIGH - hadoop-MOADX-0001: FederationJCache/FederationCaffeineCache YARN default config HIGH - celery-MOADX-0001: BaseBackend.get_task_meta chord fan-in Redis stampede HIGH - traefik-MOADX-0001: CNAMEFlatten data race + N*30s DNS stampede HIGH - kafka-MOADX-0001: CachedConnectors.lookup classloader scan MEDIUM - spring-MOADX-0001: AbstractFallbackCacheOperationSource @Cacheable cold start MEDIUM - django-MOADX-0001: cached.Loader.get_template template compile MEDIUM CLEAN: Consul (singleflight), Kubernetes (RWMutex), Prometheus (per-scraper), Pulsar (AsyncLoadingCache), Cassandra (LoadingCache), Flink (CAS loop), ZooKeeper (synchronized), Airflow (hand-rolled singleflight)
This commit is contained in:
parent
1f48d1b89c
commit
9325c98470
9 changed files with 1111 additions and 0 deletions
|
|
@ -0,0 +1,156 @@
|
|||
# UNDF: (pending)
|
||||
# hibernate-orm-MOADX-0001: QueryInterpretationCacheStandardImpl — cache stampede O(N) concurrent plan builds on cold start
|
||||
|
||||
## MOAD-0006 Candidate — The Thundering Herd
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| ID | hibernate-orm-MOADX-0001 |
|
||||
| Severity | HIGH |
|
||||
| Ecosystem | hibernate-orm |
|
||||
| File | `hibernate-core/src/main/java/org/hibernate/query/internal/QueryInterpretationCacheStandardImpl.java` |
|
||||
| Lines | 81–103 (`resolveSelectQueryPlan`), 115–149 (`resolveHqlInterpretation`) |
|
||||
| Pattern | get → null check → compute → put (no lock, no computeIfAbsent) |
|
||||
| Trigger | N concurrent requests with same uncached query → N concurrent expensive plan builds |
|
||||
| Hot path | Per-query execution path: every `session.createQuery()` / `session.createMutationQuery()` call |
|
||||
| Compute cost | HQL → SQM AST translation + SQL query plan building (type resolution, join analysis, SQL generation strategy) |
|
||||
|
||||
## Defect
|
||||
|
||||
`QueryInterpretationCacheStandardImpl` is Hibernate ORM's query plan cache, used on every HQL/JPQL query execution. Two methods exhibit the classic check-then-act stampede pattern on the underlying `InternalCache` (`BoundedConcurrentHashMap`):
|
||||
|
||||
### `resolveSelectQueryPlan` (lines 81–103)
|
||||
|
||||
```java
|
||||
@Override
|
||||
public <K extends Key, R> SelectQueryPlan<R> resolveSelectQueryPlan(
|
||||
K key,
|
||||
Function<K, SelectQueryPlan<R>> creator) {
|
||||
@SuppressWarnings("unchecked")
|
||||
final var cached = (SelectQueryPlan<R>) queryPlanCache.get( key ); // 1. read
|
||||
if ( cached != null ) {
|
||||
return cached; // 2. hit
|
||||
}
|
||||
|
||||
final var plan = creator.apply( key ); // 3. expensive: SQL plan build
|
||||
queryPlanCache.put( key.prepareForStore(), plan ); // 4. write
|
||||
return plan;
|
||||
}
|
||||
```
|
||||
|
||||
`creator.apply(key)` calls `buildSelectQueryPlan()` → `QuerySplitter.split()` + `buildConcreteQueryPlan()` → `new ConcreteSqmSelectQueryPlan(...)`. This involves SQM polymorphism expansion, entity graph resolution, fetch strategy analysis, and SQL generation strategy computation.
|
||||
|
||||
### `resolveHqlInterpretation` (lines 115–149)
|
||||
|
||||
```java
|
||||
@Override
|
||||
public <R> HqlInterpretation<R> resolveHqlInterpretation(
|
||||
String queryString,
|
||||
Class<R> expectedResultType,
|
||||
HqlTranslator translator) {
|
||||
...
|
||||
final var existing = hqlInterpretationCache.get( cacheKey ); // 1. read
|
||||
if ( existing != null ) {
|
||||
return (HqlInterpretation<R>) existing; // 2. hit
|
||||
}
|
||||
// optional second lookup for query-only key
|
||||
final var hqlInterpretation =
|
||||
createHqlInterpretation( queryString, expectedResultType, translator, statistics ); // 3. expensive
|
||||
hqlInterpretationCache.put( cacheKey, hqlInterpretation ); // 4. write
|
||||
return hqlInterpretation;
|
||||
}
|
||||
```
|
||||
|
||||
`createHqlInterpretation()` calls `translator.translate(queryString, expectedResultType)` — full HQL lexing, parsing, AST construction, semantic analysis (type binding, relationship traversal, parameter extraction).
|
||||
|
||||
### Inconsistency
|
||||
|
||||
`resolveNativeQueryParameters` (line 192) in the *same class* correctly uses `computeIfAbsent`:
|
||||
|
||||
```java
|
||||
return nativeQueryParamCache.computeIfAbsent( queryString, creator );
|
||||
```
|
||||
|
||||
This confirms the fix was known but not applied consistently.
|
||||
|
||||
## Stampede Scenario
|
||||
|
||||
1. Application starts and first `N` concurrent HTTP requests all execute the same JPQL query (e.g. `SELECT u FROM User u WHERE u.id = :id`).
|
||||
2. All N threads call `resolveHqlInterpretation` simultaneously.
|
||||
3. All N threads see a cache miss (step 1) — the cache is empty.
|
||||
4. All N threads enter `createHqlInterpretation` simultaneously.
|
||||
5. All N threads independently: lex and parse the HQL string, build an SQM AST, resolve entity types, extract parameters.
|
||||
6. All N writes race to `put` — result is the same but work was done N times.
|
||||
7. Under moderate load (N=50 threads, ~20 query types), 1000 duplicate plan builds execute before the cache warms.
|
||||
|
||||
Same cascade applies after Hibernate's `queryPlanCacheMaxSize` evicts a plan (default 2048) — a popular query that gets evicted causes a burst of recomputes.
|
||||
|
||||
## Fix
|
||||
|
||||
Replace the get-null-compute-put triple with a single `computeIfAbsent` call, consistent with what `resolveNativeQueryParameters` already does.
|
||||
|
||||
### `resolveSelectQueryPlan`
|
||||
|
||||
```java
|
||||
@Override
|
||||
public <K extends Key, R> SelectQueryPlan<R> resolveSelectQueryPlan(
|
||||
K key,
|
||||
Function<K, SelectQueryPlan<R>> creator) {
|
||||
LOG.tracef( "Resolving cached query plan for [%s]", key );
|
||||
final var statistics = getStatistics();
|
||||
final boolean statisticsEnabled = statistics.isStatisticsEnabled();
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
final var plan = (SelectQueryPlan<R>) queryPlanCache.computeIfAbsent(
|
||||
key,
|
||||
k -> {
|
||||
if ( statisticsEnabled ) {
|
||||
statistics.queryPlanCacheMiss( k.getQueryString() );
|
||||
}
|
||||
return creator.apply( (K) k );
|
||||
});
|
||||
|
||||
if ( statisticsEnabled && plan != null ) {
|
||||
// hit was recorded if computeIfAbsent returned without calling the function
|
||||
// (we cannot distinguish hit from miss here without a flag, but misses are counted inside)
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
```
|
||||
|
||||
Or simpler — use the existing `computeIfAbsent` interface on `InternalCache`:
|
||||
|
||||
```java
|
||||
final var plan = (SelectQueryPlan<R>) queryPlanCache.computeIfAbsent(
|
||||
key,
|
||||
k -> creator.apply( (K) k ) );
|
||||
```
|
||||
|
||||
### `resolveHqlInterpretation`
|
||||
|
||||
```java
|
||||
@Override
|
||||
public <R> HqlInterpretation<R> resolveHqlInterpretation(
|
||||
String queryString,
|
||||
Class<R> expectedResultType,
|
||||
HqlTranslator translator) {
|
||||
final Object cacheKey = expectedResultType != null
|
||||
? new HqlInterpretationCacheKey( queryString, expectedResultType )
|
||||
: queryString;
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
final var result = (HqlInterpretation<R>) hqlInterpretationCache.computeIfAbsent(
|
||||
cacheKey,
|
||||
k -> createHqlInterpretation( queryString, expectedResultType, translator, getStatistics() ) );
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
Note: `InternalCache.computeIfAbsent` is already defined in the interface (line in `InternalCache.java`) and implemented in `LegacyInternalCacheImplementation` via `BoundedConcurrentHashMap.computeIfAbsent`. The infrastructure is in place — only the call sites need updating.
|
||||
|
||||
## Severity Rationale
|
||||
|
||||
- **HIGH**: Called on every query execution, not just startup. Cache misses recur after eviction.
|
||||
- Compute cost: HQL parsing + type resolution is CPU-intensive and involves synchronization inside the HQL translator.
|
||||
- At N=100 concurrent requests on a cold start (k8s pod restart, cache eviction), 100 redundant plan builds stall the thread pool simultaneously — classic thundering herd causing cascading latency.
|
||||
- The fix requires changing two call sites to use the already-present `computeIfAbsent` API.
|
||||
Loading…
Add table
Add a link
Reference in a new issue