# UNDF: (pending)
# hadoop-MOADX-0001: FederationJCache/FederationCaffeineCache — cache stampede on YARN Federation state store

## MOAD-0006 Candidate — The Thundering Herd

| Field | Value |
|-------|-------|
| ID | hadoop-MOADX-0001 |
| Severity | HIGH |
| Ecosystem | Apache Hadoop |
| Files | `hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java` lines 101–138; `FederationCaffeineCache.java` lines 89–125 |
| Cache type | YARN Federation state store (sub-cluster registry, policy configurations, application home sub-cluster) |
| Pattern | get → null → stateStore.getXxx() → put (no lock, no CAS) |
| Trigger | N concurrent Router requests all miss the same cache key → N×stateStore calls hit ZooKeeper/SQL simultaneously |
| Hot path | YARN Router per-request path: every application submission, every allocation, every cluster query goes through `FederationStateStoreFacade.getSubClusters()` or `getApplicationHomeSubCluster()` |

## Defect

Both `FederationJCache` and `FederationCaffeineCache` implement the same unsynchronized check-then-act pattern across three methods:

### `FederationJCache.getSubClusters` (lines 101–110)

```java
public Map<SubClusterId, SubClusterInfo> getSubClusters(boolean filterInactiveSubClusters)
    throws YarnException {
  final String cacheKey = buildCacheKey(className, GET_SUBCLUSTERS_CACHEID,
      Boolean.toString(filterInactiveSubClusters));
  CacheRequest<String, ?> cacheRequest = cache.get(cacheKey);   // (1) all N threads: miss
  if (cacheRequest == null) {
    cacheRequest = buildGetSubClustersCacheRequest(className, filterInactiveSubClusters);  // (2) N×stateStore.getSubClusters() → ZooKeeper/SQL
    cache.put(cacheKey, cacheRequest);   // (3) N redundant puts
  }
  return buildSubClusterInfoMap(cacheRequest);
}
```

The same pattern exists verbatim in:
- `FederationJCache.getPoliciesConfigurations` (lines 113–123) — calls `stateStore.getPoliciesConfigurations()`
- `FederationJCache.getApplicationHomeSubCluster` (lines 125–138) — calls `stateStore.getApplicationHomeSubCluster()`
- All three methods duplicated identically in `FederationCaffeineCache.java`

### The state store calls are network calls

`buildGetSubClustersCacheRequest` calls `stateStore.getSubClusters(request)` which in production is backed by:
- `ZookeeperFederationStateStore` — ZooKeeper read (10–50ms each)
- `SQLFederationStateStore` — SQL database query (5–100ms each)

When TTL expires (default: `yarn.federation.cache-ttl.secs` = 60s) under load, N threads all miss simultaneously and all fire the state store call.

### Default configuration makes this the production path

`YarnConfiguration.java`:
```java
public static final String FEDERATION_CACHE_CLASS_DEFAULT =
    "org.apache.hadoop.yarn.server.federation.cache.FederationJCache";
```

`FederationJCache` is used unless explicitly overridden. `FederationCaffeineCache` was added as an alternative but carries the exact same defect.

## Impact

YARN Router in a large federation (50+ sub-clusters, 10,000+ applications) issues these calls on every resource allocation request. At cache TTL expiry under high load:
- N concurrent Router threads all miss `getSubClusters`
- N ZooKeeper reads fire simultaneously
- ZooKeeper watch limit or connection pool exhausted
- Allocation requests time out
- Entire allocation path stalls until cache is populated

## Fix

### Option 1: Use `computeIfAbsent` with Caffeine LoadingCache (preferred for `FederationCaffeineCache`)

```java
// FederationCaffeineCache — replace manual check-then-act with LoadingCache
private LoadingCache<String, CacheRequest> cache;

// In initCache():
this.cache = Caffeine.newBuilder()
    .maximumSize(cacheEntityNums)
    .expireAfterWrite(cacheTimeToLive, TimeUnit.SECONDS)
    .build(key -> loadFromStateStore(key));  // loader called exactly once per key per miss

// In getSubClusters():
public Map<SubClusterId, SubClusterInfo> getSubClusters(boolean filterInactiveSubClusters)
    throws YarnException {
  final String cacheKey = buildCacheKey(className, GET_SUBCLUSTERS_CACHEID,
      Boolean.toString(filterInactiveSubClusters));
  return buildSubClusterInfoMap(cache.get(cacheKey));  // LoadingCache serializes misses per key
}
```

### Option 2: Synchronized block around miss path (for `FederationJCache`)

```java
public Map<SubClusterId, SubClusterInfo> getSubClusters(boolean filterInactiveSubClusters)
    throws YarnException {
  final String cacheKey = buildCacheKey(className, GET_SUBCLUSTERS_CACHEID,
      Boolean.toString(filterInactiveSubClusters));
  CacheRequest<String, ?> cacheRequest = cache.get(cacheKey);
  if (cacheRequest == null) {
    synchronized (this) {
      cacheRequest = cache.get(cacheKey);  // double-checked locking
      if (cacheRequest == null) {
        cacheRequest = buildGetSubClustersCacheRequest(className, filterInactiveSubClusters);
        cache.put(cacheKey, cacheRequest);
      }
    }
  }
  return buildSubClusterInfoMap(cacheRequest);
}
```

Apply the same fix to `getPoliciesConfigurations` and `getApplicationHomeSubCluster` in both `FederationJCache` and `FederationCaffeineCache`.
