java-topology/defects/kubernetes/patch/kubernetes-0003-job-tracking-finalizer-redundant-scan.md

2.5 KiB
Raw Blame History

UNDF: UNDF-2026-000000138

kubernetes-0003: trackJobStatusAndRemoveFinalizers — redundant hasJobTrackingFinalizer scan per pod

Severity: MEDIUM CWE: CWE-407 (Algorithmic Complexity — repeated linear membership test in reconciliation loop) Speedup: ~2x for the second pod pass; scales with pod count and finalizer list depth Target: Kubernetes (kubernetes/kubernetes) Files:

  • pkg/controller/job/job_controller.go:1357hasJobTrackingFinalizer(pod) called in a second loop over all pods, redundant with the set already built at line 1342-1347

Description

trackJobStatusAndRemoveFinalizers is the hot-path reconciliation function for batch jobs. It runs on every Job sync cycle. The function makes two full passes over jobCtx.pods:

Pass 1 (line 1343): builds uidsWithFinalizer set by calling hasJobTrackingFinalizer(p) — which itself calls slices.Contains(p.Finalizers, batch.JobTrackingFinalizer) — for every pod.

Pass 2 (line 1357): calls hasJobTrackingFinalizer(pod) again to decide whether to process each pod. This is the same O(F) linear scan already answered by membership in uidsWithFinalizer.

For a job with P pods and F finalizers per pod:

  • Current: O(2 × P × F) — two slices.Contains scans per pod per sync
  • Fixed: O(P × F + P) — one slices.Contains scan (pass 1) + O(1) map lookup (pass 2)

At P=10,000 pods (large batch ML training job), this is 10,000 redundant linear finalizer scans per reconciliation cycle.

Root Cause

// Pass 1: build set — O(P × F)
uidsWithFinalizer := make(sets.Set[types.UID], len(jobCtx.pods))
for _, p := range jobCtx.pods {
    if hasJobTrackingFinalizer(p) && ... {   // slices.Contains each time
        uidsWithFinalizer.Insert(p.UID)
    }
}

// Pass 2: redundant re-scan — O(P × F) again
for _, pod := range jobCtx.pods {
    if !hasJobTrackingFinalizer(pod) || ... {  // slices.Contains again — use uidsWithFinalizer instead
        continue
    }

Fix: in pass 2, replace hasJobTrackingFinalizer(pod) with uidsWithFinalizer.Has(pod.UID), which is already built and correct.

Patch

See kubernetes-0003-job-tracking-finalizer-redundant-scan.patch

Complexity Before

Two passes over pods: O(2 × P × F) slices.Contains calls per sync

Complexity After

One slices.Contains pass (unavoidable) + O(1) set lookup: O(P × F + P)

Reproduction

cd defects/kubernetes/unit && javac -d . *.java && java -ea unit.KubernetesTest