java-topology/docs/tickets/tidb-0002-predicate-simplification-remove-values-contains.md

tidb-0002: mergeInAndNotEQLists removeValues slices.Contains — O(N²)

Severity: MEDIUM File: pkg/planner/core/rule/rule_predicate_simplification.go Line: 267 Status: PATCHED

Description

mergeInAndNotEQLists simplifies NE/IN-list predicate pairs in the planner. It builds removeValues []int containing predicate indices to remove, then filters them out:

// O(N²) inner loop builds removeValues
for i := range predicates {
    for j := i + 1; j < len(predicates); j++ {
        // ... may append i or j to removeValues
    }
}
// O(N) loop with O(|removeValues|) check per iteration = O(N²) filter
newValues := make([]expression.Expression, 0, len(predicates))
for i, value := range predicates {
    if !(slices.Contains(removeValues, i)) {   // O(|removeValues|) scan
        newValues = append(newValues, value)
    }
}

The O(N²) inner double loop is structural (all pairs must be checked), but the final filter adds an additional O(N × |removeValues|) pass where a bitset or map[int]struct{} would make it O(N).

For a query with 100 IN-list predicates, the filter alone performs up to 10 000 index comparisons rather than 100.

Root Cause

removeValues is a []int slice used as a set. slices.Contains performs a linear scan on every iteration of the output loop.

Fix

Replace removeValues []int with removeSet map[int]struct{}:

removeSet := make(map[int]struct{})
// ... replace: removeValues = append(removeValues, i/j)
//        with: removeSet[i] = struct{}{}  /  removeSet[j] = struct{}{}

newValues := make([]expression.Expression, 0, len(predicates))
for i, value := range predicates {
    if _, skip := removeSet[i]; !skip {
        newValues = append(newValues, value)
    }
}

Speedup

Filter pass: O(N²) → O(N). At N=100 predicates: ~10 000 comparisons → ~100.