java-topology/docs/tickets/redis-0001-sinter-listpack-quadratic-membership.md

redis-0001: SINTER/SDIFF on listpack sets — O(N×M) quadratic membership

Target: redis/redis Severity: HIGH CWE: CWE-407 (Inefficient Algorithmic Complexity) File: src/t_set.c:1472 (sinterGenericCommand) Status: PATCHED

Description

sinterGenericCommand iterates every element of the smallest set (outer loop, N elements) and for each element calls setTypeIsMemberAux on each of the remaining sets. When those sets use OBJ_ENCODING_LISTPACK (the default for sets with ≤ 128 entries, configured via set-max-listpack-entries), setTypeIsMemberAux dispatches to lpFind, which is an O(M) linear scan through the listpack blob.

Result: O(N × M) per intersection query instead of O(N) with a hash table.

At the default threshold of 128 entries:

• Slow (listpack): 128 × 128 = 16,384 comparisons
• Fast (hash table): 128 × O(1) = 128 comparisons
• Worst-case speedup: 128×

The same quadratic path exists in sunionDiffGenericCommand DIFF algorithm 1 (comment in source confirms "O(N*M) where N is the size of the first set and M the number of sets") when inner sets are listpack-encoded.

Hot paths

• SINTER key1 key2 [...] — intersection query
• SINTERCARD numkeys key1 key2 [LIMIT n] — cardinality of intersection
• SINTERSTORE dst key1 key2 [...] — store intersection
• SDIFF key1 key2 (algo 1) — difference query when sets are small

Root cause

src/t_set.c:1474–1477:

while((encoding = setTypeNext(&si, &str, &len, &intobj)) != -1) {
    for (j = 1; j < setnum; j++) {
        if (!setTypeIsMemberAux(sets[j].set, str, len, intobj,
                                encoding == OBJ_ENCODING_HT))

setTypeIsMemberAux at encoding OBJ_ENCODING_LISTPACK calls lpFind, which walks the packed byte array from the beginning — O(M).

Fix

Before entering the intersection loop, convert any listpack-encoded set to a temporary hash table (dictCreate) so that subsequent membership checks are O(1). The temporary dict is freed after the loop. Sets already using OBJ_ENCODING_HT or OBJ_ENCODING_INTSET are left untouched.

See patch: defects/redis/patch/0001-sinter-listpack-promote-to-htset.patch

Also affects

• valkey/valkey — identical sinterGenericCommand / sunionDiffGenericCommand code paths.

Ops ratio (unit test)

N=128, M=128:

• slow ops: 16,384
• fast ops: 128
• speedup: 128×