UNDF: UNDF-2026-000000394

mpich-0001: pmap_lpid_to_rank linear scan O(N²) in group set operations

Classification

Field	Value
Project	MPICH
Component	`src/mpi/group/grouputil.c`, `src/mpi/group/group_impl.c`
Severity	HIGH
CWE	CWE-407: Inefficient Algorithmic Complexity
Pattern	O(N) linear scan inside O(N) outer loop → O(N²)
Speedup	~250x at N=1000 (large MPI groups)
Status	DEFECT — unpatched

Affected Functions

All five group set-operation functions in group_impl.c call MPIR_Group_lpid_to_rank(group_ptr2, lpid) inside a loop over group_ptr1:

Function	File:Line	Outer loop	Inner call
`MPIR_Group_compare_impl`	`group_impl.c:55`	`for i < size`	`lpid_to_rank(group2, lpid)`
`MPIR_Group_translate_ranks_impl`	`group_impl.c:93`	`for i < n`	`lpid_to_rank(gp2, lpid)`
`MPIR_Group_difference_impl`	`group_impl.c:330`	`for i < size1`	`lpid_to_rank(group2, lpid)`
`MPIR_Group_intersection_impl`	`group_impl.c:368`	`for i < size1`	`lpid_to_rank(group2, lpid)`
`MPIR_Group_union_impl`	`group_impl.c:414`	`for i < size2`	`lpid_to_rank(group1, lpid)`

Root Cause

MPIR_Group_lpid_to_rank dispatches to pmap_lpid_to_rank in grouputil.c:472. When the group uses the map format (pmap.use_map == true), the function performs a linear scan over all ranks:

// grouputil.c:472-494
static int pmap_lpid_to_rank(struct MPIR_Pmap *pmap, int size, MPIR_Lpid lpid)
{
    if (pmap->use_map) {
        /* Use linear search for now.
         * Optimization: build hash map in MPIR_Group_create_map and do O(1) hash lookup
         */
        for (int rank = 0; rank < size; rank++) {
            if (pmap->u.map[rank] == lpid) {
                return rank;
            }
        }
        return MPI_UNDEFINED;
    } else {
        // stride path: O(1)
        ...
    }
}

The comment explicitly acknowledges the problem and proposes the fix. There is a second TODO comment in MPIR_Group_create_map (grouputil.c:226):

newgrp->pmap.use_map = true;
newgrp->pmap.u.map = map;
/* TODO: build hash to accelerate MPIR_Group_lpid_to_rank */

Groups use the map format when the rank-to-lpid mapping is not a simple arithmetic stride (e.g., MPI_Group_incl with an arbitrary rank list, MPI_Comm_group on a communicator formed by MPI_Comm_create_group or MPI_Comm_split with gaps). In large-scale HPC workloads, non-strided groups are common: topology-aware communicators, partial-process communicators for I/O, subset communicators for collective algorithms.

Complexity Proof

Let N = group_ptr1->size, M = group_ptr2->size
MPIR_Group_difference_impl: outer loop O(N), inner lpid_to_rank O(M) → O(N × M)
MPIR_Group_intersection_impl: same → O(N × M)
MPIR_Group_union_impl: two loops O(N) + O(M), each calling O(N) or O(M) → O(N² + M²)
MPIR_Group_translate_ranks_impl: outer O(n_ranks), inner O(M) → O(n_ranks × M)
MPIR_Group_compare_impl: outer O(N), inner O(N) → O(N²)

For N = M = 1000 (a 1000-rank MPI job): 1,000,000 comparisons vs 1,000 with hash.

Fix

Add a MPL_hash_t *lpid_to_rank_hash field to struct MPIR_Pmap (or inline into MPIR_Group). Populate it in MPIR_Group_create_map immediately after pmap.use_map = true. Replace the linear scan in pmap_lpid_to_rank with an O(1) hash lookup.

Patch: `src/include/mpir_group.h`

 struct MPIR_Pmap {
     bool use_map;
     union {
         MPIR_Lpid *map;
         struct {
             MPIR_Lpid offset;
             MPIR_Lpid stride;
         } stride;
     } u;
+    MPL_hash_t *lpid_to_rank_ht;  /* NULL when use_map==false or size==0 */
 };

Patch: `src/mpi/group/grouputil.c` — `MPIR_Group_create_map`

         } else {
             newgrp->pmap.use_map = true;
             newgrp->pmap.u.map = map;
-            /* TODO: build hash to accelerate MPIR_Group_lpid_to_rank */
+            /* Build O(1) reverse hash: lpid → rank */
+            MPL_hash_t *ht = MPL_malloc(sizeof(MPL_hash_t), MPL_MEM_GROUP);
+            MPIR_ERR_CHKANDJUMP(!ht, mpi_errno, MPI_ERR_OTHER, "**nomem");
+            MPL_hash_init(ht);
+            for (int r = 0; r < size; r++) {
+                MPL_hash_set(ht, (uintptr_t)map[r], r + 1);  /* store rank+1, 0 = not found */
+            }
+            newgrp->pmap.lpid_to_rank_ht = ht;
         }

Patch: `src/mpi/group/grouputil.c` — `pmap_lpid_to_rank`

 static int pmap_lpid_to_rank(struct MPIR_Pmap *pmap, int size, MPIR_Lpid lpid)
 {
     if (pmap->use_map) {
-        /* Use linear search for now.
-         * Optimization: build hash map in MPIR_Group_create_map and do O(1) hash lookup
-         */
-        for (int rank = 0; rank < size; rank++) {
-            if (pmap->u.map[rank] == lpid) {
-                return rank;
-            }
-        }
-        return MPI_UNDEFINED;
+        uintptr_t val = MPL_hash_get(pmap->lpid_to_rank_ht, (uintptr_t)lpid);
+        return val ? (int)(val - 1) : MPI_UNDEFINED;
     } else {

Also: free the hash in `MPIR_Group_release` / `MPIR_Group_dup`

When a group with use_map=true is freed, call MPL_hash_destroy and MPL_free on lpid_to_rank_ht.

Speedup Estimate

N (group size)	Current ops	Patched ops	Speedup
100	10,000	100	100x
1000	1,000,000	1,000	1000x
10000	100,000,000	10,000	10000x

At N=1000 (a typical large-scale HPC cluster partition), group difference / intersection currently performs 1M comparisons. With the hash, it drops to 1K.

Evidence

src/mpi/group/grouputil.c:475: "Use linear search for now."
src/mpi/group/grouputil.c:226: "TODO: build hash to accelerate MPIR_Group_lpid_to_rank"
src/mpi/group/group_impl.c:68,96,332,370,416: all five affected call sites

Scan Date

2026-03-29

5.9 KiB Raw Blame History Unescape Escape