no-stone-unturned wave: 8 new defects, 15 CLEAN confirmations; count 621→629
New defects (all PASS): - exim-0001: same_hosts() MX-segment O(H²) → AVL set O(H log H), 10.5x at H=20 - minecraft-0001: DependencySorter.isCyclic no visited set O(E^D) → O(E), 342,000x at D=24 - minecraft-0002: PistonStructureResolver toPush ArrayList O(N²) → HashSet O(N) - minecraft-0003: RedstoneWireEvaluator Deque.contains O(N²) → HashSet O(N) - minecraft-0004: MoveThroughVillageGoal visited List O(N²) → HashSet O(N) - mpich-0001: group_lpid_to_rank O(N²) → HashMap O(N), 313x at N=1000 - ompi-0001: group_overlap process-name scan O(N×M) → HashMap O(N+M), 2048x - pcl-0001: RegionGrowing::getSegmentFromPoint O(C×S) → point_labels[] O(1), 50000x CLEAN confirmed: esbuild, express, koa, ktor, lucene, mpich-recvq, ompi-startup, prosody, roda, rust/rustc-wave2, signal-server, solana, wiredtiger, wireguard-tools, linux-kernel (pointer to linux/)
This commit is contained in:
parent
dd72c2ba0d
commit
a629bd0bbf
46 changed files with 2687 additions and 128 deletions
184
defects/ompi/patch/ompi-0001-group-ops-process-name-hashmap.md
Normal file
184
defects/ompi/patch/ompi-0001-group-ops-process-name-hashmap.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
# ompi-0001: ompi_group O(N×M) nested process-name scan in group set operations
|
||||
|
||||
## Classification
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Project | Open MPI |
|
||||
| Component | `ompi/group/group.c` |
|
||||
| Severity | HIGH |
|
||||
| CWE | CWE-407: Inefficient Algorithmic Complexity |
|
||||
| Pattern | O(N) process-name scan inside O(N) outer loop → O(N×M) |
|
||||
| Speedup | ~250x at N=M=500 (large MPI groups, non-sparse path) |
|
||||
| Status | DEFECT — unpatched |
|
||||
|
||||
## Affected Functions
|
||||
|
||||
Three functions in `ompi/group/group.c` use nested loops over
|
||||
`ompi_group_get_proc_name` + `opal_compare_proc` with no hash acceleration
|
||||
on the dense (non-sparse) code path:
|
||||
|
||||
| Function | File:Line | Complexity |
|
||||
|----------|-----------|-----------|
|
||||
| `ompi_group_translate_ranks` | `group.c:47` (fallback at ~105) | O(n_ranks × M) |
|
||||
| `ompi_group_intersection` | `group.c:444` | O(N × M) |
|
||||
| `ompi_group_overlap` | `group.c:629` | O(N × M) |
|
||||
|
||||
A fourth function `ompi_group_compare` (group.c:491) contains the same
|
||||
nested loop pattern:
|
||||
|
||||
| Function | File:Line | Complexity |
|
||||
|----------|-----------|-----------|
|
||||
| `ompi_group_compare` | `group.c:491` | O(N²) |
|
||||
|
||||
## Root Cause
|
||||
|
||||
### `ompi_group_translate_ranks` (dense fallback, group.c:105–130)
|
||||
|
||||
When neither group is the parent of the other (the sparse fast-path is
|
||||
skipped), the function falls through to a dense O(n_ranks × M) scan:
|
||||
|
||||
```c
|
||||
/* loop over all ranks */
|
||||
for (int proc = 0; proc < n_ranks; ++proc) {
|
||||
ompi_process_name_t proc1_name, proc2_name;
|
||||
int rank = ranks1[proc];
|
||||
proc1_name = ompi_group_get_proc_name(group1, rank);
|
||||
ranks2[proc] = MPI_UNDEFINED;
|
||||
for (int proc2 = 0; proc2 < group2->grp_proc_count; ++proc2) {
|
||||
proc2_name = ompi_group_get_proc_name(group2, proc2);
|
||||
if(0 == opal_compare_proc(proc1_name, proc2_name)) {
|
||||
ranks2[proc] = proc2;
|
||||
break;
|
||||
}
|
||||
} /* end proc2 loop */
|
||||
} /* end proc loop */
|
||||
```
|
||||
|
||||
### `ompi_group_intersection` (group.c:464–484)
|
||||
|
||||
Same double loop:
|
||||
|
||||
```c
|
||||
for (proc1 = 0; proc1 < group1_pointer->grp_proc_count; proc1++) {
|
||||
proc1_name = ompi_group_get_proc_name(group1_pointer, proc1);
|
||||
for (proc2 = 0; proc2 < group2_pointer->grp_proc_count; proc2++) {
|
||||
proc2_name = ompi_group_get_proc_name(group2_pointer, proc2);
|
||||
if(0 == opal_compare_proc(proc1_name, proc2_name)) {
|
||||
ranks_included[k] = proc1;
|
||||
k++;
|
||||
break;
|
||||
}
|
||||
} /* end proc2 loop */
|
||||
} /* end proc1 loop */
|
||||
```
|
||||
|
||||
### `ompi_group_overlap` (group.c:629–640)
|
||||
|
||||
```c
|
||||
for (int i = 0 ; i < group1->grp_proc_count ; ++i) {
|
||||
opal_process_name_t proc1 = ompi_group_get_proc_name(group1, i);
|
||||
for (int j = 0 ; j < group2->grp_proc_count ; ++j) {
|
||||
opal_process_name_t proc2 = ompi_group_get_proc_name(group2, j);
|
||||
if (0 == opal_compare_proc (proc1, proc2)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The sparse group optimisation (`#if OMPI_GROUP_SPARSE`) exists for
|
||||
parent-child relationships (e.g., a subgroup derived directly from a
|
||||
parent communicator). However when two **independent** groups are compared —
|
||||
the typical case in MPI collective communicator creation, multi-communicator
|
||||
algorithms, and fault-tolerance operations — the sparse fast-path is skipped
|
||||
and the O(N×M) path executes.
|
||||
|
||||
## Complexity Proof
|
||||
|
||||
- N = `group1->grp_proc_count`, M = `group2->grp_proc_count`
|
||||
- `ompi_group_translate_ranks` (dense path): O(n_ranks × M)
|
||||
- `ompi_group_intersection`: O(N × M) comparisons
|
||||
- `ompi_group_overlap`: O(N × M) comparisons, worst case
|
||||
|
||||
For N = M = 500 (a 500-rank MPI group): 250,000 name comparisons vs 500 with hash.
|
||||
|
||||
`opal_compare_proc` compares two `opal_process_name_t` structs (jobid + vpid),
|
||||
typically an 8-byte comparison — lightweight but executed N×M times.
|
||||
|
||||
## Fix
|
||||
|
||||
Build an `opal_hash_table_t` keyed on `opal_process_name_t` (encoded as
|
||||
`uint64_t`) mapping to rank index for `group2` before entering the outer loop.
|
||||
Replace the inner `for (proc2...)` with a single O(1) hash lookup.
|
||||
|
||||
### Patch: `ompi/group/group.c` — `ompi_group_translate_ranks` fallback
|
||||
|
||||
```diff
|
||||
+ /* Build reverse hash for group2: proc_name_key -> rank */
|
||||
+ opal_hash_table_t *g2_ht = OBJ_NEW(opal_hash_table_t);
|
||||
+ opal_hash_table_init(g2_ht, group2->grp_proc_count * 2);
|
||||
+ for (int i = 0; i < group2->grp_proc_count; i++) {
|
||||
+ opal_process_name_t n = ompi_group_get_proc_name(group2, i);
|
||||
+ uint64_t key = (((uint64_t)n.jobid) << 32) | n.vpid;
|
||||
+ opal_hash_table_set_value_uint64(g2_ht, key, (void *)(uintptr_t)(i + 1));
|
||||
+ }
|
||||
+
|
||||
/* loop over all ranks */
|
||||
for (int proc = 0; proc < n_ranks; ++proc) {
|
||||
ompi_process_name_t proc1_name;
|
||||
int rank = ranks1[proc];
|
||||
if ( MPI_PROC_NULL == rank) {
|
||||
ranks2[proc] = MPI_PROC_NULL;
|
||||
continue;
|
||||
}
|
||||
proc1_name = ompi_group_get_proc_name(group1, rank);
|
||||
- ranks2[proc] = MPI_UNDEFINED;
|
||||
- for (int proc2 = 0; proc2 < group2->grp_proc_count; ++proc2) {
|
||||
- proc2_name = ompi_group_get_proc_name(group2, proc2);
|
||||
- if(0 == opal_compare_proc(proc1_name, proc2_name)) {
|
||||
- ranks2[proc] = proc2;
|
||||
- break;
|
||||
- }
|
||||
- } /* end proc2 loop */
|
||||
+ uint64_t key = (((uint64_t)proc1_name.jobid) << 32) | proc1_name.vpid;
|
||||
+ void *val = NULL;
|
||||
+ opal_hash_table_get_value_uint64(g2_ht, key, &val);
|
||||
+ ranks2[proc] = val ? (int)((uintptr_t)val - 1) : MPI_UNDEFINED;
|
||||
} /* end proc loop */
|
||||
+
|
||||
+ OBJ_RELEASE(g2_ht);
|
||||
```
|
||||
|
||||
The same pattern applies to `ompi_group_intersection` and
|
||||
`ompi_group_overlap`: build a hash of group2 process names before the outer
|
||||
loop, replace the inner loop with a single lookup.
|
||||
|
||||
## Speedup Estimate
|
||||
|
||||
| N = M (group size) | Current comparisons | Patched | Speedup |
|
||||
|--------------------|---------------------|---------|---------|
|
||||
| 100 | 10,000 | 200 | 50x |
|
||||
| 500 | 250,000 | 1,000 | 250x |
|
||||
| 2000 | 4,000,000 | 4,000 | 1000x |
|
||||
|
||||
In a 2000-rank collective algorithm that calls `MPI_Group_translate_ranks`
|
||||
during communicator setup, this reduces 4M name comparisons to 4K.
|
||||
|
||||
## Relationship to Existing Sparse Optimisation
|
||||
|
||||
The existing `#if OMPI_GROUP_SPARSE` fast-path handles only parent-child
|
||||
group relationships (a subgroup derived from a parent communicator). The
|
||||
O(N×M) path triggers for **all other combinations**, which includes any two
|
||||
independently-created groups. The hash fix handles the general case and
|
||||
does not conflict with the sparse path.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `ompi/group/group.c:105–130`: nested loop in `ompi_group_translate_ranks`
|
||||
- `ompi/group/group.c:464–484`: nested loop in `ompi_group_intersection`
|
||||
- `ompi/group/group.c:629–640`: nested loop in `ompi_group_overlap`
|
||||
- `ompi/group/group.c:527–544`: nested loop in `ompi_group_compare`
|
||||
|
||||
## Scan Date
|
||||
2026-03-29
|
||||
|
|
@ -0,0 +1,108 @@
|
|||
# UNDF: (pending)
|
||||
--- a/ompi/group/group.c
|
||||
+++ b/ompi/group/group.c
|
||||
@@ -98,6 +98,17 @@ int ompi_group_translate_ranks ( ompi_group_t *group1,
|
||||
#endif
|
||||
|
||||
+ /* Build reverse-hash of group2: (jobid<<32|vpid) -> rank+1 */
|
||||
+ opal_hash_table_t *g2_ht = OBJ_NEW(opal_hash_table_t);
|
||||
+ if (NULL == g2_ht) return MPI_ERR_NO_MEM;
|
||||
+ opal_hash_table_init(g2_ht, group2->grp_proc_count * 2 + 1);
|
||||
+ for (int _i = 0; _i < group2->grp_proc_count; _i++) {
|
||||
+ opal_process_name_t _n = ompi_group_get_proc_name(group2, _i);
|
||||
+ uint64_t _k = (((uint64_t)_n.jobid) << 32) | (uint64_t)_n.vpid;
|
||||
+ opal_hash_table_set_value_uint64(g2_ht, _k, (void *)(uintptr_t)(_i + 1));
|
||||
+ }
|
||||
+
|
||||
/* loop over all ranks */
|
||||
for (int proc = 0; proc < n_ranks; ++proc) {
|
||||
ompi_process_name_t proc1_name, proc2_name;
|
||||
@@ -110,17 +121,13 @@ int ompi_group_translate_ranks ( ompi_group_t *group1,
|
||||
proc1_name = ompi_group_get_proc_name(group1, rank);
|
||||
/* initialize to no "match" */
|
||||
ranks2[proc] = MPI_UNDEFINED;
|
||||
- for (int proc2 = 0; proc2 < group2->grp_proc_count; ++proc2) {
|
||||
- proc2_name = ompi_group_get_proc_name(group2, proc2);
|
||||
- if(0 == opal_compare_proc(proc1_name, proc2_name)) {
|
||||
- ranks2[proc] = proc2;
|
||||
- break;
|
||||
- }
|
||||
- } /* end proc2 loop */
|
||||
+ uint64_t _k = (((uint64_t)proc1_name.jobid) << 32) | (uint64_t)proc1_name.vpid;
|
||||
+ void *_v = NULL;
|
||||
+ opal_hash_table_get_value_uint64(g2_ht, _k, &_v);
|
||||
+ ranks2[proc] = _v ? (int)((uintptr_t)_v - 1) : MPI_UNDEFINED;
|
||||
} /* end proc loop */
|
||||
|
||||
+ OBJ_RELEASE(g2_ht);
|
||||
return MPI_SUCCESS;
|
||||
}
|
||||
|
||||
@@ -453,6 +460,17 @@ int ompi_group_intersection(ompi_group_t* group1,ompi_group_t* group2,
|
||||
k = 0;
|
||||
+
|
||||
+ /* Build reverse-hash of group2 for O(1) membership test */
|
||||
+ opal_hash_table_t *g2_ht = OBJ_NEW(opal_hash_table_t);
|
||||
+ if (NULL == g2_ht) { free(ranks_included); return MPI_ERR_NO_MEM; }
|
||||
+ opal_hash_table_init(g2_ht, group2_pointer->grp_proc_count * 2 + 1);
|
||||
+ for (int _i = 0; _i < group2_pointer->grp_proc_count; _i++) {
|
||||
+ opal_process_name_t _n = ompi_group_get_proc_name(group2_pointer, _i);
|
||||
+ uint64_t _k = (((uint64_t)_n.jobid) << 32) | (uint64_t)_n.vpid;
|
||||
+ opal_hash_table_set_value_uint64(g2_ht, _k, (void *)(uintptr_t)1);
|
||||
+ }
|
||||
+
|
||||
/* determine the list of included processes for the incl-method */
|
||||
for (proc1 = 0; proc1 < group1_pointer->grp_proc_count; proc1++) {
|
||||
proc1_name = ompi_group_get_proc_name(group1_pointer , proc1);
|
||||
- /* check to see if this proc is in group2 */
|
||||
- for (proc2 = 0; proc2 < group2_pointer->grp_proc_count; proc2++) {
|
||||
- proc2_name = ompi_group_get_proc_name(group2_pointer ,proc2);
|
||||
- if(0 == opal_compare_proc(proc1_name, proc2_name)) {
|
||||
- ranks_included[k] = proc1;
|
||||
- k++;
|
||||
- break;
|
||||
- }
|
||||
- } /* end proc2 loop */
|
||||
+ uint64_t _k = (((uint64_t)proc1_name.jobid) << 32) | (uint64_t)proc1_name.vpid;
|
||||
+ void *_v = NULL;
|
||||
+ opal_hash_table_get_value_uint64(g2_ht, _k, &_v);
|
||||
+ if (_v) { ranks_included[k++] = proc1; }
|
||||
} /* end proc1 loop */
|
||||
|
||||
+ OBJ_RELEASE(g2_ht);
|
||||
result = ompi_group_incl(group1, k, ranks_included, new_group);
|
||||
|
||||
@@ -629,14 +643,20 @@ bool ompi_group_overlap (const ompi_group_t *group1, const ompi_group_t *group2)
|
||||
{
|
||||
- for (int i = 0 ; i < group1->grp_proc_count ; ++i) {
|
||||
- opal_process_name_t proc1 = ompi_group_get_proc_name (group1, i);
|
||||
- for (int j = 0 ; j < group2->grp_proc_count ; ++j) {
|
||||
- opal_process_name_t proc2 = ompi_group_get_proc_name (group2, j);
|
||||
- if (0 == opal_compare_proc (proc1, proc2)) {
|
||||
- return true;
|
||||
- }
|
||||
- }
|
||||
- }
|
||||
- return false;
|
||||
+ /* Build hash of smaller group for O(N+M) instead of O(N*M) */
|
||||
+ const ompi_group_t *small = (group1->grp_proc_count <= group2->grp_proc_count) ? group1 : group2;
|
||||
+ const ompi_group_t *large = (small == group1) ? group2 : group1;
|
||||
+ opal_hash_table_t *ht = OBJ_NEW(opal_hash_table_t);
|
||||
+ if (NULL == ht) return false; /* conservative: assume no overlap on OOM */
|
||||
+ opal_hash_table_init(ht, small->grp_proc_count * 2 + 1);
|
||||
+ for (int i = 0; i < small->grp_proc_count; i++) {
|
||||
+ opal_process_name_t n = ompi_group_get_proc_name(small, i);
|
||||
+ uint64_t k = (((uint64_t)n.jobid) << 32) | (uint64_t)n.vpid;
|
||||
+ opal_hash_table_set_value_uint64(ht, k, (void *)(uintptr_t)1);
|
||||
+ }
|
||||
+ bool found = false;
|
||||
+ for (int i = 0; i < large->grp_proc_count && !found; i++) {
|
||||
+ opal_process_name_t n = ompi_group_get_proc_name(large, i);
|
||||
+ uint64_t k = (((uint64_t)n.jobid) << 32) | (uint64_t)n.vpid;
|
||||
+ void *v = NULL;
|
||||
+ opal_hash_table_get_value_uint64(ht, k, &v);
|
||||
+ found = (v != NULL);
|
||||
+ }
|
||||
+ OBJ_RELEASE(ht);
|
||||
+ return found;
|
||||
}
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
# ompi: CWE-407 scan result — CLEAN
|
||||
|
||||
## Scan Date
|
||||
2026-03-27
|
||||
|
||||
## Scope
|
||||
- `ompi/mca/` — MCA component registration and lookup
|
||||
- `opal/class/` — data structure implementations
|
||||
- `ompi/communicator/` — communicator management
|
||||
- `opal/mca/base/` — component find, repository, alias, var systems
|
||||
|
||||
## Findings
|
||||
|
||||
### Linear patterns found
|
||||
Multiple `OPAL_LIST_FOREACH` + `strcmp` patterns exist in component selection
|
||||
(`btl_base_select.c`, `mca_base_component_find.c`, `mca_base_component_repository.c`,
|
||||
etc.), but **none meet the CWE-407 threshold**:
|
||||
|
||||
1. **Component selection at startup** (`btl_base_select.c:71-96`): outer loop
|
||||
over M registered components (M ≤ ~20), inner `while` over N requested
|
||||
names (N ≤ user CLI argc). Called once at MPI_Init. O(M × N) ≈ O(400).
|
||||
Not a performance defect.
|
||||
|
||||
2. **`component_find_check`** (`mca_base_component_find.c:336-373`): outer
|
||||
loop over N requested names, inner `OPAL_LIST_FOREACH` over M components.
|
||||
Same analysis: both bounds are tiny and the function runs once at startup.
|
||||
|
||||
3. **`mca_base_component_repository_open`** (`mca_base_component_repository.c:388`):
|
||||
single O(M) scan to check for duplicate component load. Called once per
|
||||
component at startup. Not a hot path.
|
||||
|
||||
### Hash tables already present
|
||||
The MCA base layer uses `opal_hash_table` for variable/group/pvar/alias
|
||||
lookups (`mca_base_var.c`, `mca_base_alias.c`, `mca_base_component_repository.c`).
|
||||
The component *repository* is hash-indexed by framework name. Only the
|
||||
per-framework `framework_components` linked list uses linear scan, and that
|
||||
list is always small (< 20 entries).
|
||||
|
||||
## Verdict
|
||||
**CLEAN** — no CWE-407 defect. All linear membership tests occur in
|
||||
one-time startup paths over bounded-small sets.
|
||||
329
defects/ompi/unit/OmpiGroupTest.java
Normal file
329
defects/ompi/unit/OmpiGroupTest.java
Normal file
|
|
@ -0,0 +1,329 @@
|
|||
package unit;
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* OmpiGroupTest — CWE-407 benchmark for ompi-0001
|
||||
*
|
||||
* ompi-0001: ompi_group O(N×M) nested process-name scan in group set operations
|
||||
* SLOW: O(N×M) — nested for-loops comparing opal_process_name_t structs
|
||||
* FAST: O(N+M) — HashMap keyed on (jobid<<32|vpid) built once from group2
|
||||
*
|
||||
* Affected functions in ompi/group/group.c:
|
||||
* - ompi_group_translate_ranks (dense fallback at line 105) O(n_ranks × M)
|
||||
* - ompi_group_intersection (line 444) O(N × M)
|
||||
* - ompi_group_overlap (line 629) O(N × M)
|
||||
* - ompi_group_compare (line 491) O(N²)
|
||||
*
|
||||
* The sparse fast-path (#if OMPI_GROUP_SPARSE) only applies to parent-child
|
||||
* group relationships. Independent groups (the common case in MPI collective
|
||||
* communicator creation) always hit the O(N×M) path.
|
||||
*/
|
||||
public class OmpiGroupTest {
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Benchmark harness
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
static void bench(String label, Runnable slow, Runnable fast, long sOps, long fOps) {
|
||||
slow.run(); fast.run();
|
||||
long t0 = System.nanoTime(); slow.run(); long sMs = (System.nanoTime() - t0) / 1_000_000;
|
||||
long t1 = System.nanoTime(); fast.run(); long fMs = (System.nanoTime() - t1) / 1_000_000;
|
||||
System.out.printf(" %-60s slow:%5dms (%,d ops) fast:%5dms (%,d ops) speedup:%.0fx%n",
|
||||
label, sMs, sOps, fMs, fOps, fOps > 0 ? (double) sOps / fOps : 0);
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Model
|
||||
//
|
||||
// An MPI process is identified by (jobid, vpid) — an opal_process_name_t.
|
||||
// We model this as a long: (jobid << 32) | vpid.
|
||||
//
|
||||
// A group is an array of such process names.
|
||||
//
|
||||
// ompi_group_translate_ranks (dense path) does:
|
||||
// for proc = 0 .. n_ranks-1:
|
||||
// name1 = group1[ranks[proc]]
|
||||
// for proc2 = 0 .. group2.size-1:
|
||||
// name2 = group2[proc2]
|
||||
// if opal_compare_proc(name1, name2) == 0:
|
||||
// result[proc] = proc2; break
|
||||
//
|
||||
// ompi_group_intersection does:
|
||||
// for proc1 = 0 .. group1.size-1:
|
||||
// name1 = group1[proc1]
|
||||
// for proc2 = 0 .. group2.size-1:
|
||||
// name2 = group2[proc2]
|
||||
// if opal_compare_proc(name1, name2) == 0:
|
||||
// included[k++] = proc1; break
|
||||
//
|
||||
// ompi_group_overlap does:
|
||||
// for i = 0 .. group1.size-1:
|
||||
// for j = 0 .. group2.size-1:
|
||||
// if same_proc: return true
|
||||
// return false
|
||||
//
|
||||
// opal_compare_proc compares two 64-bit process names: one comparison each.
|
||||
// =========================================================================
|
||||
|
||||
// --- SLOW: ompi_group_translate_ranks dense fallback O(n_ranks × M) ---
|
||||
|
||||
static long translateRanksSlow(long[] group1, int[] ranks, long[] group2) {
|
||||
long ops = 0;
|
||||
int[] result = new int[ranks.length];
|
||||
for (int i = 0; i < ranks.length; i++) {
|
||||
long name1 = group1[ranks[i]];
|
||||
result[i] = -1;
|
||||
for (int proc2 = 0; proc2 < group2.length; proc2++) {
|
||||
ops++;
|
||||
if (group2[proc2] == name1) { result[i] = proc2; break; }
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// --- FAST: O(n_ranks + M) with hash ---
|
||||
|
||||
static long translateRanksFast(long[] group1, int[] ranks, long[] group2) {
|
||||
long ops = 0;
|
||||
// Build reverse map: process_name -> rank in group2
|
||||
Map<Long, Integer> revMap = new HashMap<>(group2.length * 2);
|
||||
for (int r = 0; r < group2.length; r++) {
|
||||
revMap.put(group2[r], r);
|
||||
ops++;
|
||||
}
|
||||
int[] result = new int[ranks.length];
|
||||
for (int i = 0; i < ranks.length; i++) {
|
||||
long name1 = group1[ranks[i]];
|
||||
ops++;
|
||||
result[i] = revMap.getOrDefault(name1, -1);
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// --- SLOW: ompi_group_intersection O(N × M) ---
|
||||
|
||||
static long intersectionSlow(long[] group1, long[] group2) {
|
||||
long ops = 0;
|
||||
List<Integer> included = new ArrayList<>();
|
||||
for (int proc1 = 0; proc1 < group1.length; proc1++) {
|
||||
long name1 = group1[proc1];
|
||||
for (int proc2 = 0; proc2 < group2.length; proc2++) {
|
||||
ops++;
|
||||
if (group2[proc2] == name1) { included.add(proc1); break; }
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// --- FAST: ompi_group_intersection O(N + M) with hash ---
|
||||
|
||||
static long intersectionFast(long[] group1, long[] group2) {
|
||||
long ops = 0;
|
||||
Set<Long> g2Set = new HashSet<>(group2.length * 2);
|
||||
for (long name : group2) { g2Set.add(name); ops++; }
|
||||
List<Integer> included = new ArrayList<>();
|
||||
for (int proc1 = 0; proc1 < group1.length; proc1++) {
|
||||
ops++;
|
||||
if (g2Set.contains(group1[proc1])) included.add(proc1);
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// --- SLOW: ompi_group_overlap O(N × M) ---
|
||||
|
||||
static long overlapSlow(long[] group1, long[] group2) {
|
||||
long ops = 0;
|
||||
for (long name1 : group1) {
|
||||
for (long name2 : group2) {
|
||||
ops++;
|
||||
if (name1 == name2) return ops;
|
||||
}
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// --- FAST: ompi_group_overlap O(N + M) ---
|
||||
|
||||
static long overlapFast(long[] group1, long[] group2) {
|
||||
long ops = 0;
|
||||
// Hash the smaller group
|
||||
long[] small = group1.length <= group2.length ? group1 : group2;
|
||||
long[] large = small == group1 ? group2 : group1;
|
||||
Set<Long> smallSet = new HashSet<>(small.length * 2);
|
||||
for (long name : small) { smallSet.add(name); ops++; }
|
||||
for (long name : large) {
|
||||
ops++;
|
||||
if (smallSet.contains(name)) return ops;
|
||||
}
|
||||
return ops;
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Correctness verification
|
||||
// =========================================================================
|
||||
|
||||
static boolean PASS = true;
|
||||
|
||||
static void assertIntersectionEqual(String test, long[] group1, long[] group2) {
|
||||
// Collect intersection ranks using both methods
|
||||
List<Integer> slowResult = new ArrayList<>();
|
||||
for (int p1 = 0; p1 < group1.length; p1++) {
|
||||
boolean found = false;
|
||||
for (long n2 : group2) if (group1[p1] == n2) { found = true; break; }
|
||||
if (found) slowResult.add(p1);
|
||||
}
|
||||
Set<Long> g2Set = new HashSet<>();
|
||||
for (long n : group2) g2Set.add(n);
|
||||
List<Integer> fastResult = new ArrayList<>();
|
||||
for (int p1 = 0; p1 < group1.length; p1++) {
|
||||
if (g2Set.contains(group1[p1])) fastResult.add(p1);
|
||||
}
|
||||
boolean ok = slowResult.equals(fastResult);
|
||||
System.out.printf(" CORRECTNESS %-40s %s%n", test, ok ? "PASS" : "FAIL");
|
||||
if (!ok) PASS = false;
|
||||
}
|
||||
|
||||
static void assertOverlapEqual(String test, long[] group1, long[] group2, boolean expected) {
|
||||
boolean slowHas = overlapSlow(group1, group2) > 0 && hasOverlap(group1, group2);
|
||||
boolean fastHas = overlapFast(group1, group2) > 0 && hasOverlap(group1, group2);
|
||||
boolean ok = slowHas == fastHas && slowHas == expected;
|
||||
System.out.printf(" CORRECTNESS %-40s %s (expected=%b actual=%b)%n",
|
||||
test, ok ? "PASS" : "FAIL", expected, slowHas);
|
||||
if (!ok) PASS = false;
|
||||
}
|
||||
|
||||
static boolean hasOverlap(long[] g1, long[] g2) {
|
||||
Set<Long> s = new HashSet<>();
|
||||
for (long x : g2) s.add(x);
|
||||
for (long x : g1) if (s.contains(x)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
static void assertTranslateEqual(String test, long[] group1, int[] ranks, long[] group2) {
|
||||
// slow
|
||||
int[] slowResult = new int[ranks.length];
|
||||
for (int i = 0; i < ranks.length; i++) {
|
||||
long name1 = group1[ranks[i]];
|
||||
slowResult[i] = -1;
|
||||
for (int r = 0; r < group2.length; r++) {
|
||||
if (group2[r] == name1) { slowResult[i] = r; break; }
|
||||
}
|
||||
}
|
||||
// fast
|
||||
Map<Long, Integer> revMap = new HashMap<>();
|
||||
for (int r = 0; r < group2.length; r++) revMap.put(group2[r], r);
|
||||
int[] fastResult = new int[ranks.length];
|
||||
for (int i = 0; i < ranks.length; i++) {
|
||||
fastResult[i] = revMap.getOrDefault(group1[ranks[i]], -1);
|
||||
}
|
||||
boolean ok = Arrays.equals(slowResult, fastResult);
|
||||
System.out.printf(" CORRECTNESS %-40s %s%n", test, ok ? "PASS" : "FAIL");
|
||||
if (!ok) PASS = false;
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Main
|
||||
// =========================================================================
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.out.println("=== ompi-0001: ompi_group O(N×M) nested process-name scan ===");
|
||||
System.out.println();
|
||||
|
||||
// Build process-name arrays: jobid=0, vpid=rank (single job, typical case)
|
||||
// Non-overlapping portion: g1 = procs 0..N-1, g2 = procs N/2..3N/2-1
|
||||
|
||||
System.out.println("--- Correctness ---");
|
||||
|
||||
// Intersection tests
|
||||
long[] ga = {10L, 20L, 30L, 40L, 50L};
|
||||
long[] gb = {20L, 40L, 60L, 80L};
|
||||
assertIntersectionEqual("intersection {10,20,30,40,50} ∩ {20,40,60,80}", ga, gb);
|
||||
|
||||
long[] gc = {1L, 2L, 3L, 4L};
|
||||
long[] gd = {5L, 6L, 7L, 8L};
|
||||
assertIntersectionEqual("disjoint groups: intersection = empty", gc, gd);
|
||||
|
||||
long[] ge = {1L, 2L, 3L};
|
||||
long[] gf = {1L, 2L, 3L};
|
||||
assertIntersectionEqual("identical groups: intersection = group1", ge, gf);
|
||||
|
||||
// Overlap tests
|
||||
assertOverlapEqual("overlap: groups share proc 20", ga, gb, true);
|
||||
assertOverlapEqual("overlap: disjoint groups", gc, gd, false);
|
||||
|
||||
// Translate ranks tests
|
||||
long[] grp1 = {100L, 200L, 300L, 400L, 500L};
|
||||
long[] grp2 = {200L, 400L, 600L, 800L};
|
||||
int[] ranks = {0, 1, 2, 3, 4};
|
||||
assertTranslateEqual("translate_ranks: partial overlap", grp1, ranks, grp2);
|
||||
|
||||
System.out.println();
|
||||
System.out.println("--- Performance ---");
|
||||
|
||||
int[] sizes = {100, 500, 1000};
|
||||
for (int N : sizes) {
|
||||
final long[] g1 = new long[N], g2 = new long[N];
|
||||
// jobid=1, vpid=i for group1; jobid=1, vpid=i+N/2 for group2 (partial overlap)
|
||||
for (int i = 0; i < N; i++) {
|
||||
g1[i] = (1L << 32) | i;
|
||||
g2[i] = (1L << 32) | (i + N / 2);
|
||||
}
|
||||
int[] allRanks = new int[N];
|
||||
for (int i = 0; i < N; i++) allRanks[i] = i;
|
||||
|
||||
final long[] slowOps = {0}, fastOps = {0};
|
||||
|
||||
bench(
|
||||
String.format("group_translate_ranks N=%d", N),
|
||||
() -> slowOps[0] = translateRanksSlow(g1, allRanks, g2),
|
||||
() -> fastOps[0] = translateRanksFast(g1, allRanks, g2),
|
||||
slowOps[0], fastOps[0]
|
||||
);
|
||||
|
||||
bench(
|
||||
String.format("group_intersection N=%d", N),
|
||||
() -> slowOps[0] = intersectionSlow(g1, g2),
|
||||
() -> fastOps[0] = intersectionFast(g1, g2),
|
||||
slowOps[0], fastOps[0]
|
||||
);
|
||||
|
||||
bench(
|
||||
String.format("group_overlap N=%d (worst: no overlap)", N),
|
||||
() -> slowOps[0] = overlapSlow(g1, g2),
|
||||
() -> fastOps[0] = overlapFast(g1, g2),
|
||||
slowOps[0], fastOps[0]
|
||||
);
|
||||
}
|
||||
|
||||
// Large scale: 4096 processes, two independent jobs (cross-job group ops
|
||||
// are common in MPI applications with spawned processes)
|
||||
{
|
||||
final int N = 4096;
|
||||
final long[] g1 = new long[N], g2 = new long[N];
|
||||
for (int i = 0; i < N; i++) {
|
||||
// jobid=1 for g1, jobid=2 for g2 — completely disjoint (worst case for overlap)
|
||||
g1[i] = (1L << 32) | i;
|
||||
g2[i] = (2L << 32) | i;
|
||||
}
|
||||
int[] allRanks = new int[N];
|
||||
for (int i = 0; i < N; i++) allRanks[i] = i;
|
||||
final long[] slowOps = {0}, fastOps = {0};
|
||||
bench(
|
||||
String.format("group_overlap disjoint N=%d (full scan worst case)", N),
|
||||
() -> slowOps[0] = overlapSlow(g1, g2),
|
||||
() -> fastOps[0] = overlapFast(g1, g2),
|
||||
slowOps[0], fastOps[0]
|
||||
);
|
||||
bench(
|
||||
String.format("group_intersection N=%d (disjoint)", N),
|
||||
() -> slowOps[0] = intersectionSlow(g1, g2),
|
||||
() -> fastOps[0] = intersectionFast(g1, g2),
|
||||
slowOps[0], fastOps[0]
|
||||
);
|
||||
}
|
||||
|
||||
System.out.println();
|
||||
System.out.println("Result: " + (PASS ? "PASS" : "FAIL"));
|
||||
if (!PASS) System.exit(1);
|
||||
}
|
||||
}
|
||||
BIN
defects/ompi/unit/unit/OmpiGroupTest.class
Normal file
BIN
defects/ompi/unit/unit/OmpiGroupTest.class
Normal file
Binary file not shown.
Loading…
Add table
Add a link
Reference in a new issue