wave12: 501/237 — ClickHouse/Druid/Pinot + Ansible/OpenTofu/Pulumi + Celery/Camel + VictoriaMetrics/Ceph

defects/ceph/patch/ceph-0001-osdmap-underfull-set.md

@@ -0,0 +1,105 @@
+# ceph-0001 — `OSDMap::calc_pg_upmaps`: O(N×U) `std::find` on `underfull` vector inside OSD scan loop
+
+## Status
+PATCHED
+
+## Severity
+HIGH (>50× speedup at N=1000 OSDs, U=500 underfull)
+
+## Location
+`src/osd/OSDMap.cc`, function `OSDMap::calc_pg_upmaps()`, line 5981–5983:
+
+```cpp
+for (auto& [deviation, osd] : deviation_osd) {
+    if (std::find(underfull.begin(), underfull.end(), osd) ==
+                  underfull.end())
+        break;
+```
+
+## Description
+`calc_pg_upmaps()` is Ceph's cluster rebalancing algorithm. It runs up to
+`max` iterations (default configurable, commonly 100) trying to add or remove
+PG upmap entries to equalize OSD fill levels.
+
+Inside the outer `while (max--)` loop, after failing to find improvements for
+overfull OSDs, the code scans `deviation_osd` (a `multimap<float, int>` of
+**all OSDs sorted by deviation**) looking for underfull OSDs to try
+`try_drop_remap_underfull` on.
+
+For each OSD in `deviation_osd` it calls:
+```cpp
+std::find(underfull.begin(), underfull.end(), osd)
+```
+`underfull` is a `vector<int>` populated by `fill_overfull_underfull()` with
+all OSDs whose deviation is below `-max_deviation`. In a large cluster this
+can be hundreds of entries.
+
+The loop runs over **all OSDs** in `deviation_osd`, calling `std::find` on the
+`underfull` vector each time:
+
+```
+Cost per while-iteration = O(N_osds × |underfull|)
+Total cost = max_iters × O(N_osds × |underfull|)
+```
+
+In a 1000-OSD cluster with 500 underfull OSDs and max=100:
+```
+100 × 1000 × 500 = 50,000,000 linear comparisons
+```
+
+`underfull` is **read-only** in this loop — it is the ideal candidate for
+conversion to `std::unordered_set<int>` for O(1) membership test.
+
+The same defect appears in CrushWrapper's `try_remap_rule()` (called via
+`try_pg_upmap`), where `std::find(orig.begin(), orig.end(), item)` is called
+inside `for (auto item : underfull)`, making it doubly-nested O(U × |orig|).
+
+## Patch
+
+### OSDMap.cc — deviation_osd scan loop
+
+```cpp
+// Before the while(max--) loop, or after fill_overfull_underfull():
+std::unordered_set<int> underfull_set(underfull.begin(), underfull.end());
+std::unordered_set<int> more_underfull_set(more_underfull.begin(), more_underfull.end());
+
+// In the scan loop (line 5981):
+for (auto& [deviation, osd] : deviation_osd) {
+-   if (std::find(underfull.begin(), underfull.end(), osd) ==
+-                 underfull.end())
++   if (underfull_set.find(osd) == underfull_set.end())
+        break;
+    ...
+}
+```
+
+Note: `underfull` is rebuilt each `while` iteration so the set must be rebuilt
+after each `fill_overfull_underfull()` call. Alternatively, change
+`fill_overfull_underfull()` to return an `unordered_set` directly.
+
+### CrushWrapper.cc — try_remap_rule underfull inner loop
+
+```cpp
+// Before: for (auto item : underfull) { ... std::find(orig.begin(), orig.end(), item) ... }
+// Build orig_set once before the underfull loop:
+std::unordered_set<int> orig_set(orig.begin(), orig.end());
+
+for (auto item : underfull) {
+    ...
+-   if (std::find(orig.begin(), orig.end(), item) != orig.end()) {
++   if (orig_set.count(item)) {
+        continue;
+    }
+    ...
+}
+```
+
+## Speedup estimate
+| N OSDs | |underfull| | max_iters | Defective ops | Fixed ops  | Ratio |
+|--------|------------|-----------|--------------|------------|-------|
+| 100    | 50         | 100       | 500,000      | 10,000     | 50×   |
+| 1,000  | 500        | 100       | 50,000,000   | 100,000    | 500×  |
+| 5,000  | 2,500      | 100       | 1,250,000,000| 500,000    | 2500× |
+
+## Patch file
+See `ceph-0001-osdmap-underfull-set.patch`

defects/ceph/patch/ceph-0001-osdmap-underfull-set.patch

@@ -0,0 +1,59 @@
+diff --git a/src/osd/OSDMap.cc b/src/osd/OSDMap.cc
+--- a/src/osd/OSDMap.cc
++++ b/src/osd/OSDMap.cc
+@@ -5818,6 +5818,8 @@ int OSDMap::calc_pg_upmaps(
+   while (max--) {
+     ldout(cct, 30) << "Top of loop #" << max+1 << dendl;
+     // build overfull and underfull
+     set<int> overfull;
+     set<int> more_overfull;
+     bool using_more_overfull = false;
+     vector<int> underfull;
+     vector<int> more_underfull;
+     fill_overfull_underfull(cct, deviation_osd, max_deviation,
+                             overfull, more_overfull,
+                             underfull, more_underfull);
++    // Pre-build O(1) lookup sets — underfull/more_underfull are read-only in
++    // the scan loops below, so unordered_set avoids O(N×U) std::find calls.
++    std::unordered_set<int> underfull_set(underfull.begin(), underfull.end());
++    std::unordered_set<int> more_underfull_set(more_underfull.begin(), more_underfull.end());
+
+@@ -5979,8 +5981,7 @@ int OSDMap::calc_pg_upmaps(
+     for (auto& [deviation, osd] : deviation_osd) {
+-      if (std::find(underfull.begin(), underfull.end(), osd) ==
+-                    underfull.end())
++      if (underfull_set.find(osd) == underfull_set.end())
+         break;
+       float target = osd_weight[osd] * pgs_per_weight;
+
+diff --git a/src/crush/CrushWrapper.cc b/src/crush/CrushWrapper.cc
+--- a/src/crush/CrushWrapper.cc
++++ b/src/crush/CrushWrapper.cc
+@@ -4118,6 +4118,8 @@ int CrushWrapper::try_remap_rule(
+     for (int pos = 0; pos < fanout; ++pos) {
+       if (type > 0) {
+         // non-leaf: no change
+       } else {
+         // leaf
+         bool replaced = false;
+         if (overfull.count(*i)) {
++          // Build orig_set once per leaf slot — O(1) membership below
++          std::unordered_set<int> orig_set(orig.begin(), orig.end());
+           for (auto item : underfull) {
+             if (used.count(item)) continue;
+             if (!subtree_contains(from, item)) continue;
+-            if (std::find(orig.begin(), orig.end(), item) != orig.end()) {
++            if (orig_set.count(item)) {
+               ldout(cct, 20) << __func__ << "   in orig " << orig << dendl;
+               continue;
+             }
+@@ -4152,8 +4154,6 @@ int CrushWrapper::try_remap_rule(
+           if (!replaced) {
+             for (auto item : more_underfull) {
+               if (used.count(item)) continue;
+               if (!subtree_contains(from, item)) continue;
+-              if (std::find(orig.begin(), orig.end(), item) != orig.end()) {
++              if (orig_set.count(item)) {
+                 ldout(cct, 20) << __func__ << "   in orig " << orig << dendl;
+                 continue;
+               }