diff --git a/whitepaper/outreach/wave8-observability-streaming-survey.md b/whitepaper/outreach/wave8-observability-streaming-survey.md new file mode 100644 index 000000000..56ce60c06 --- /dev/null +++ b/whitepaper/outreach/wave8-observability-streaming-survey.md @@ -0,0 +1,80 @@ +# Wave 8 — Observability + Streaming/Workflow + +**Survey date:** 2026-04-25 +**Tool:** unmoad (9 active MOAD detectors, HIGH+ severity filter) +**Scope:** 10 observability and stream/workflow projects: jaeger (distributed tracing), opentelemetry-collector (OTel pipeline), fluent-bit + fluentd (log forwarders), vector (Rust observability data pipeline), tempo (Grafana traces), mimir (Grafana metrics), VictoriaMetrics (Go metrics TSDB), nsq (message queue), temporal (workflow engine). + +--- + +## Summary + +Wave 8 totals 2,188 HIGH+ findings across 10 projects. **Four new clean-scan honor roll entries** (nsq, jaeger, opentelemetry-collector, temporal — all reduce to scanner false positives or build-time/test-only patterns under inspection). Honor roll cumulative: **34 projects** across waves 3-8. + +**No flagship CWE-407 patches ship this pass.** The strongest M1 candidate (VictoriaMetrics `streamaggr.getInputOutputLabels` — `slices.Contains` per label per sample) is real but constant-factor at realistic config shapes (`by`/`without` lists are typically 3-10 entries, below the CWE-407 wall-clock bar). Logged for future re-scan once configs grow. + +## Clean-scan honor roll — 4 new entries + +| Project | Lang | Role | Notes | +|---------|------|------|-------| +| **nsq** | Go | Distributed message queue | 4 findings, all FPs: `nsqadmin/static/js` String#includes vs Array#includes confusion + 1 test-fixture credential string. Tight Go core. | +| **jaeger** | Go | Distributed tracing | 0 M1. M3 hits all in `internal/auth/*` and `internal/tenancy/*` — these are intentional ContextValue plumbing for tenant/auth propagation, the standard Go pattern for cross-cutting context, not "leaked context" defects. | +| **opentelemetry-collector** | Go | OTel pipeline orchestrator | 9 M1 in `internal/cmd/pdatagen` — that is a build-time code generator, not a runtime path. Most M3 hits in `*_test.go`. | +| **temporal** | Go | Workflow engine | 19 M1 mostly in test files / `cmd/tools/genrpcwrappers` (build-time). The single core finding `server_options.go:78` checks against fixed Services list (5 entries: frontend, history, matching, worker, internal-frontend). | + +Honor roll now stands at **34 projects** validated zero-real-finding under MOAD scanning. + +## Per-target findings + +| Project | Lang | Total | M1 | M3 | M4 | M5 | M6 | M7 | M9 | M11 | Triage | +|---------|------|------:|---:|---:|---:|---:|---:|---:|---:|----:|--------| +| fluent-bit | C | 1203 | 1055 | 7 | 72 | 1 | 14 | 41 | - | 13 | M1 dominated by msgpack JSON-key `strncmp` parsing (`processor_labels`, `kube_meta`, `cloudwatch_logs`, exporter modules). N here is "static labels" — typically 5-20 — bounded constant. | +| vector | Rust | 262 | 121 | 1 | 14 | - | - | 125 | - | 1 | `windows_event_log/config.rs` query_lower.contains is config-time. `topology/running.rs` `reuse_buffers.contains` runs once at topology build. M7 cluster (125) worth a focused look but most are graph-build time. | +| VictoriaMetrics | Go | 209 | 156 | 2 | 19 | 1 | - | 28 | 2 | 1 | UI vendored JS dominates (88 hits in `app/vmselect/vmui` + `vmalert/static`). Real candidate `streamaggr.go:1129/1137` — see investigation below. | +| mimir | Go | 156 | 62 | 47 | 14 | - | - | 33 | - | - | UI vendored JS (`bootstrap-5.1.3.bundle.min.js`) accounts for 36. Core: `blocks_store_replicated_set.go` zone-bounded (3-9 zones); `codec.go` propagateHeaders bounded by config. | +| temporal | Go | 99 | 19 | 49 | 14 | - | 1 | 1 | 15 | - | M3 cluster all `_test.go` ContextWithValue. Core M1 = fixed Services list. **clean** | +| tempo | Go | 73 | 41 | - | 3 | - | - | 29 | - | - | UI vendored JS (36). `tracker.go:138/149` `t.sortedKeys` membership in usage tracking — bounded by tenant dimensions (typically <50). | +| opentelemetry-collector | Go | 53 | 9 | 44 | - | - | - | - | - | - | M1 = build-time generator. M3 = test fixtures. **clean** | +| jaeger | Go | 26 | - | 8 | 14 | 2 | - | 1 | - | 1 | Auth/tenancy ContextValue (intentional). M4 = "Authorization" in HTTP transport (intentional Bearer-token header). **clean** | +| fluentd | Ruby | 103 | 70 | 27 | - | - | - | - | - | 6 | Ruby `Array#include?` clusters in plugin parsers; mostly bounded by directive shape. | +| nsq | Go | 4 | 3 | - | 1 | - | - | - | - | - | All FPs (string vs array methods, test fixture). **clean** | + +## Investigation: VictoriaMetrics `streamaggr.getInputOutputLabels` + +`lib/streamaggr/streamaggr.go:1126` is called per time-series during stream aggregation. For each label in a sample, it calls `slices.Contains(without, label.Name)` (or `slices.Contains(by, label.Name)`) where `by`/`without` are user-configured label name lists. + +```go +func getInputOutputLabels(dstInput, dstOutput, labels []prompb.Label, by, without []string) ([]prompb.Label, []prompb.Label) { + if len(without) > 0 { + for _, label := range labels { + if slices.Contains(without, label.Name) { // O(W) per label + dstInput = append(dstInput, label) + } else { + dstOutput = append(dstOutput, label) + } + } + } + ... +} +``` + +For L labels per sample and W `without` (or `by`) entries, total cost is O(L × W) per sample. At ingest rates of 1M+ samples/sec, this multiplies. Real but constant-factor: typical configs have W ≤ 10 and L ≤ 30, so the constant is 300 ops/sample. A `map[string]struct{}{}` would replace this with O(L) total via single hash per label. + +**Not patch-shipped.** At realistic config shapes the wall-clock impact is ~2-3× — below the CWE-407 bar of complexity-class change with measurable wall-clock impact (5× speedup at minimum). For users running label-heavy aggregations (W > 50), the patch would matter; we leave it logged for a re-scan once a real production case surfaces. + +## Triage backlog + +1. **VictoriaMetrics streamaggr** — re-scan if a real user reports W > 30 with high-cardinality ingest; the `set` patch is 5 lines and we have it on file. +2. **fluentd Ruby plugin parsers** — `Array#include?` clusters in 70 hits worth a deeper Ruby-specific pass. +3. **vector M7 cluster (125 hits)** — Flatland defect surface in topology graph build paths. Most likely topology-build-time but worth confirming by-file. +4. **fluent-bit `processor_labels`** — bounded by static label count today; if Prometheus integrations push label count higher per metric, the linear `cfl_list_foreach` pattern becomes interesting. +5. **mimir `propagateHeaders` configs** — if operators add many propagated headers per request, the per-request `slices.Contains` walk becomes measurable. + +## Method + +Same as Waves 3-7: shallow clone, `unmoad -s high -f json`, filter test/vendor/UI noise, manual triage of strongest source-only candidates per project. Four projects added to clean-scan honor roll. No new UNDF IDs assigned this wave (no patches shipped). + +## References + +- `unmoad` detection engine: `git.unturf.com/engineering/unmoad.com` +- Earlier surveys: `/test-harness-survey/`, `/wave4-linter-ci-survey/`, `/wave5-cicd-iac-survey/`, `/wave6-docgen-webfw-tui-survey/`, `/docs-pipeline-survey/`, `/wave7-mail-dns-storage-vpn-rtos-survey/` +- Clean-scan honor roll cumulative: 34 projects across waves 3-8