java-topology/whitepaper/outreach/wave8-observability-streaming-survey.md

Wave 8 — Observability + Streaming/Workflow

Survey date: 2026-04-25 Tool: unmoad (9 active MOAD detectors, HIGH+ severity filter) Scope: 10 observability and stream/workflow projects: jaeger (distributed tracing), opentelemetry-collector (OTel pipeline), fluent-bit + fluentd (log forwarders), vector (Rust observability data pipeline), tempo (Grafana traces), mimir (Grafana metrics), VictoriaMetrics (Go metrics TSDB), nsq (message queue), temporal (workflow engine).

---

Summary

Wave 8 totals 2,188 HIGH+ findings across 10 projects. Four new clean-scan honor roll entries (nsq, jaeger, opentelemetry-collector, temporal — all reduce to scanner false positives or build-time/test-only patterns under inspection). Honor roll cumulative: 34 projects across waves 3-8.

No flagship CWE-407 patches ship this pass. The strongest M1 candidate (VictoriaMetrics streamaggr.getInputOutputLabels — slices.Contains per label per sample) is real but constant-factor at realistic config shapes (by/without lists are typically 3-10 entries, below the CWE-407 wall-clock bar). Logged for future re-scan once configs grow.

Clean-scan honor roll — 4 new entries

Project	Lang	Role	Notes
nsq	Go	Distributed message queue	4 findings, all FPs: nsqadmin/static/js String#includes vs Array#includes confusion + 1 test-fixture credential string. Tight Go core.
jaeger	Go	Distributed tracing	0 M1. M3 hits all in internal/auth/* and internal/tenancy/* — these are intentional ContextValue plumbing for tenant/auth propagation, the standard Go pattern for cross-cutting context, not "leaked context" defects.
opentelemetry-collector	Go	OTel pipeline orchestrator	9 M1 in internal/cmd/pdatagen — that is a build-time code generator, not a runtime path. Most M3 hits in *_test.go.
temporal	Go	Workflow engine	19 M1 mostly in test files / cmd/tools/genrpcwrappers (build-time). The single core finding server_options.go:78 checks against fixed Services list (5 entries: frontend, history, matching, worker, internal-frontend).

Honor roll now stands at 34 projects validated zero-real-finding under MOAD scanning.

Per-target findings

Project	Lang	Total	M1	M3	M4	M5	M6	M7	M9	M11	Triage
fluent-bit	C	1203	1055	7	72	1	14	41	-	13	M1 dominated by msgpack JSON-key strncmp parsing (processor_labels, kube_meta, cloudwatch_logs, exporter modules). N here is "static labels" — typically 5-20 — bounded constant.
vector	Rust	262	121	1	14	-	-	125	-	1	windows_event_log/config.rs query_lower.contains is config-time. topology/running.rs reuse_buffers.contains runs once at topology build. M7 cluster (125) worth a focused look but most are graph-build time.
VictoriaMetrics	Go	209	156	2	19	1	-	28	2	1	UI vendored JS dominates (88 hits in app/vmselect/vmui + vmalert/static). Real candidate streamaggr.go:1129/1137 — see investigation below.
mimir	Go	156	62	47	14	-	-	33	-	-	UI vendored JS (bootstrap-5.1.3.bundle.min.js) accounts for 36. Core: blocks_store_replicated_set.go zone-bounded (3-9 zones); codec.go propagateHeaders bounded by config.
temporal	Go	99	19	49	14	-	1	1	15	-	M3 cluster all _test.go ContextWithValue. Core M1 = fixed Services list. clean
tempo	Go	73	41	-	3	-	-	29	-	-	UI vendored JS (36). tracker.go:138/149 t.sortedKeys membership in usage tracking — bounded by tenant dimensions (typically <50).
opentelemetry-collector	Go	53	9	44	-	-	-	-	-	-	M1 = build-time generator. M3 = test fixtures. clean
jaeger	Go	26	-	8	14	2	-	1	-	1	Auth/tenancy ContextValue (intentional). M4 = "Authorization" in HTTP transport (intentional Bearer-token header). clean
fluentd	Ruby	103	70	27	-	-	-	-	-	6	Ruby Array#include? clusters in plugin parsers; mostly bounded by directive shape.
nsq	Go	4	3	-	1	-	-	-	-	-	All FPs (string vs array methods, test fixture). clean

Investigation: VictoriaMetrics streamaggr.getInputOutputLabels

lib/streamaggr/streamaggr.go:1126 is called per time-series during stream aggregation. For each label in a sample, it calls slices.Contains(without, label.Name) (or slices.Contains(by, label.Name)) where by/without are user-configured label name lists.

func getInputOutputLabels(dstInput, dstOutput, labels []prompb.Label, by, without []string) ([]prompb.Label, []prompb.Label) {
    if len(without) > 0 {
        for _, label := range labels {
            if slices.Contains(without, label.Name) {  // O(W) per label
                dstInput = append(dstInput, label)
            } else {
                dstOutput = append(dstOutput, label)
            }
        }
    }
    ...
}

For L labels per sample and W without (or by) entries, total cost is O(L × W) per sample. At ingest rates of 1M+ samples/sec, this multiplies. Real but constant-factor: typical configs have W ≤ 10 and L ≤ 30, so the constant is 300 ops/sample. A map[string]struct{}{} would replace this with O(L) total via single hash per label.

Not patch-shipped. At realistic config shapes the wall-clock impact is ~2-3× — below the CWE-407 bar of complexity-class change with measurable wall-clock impact (5× speedup at minimum). For users running label-heavy aggregations (W > 50), the patch would matter; we leave it logged for a re-scan once a real production case surfaces.

Triage backlog

1. VictoriaMetrics streamaggr — re-scan if a real user reports W > 30 with high-cardinality ingest; the set patch is 5 lines and we have it on file.
2. fluentd Ruby plugin parsers — Array#include? clusters in 70 hits worth a deeper Ruby-specific pass.
3. vector M7 cluster (125 hits) — Flatland defect surface in topology graph build paths. Most likely topology-build-time but worth confirming by-file.
4. fluent-bit processor_labels — bounded by static label count today; if Prometheus integrations push label count higher per metric, the linear cfl_list_foreach pattern becomes interesting.
5. mimir propagateHeaders configs — if operators add many propagated headers per request, the per-request slices.Contains walk becomes measurable.

Method

Same as Waves 3-7: shallow clone, unmoad -s high -f json, filter test/vendor/UI noise, manual triage of strongest source-only candidates per project. Four projects added to clean-scan honor roll. No new UNDF IDs assigned this wave (no patches shipped).

References

• unmoad detection engine: git.unturf.com/engineering/unmoad.com
• Earlier surveys: /test-harness-survey/, /wave4-linter-ci-survey/, /wave5-cicd-iac-survey/, /wave6-docgen-webfw-tui-survey/, /docs-pipeline-survey/, /wave7-mail-dns-storage-vpn-rtos-survey/
• Clean-scan honor roll cumulative: 34 projects across waves 3-8