java-topology/whitepaper/outreach/wolfssl.md

WolfSSL — CWE-407 Disclosure Brief

Project: WolfSSL Disclosure date: 2026-03-27 Severity: HIGH Speedup: varies with S×C Status: PATCHED

---

Finding

WolfSSL's ALPN negotiation in src/tls.c performs an O(S×C) double loop when matching server and client ALPN protocol names. TLSX_ALPN_GetRequest() iterates over S server-configured ALPN names in an outer loop and for each performs an inner while scan over C client-offered names. This executes on every TLS handshake that includes the ALPN extension, which is mandatory for HTTP/2 and common for gRPC.

The Defect(s)

ID	Location	Pattern	Complexity
wolfssl-0001	src/tls.c	TLSX_ALPN_GetRequest() outer for over S server names × inner while over C client names	O(S×C) per handshake

Complexity Proof

Let S = number of ALPN protocol names configured on the server, C = number of ALPN protocol names offered by the client, L = average protocol name byte length.

TLSX_ALPN_GetRequest() iterates the server's ALPN list in an outer loop. For each server name, it traverses the linked list of client names in an inner while loop, performing a string comparison (O(L)) at each step:

For server_name_1: scan C client names → up to C×L byte comparisons
For server_name_2: scan C client names → up to C×L byte comparisons
...
For server_name_S: scan C client names → up to C×L byte comparisons
Total: S × C × L byte comparisons per handshake

Pre-building a hash set of client ALPN names (keyed by interned string pointer or FNV hash) before the outer loop reduces the inner scan to O(1) per server name:

Fixed: build hash set O(C×L) + outer loop S × O(1) = O(C×L + S)
Speedup: S× over the inner scan; net speedup depends on S/C ratio

For S = 6 server protocols and C = 8 client protocols with L = 15 bytes average (e.g., "h2", "http/1.1", "grpc"), the defective path performs 720 byte comparisons; the fixed path performs 90 (set build) + 6 (lookups) = 96.

Impact

Every WolfSSL TLS connection that includes ALPN extension processing is affected. WolfSSL is widely deployed in embedded systems, IoT devices, automotive ECUs, and RTOS environments. These resource-constrained targets have no CPU budget to spare for quadratic handshake work. High-connection-rate WolfSSL deployments (embedded HTTP/2 servers, MQTT brokers, industrial IoT gateways) processing many simultaneous TLS handshakes are most affected. The defect compounds when both S and C are large (which occurs in environments advertising backward compatibility for multiple protocol generations).

The Fix

Before the outer server-name loop in TLSX_ALPN_GetRequest(), iterate the client ALPN list once to build a hash set of client names. Replace the inner while scan with an O(1) hash set lookup for each server name.

Patch

- int TLSX_ALPN_GetRequest(TLSX* extensions, const void** data, word16* size) {
-     ALPN* server_alpn = (ALPN*)TLSX_Find(extensions, TLSX_APPLICATION_LAYER_PROTOCOL)->data;
-     ALPN* client_alpn = ...;
-     while (server_alpn) {
-         ALPN* ca = client_alpn;
-         while (ca) {
-             if (XSTRNCMP(server_alpn->protocol_name, ca->protocol_name,
-                          server_alpn->protocol_nameSz) == 0) {
-                 /* match found */
-             }
-             ca = ca->next;
-         }
-         server_alpn = server_alpn->next;
-     }
- }
+ int TLSX_ALPN_GetRequest(TLSX* extensions, const void** data, word16* size) {
+     /* build hash set of client ALPN names */
+     AlpnHashSet client_set;
+     AlpnHashSet_Init(&client_set);
+     ALPN* ca = client_alpn;
+     while (ca) { AlpnHashSet_Add(&client_set, ca->protocol_name, ca->protocol_nameSz); ca = ca->next; }
+
+     ALPN* server_alpn = ...;
+     while (server_alpn) {
+         if (AlpnHashSet_Contains(&client_set, server_alpn->protocol_name,
+                                  server_alpn->protocol_nameSz)) {
+             /* match found */
+         }
+         server_alpn = server_alpn->next;
+     }
+     AlpnHashSet_Free(&client_set);
+ }

What We Ask

Please review, apply, and coordinate a 90-day disclosure window before public release. Reply to security@undefect.com.

---

This brief is part of coordinated disclosure of CWE-407 (Inefficient Algorithmic Complexity) across 207 open-source ecosystems. Full report: https://undefect.com