OpenSSL — CWE-407 Disclosure Brief

Project: OpenSSL Disclosure date: 2026-03-27 Severity: HIGH Speedup: varies with n×m, n², C×S Status: PATCHED

Finding

OpenSSL contains three independent quadratic-complexity defects in cipher negotiation and SRTP profile matching within ssl/ssl_ciph.c and ssl/statem/extensions_srvr.c. Two are in cipher suite deduplication during TLS connection setup (one TLS 1.2, one TLS 1.3), and one is in SRTP profile matching for DTLS/WebRTC connections. All three execute on the hot path of TLS handshake processing.

The Defect(s)

ID	Location	Pattern	Complexity
openssl-0001	`ssl/ssl_ciph.c`	`SSL_get_shared_ciphers()` O(n×m) scan per connection when server stack unsorted	O(n×m)
openssl-0002	`ssl/ssl_ciph.c`	`ciphersuite_cb` TLS 1.3 dedup O(n²) during config parsing	O(n²)
openssl-0003	`ssl/statem/extensions_srvr.c`	`tls_parse_ctos_use_srtp()` outer while over C client IDs × inner for over S server profiles	O(C×S)

Complexity Proof

openssl-0001 — Let n = size of client cipher list, m = size of server cipher list.

SSL_get_shared_ciphers() finds the intersection by iterating all n client ciphers and for each scanning all m server ciphers linearly:

For client_cipher_1: scan m server ciphers
For client_cipher_2: scan m server ciphers
...
For client_cipher_n: scan m server ciphers
Total: n × m comparisons per connection

With a hash set of server cipher IDs (built once at config time), each client cipher lookup is O(1), reducing total work to O(n+m).

openssl-0002 — Let n = number of TLS 1.3 ciphersuites in the configured list.

The ciphersuite_cb deduplication iterates the accumulating output list for each new candidate, scanning up to n entries:

Total: 0 + 1 + ... + (n-1) = n(n-1)/2 = O(n²)

A bitmask indexed by the cipher table index (bounded by the small fixed TLS 1.3 suite count) reduces each check to O(1).

openssl-0003 — Let C = number of client SRTP profile IDs, S = number of server SRTP profiles.

tls_parse_ctos_use_srtp() iterates client IDs in an outer while loop and for each scans the server profile list in an inner for loop:

Total: C × S comparisons per DTLS handshake

A 32-slot Knuth hash set (the SRTP profile space is small) reduces this to O(C+S).

Impact

openssl-0001 affects all TLS 1.2 connections where the server cipher stack is not pre-sorted or filtered. High-connection-rate servers (HTTPS load balancers, API gateways, CDN edge nodes) pay this on every handshake.

openssl-0002 affects OpenSSL configuration parsing and TLS context setup when custom TLS 1.3 ciphersuite lists are configured. Applications that dynamically create SSL_CTX objects (multi-tenant servers, connection pooling) are most affected.

openssl-0003 affects all DTLS/WebRTC connections using SRTP (RFC 5764). Media servers, SFUs, and WebRTC gateways processing many simultaneous DTLS handshakes are most affected. C = 20 client profiles × S = 10 server profiles = 200 comparisons per handshake, reduced to 30 with the hash set.

The Fix

openssl-0001: Pre-build a hash set of server cipher IDs at SSL_CTX configuration time. Use it in SSL_get_shared_ciphers().

openssl-0002: Use a uint64_t bitmask indexed by position in the fixed TLS 1.3 cipher table for O(1) dedup.

openssl-0003: Build a 32-slot Knuth multiplicative hash set of server profile IDs before the outer client loop.

Patch

# openssl-0001: ssl/ssl_ciph.c
- for (i = 0; i < client_len; i++) {
-     for (j = 0; j < server_len; j++) {
-         if (client[i]->id == server[j]->id) { ... }
-     }
- }
+ /* pre-build server_id_set at ctx init */
+ for (i = 0; i < client_len; i++) {
+     if (hash_set_contains(server_id_set, client[i]->id)) { ... }
+ }

# openssl-0002: ssl/ssl_ciph.c
- /* TLS 1.3 ciphersuite dedup — O(n²) */
- for (i = 0; i < num; i++) {
-     for (j = 0; j < i; j++) {
-         if (list[j] == list[i]) { duplicate = 1; break; }
-     }
- }
+ /* bitmask dedup — O(n) */
+ uint64_t seen = 0;
+ for (i = 0; i < num; i++) {
+     int idx = cipher_table_index(list[i]);
+     if (seen & (1ULL << idx)) { duplicate = 1; continue; }
+     seen |= (1ULL << idx);
+ }

# openssl-0003: ssl/statem/extensions_srvr.c
- while (PACKET_get_net_2(&profiles_list, &id)) {
-     for (i = 0; i < server->srtp_profiles->num; i++) {
-         if (server->srtp_profiles->profiles[i].id == id) { match = 1; }
-     }
- }
+ /* build 32-slot hash set of server profile IDs first */
+ srtp_hash_set_t srv_set;
+ srtp_hash_init(&srv_set, server->srtp_profiles);
+ while (PACKET_get_net_2(&profiles_list, &id)) {
+     if (srtp_hash_contains(&srv_set, id)) { match = 1; break; }
+ }

What We Ask

Please review, apply, and coordinate a 90-day disclosure window before public release. Reply to security@undefect.com.

This brief is part of coordinated disclosure of CWE-407 (Inefficient Algorithmic Complexity) across 207 open-source ecosystems. Full report: https://undefect.com

5 KiB Raw Blame History Unescape Escape