Bug ID 499280: Client side or server side SSL handshake may fail if it involves SHA512-signed certificates in TLS1.2

Last Modified: Apr 10, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP LTM(all modules)

Known Affected Versions:
11.5.0, 11.5.1, 11.5.1 HF1, 11.5.1 HF10, 11.5.1 HF11, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.2, 11.5.2 HF1, 11.5.3, 11.5.3 HF1, 11.5.3 HF2, 11.5.4, 11.5.4 HF1, 11.5.4 HF2, 11.5.4 HF3, 11.5.4 HF4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4

Fixed In:
12.0.0, 11.6.1 HF2, 11.6.0 HF5

Opened: Jan 04, 2015
Severity: 3-Major

Symptoms

A handshake with either client-ssl or server-ssl when presented with a certificate signed/hashed with sha512 may fail.

Impact

The BIG-IP system cannot establish SSL connection with the backend server. Client fails to establish an SSL connection with the BIG-IP system.

Conditions

The issue is seen when it meets the following 3 conditions. 1. The SSL connection is using TLS1.2 2. The backend server's certificate is signed/hashed with sha512. 3. The backend server is Microsoft IIS server. More precisely, a server that strictly enforces the RFC policy for TLS1.2: 'If the client provided a 'signature_algorithms' extension, then all certificates provided by the server MUST be signed by a hash/signature algorithm pair that appears in that extension.' This kind of server rejects the SSL connection if the BIG-IP system does not advertise sha512 when sending the clienthello message. Microsoft IIS server does strictly enforce this rejection behavior, although Apache and OpenSSL servers do not. On the client side: 1. Client is trying to establish SSL connection using TLS1.2. 2. Client-ssl is configured with client-cert authentication. 3. Client is configured with sha512-signed certificate only. When the BIG-IP system sends a CertificateRequest that does not include sha512, the client might send back a null certificate.

Workaround

To workaround this: -- Use TLS1/TLS1.1/SSL3 instead of TLS1.2. -- Configure the backend server to use certificates signed/hashed with something other than sha512. -- Use a backend server other than Microsoft IIS.

Fix Information

For the serverside, the system now contains sha512 in the signature_algorithms extension when sending the clienthello with TLS1.2 (when you configures 'ANY' in the SSL sign hash option in the serverssl profile), so that the server does not reject the SSL connection because the BIG-IP system does not contain sha512 in the clienthello. sha512 is also included on the clientside, so that if the client uses sha512 to hash/sign the certvfy message, the BIG-IP system (acting as a server) does not reject to verify it (when you configures 'ANY' in the SSL sign hash option in the clientssl profile).

Behavior Change