Bug ID 718405: RSA signature PAYLOAD_AUTH mismatch with certificates

Last Modified: Sep 13, 2023

Affected Product(s):
BIG-IP TMOS(all modules)

Known Affected Versions:
11.6.0, 11.6.1, 11.6.2, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 11.6.5, 11.6.5.1, 11.6.5.2, 11.6.5.3, 12.0.0, 12.0.0 HF1, 12.1.0 HF1, 12.0.0 HF2, 12.1.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.1 HF1, 12.1.1 HF2, 12.1.2 HF1, 12.1.2 HF2, 12.1.0, 12.1.1, 12.1.2, 12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 12.1.4.1, 12.1.5, 12.1.5.1, 12.1.5.2, 12.1.5.3, 12.1.6, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5

Fixed In:
15.1.0, 14.1.0.6, 13.1.1.4

Opened: May 03, 2018

Severity: 3-Major

Symptoms

IPsec IKEv2 negotiation with other vendors may fail to establish tunnels when certificate authentication is configured, using either RSA signature or DSS. The value of PAYLOAD_AUTH does not match when the BIG-IP system compares it with what the remote peer sends. The same certificate works when the BIG-IP system is the initiator, but not when another vendor is the initiator.

Impact

IKEv2 tunnels fail to establish, failing the second IKE_AUTH exchange in the protocol.

Conditions

Interoperating with other vendors under IKEv2 while using certificates.

Workaround

Use pre-shared key authentication.

Fix Information

BIG-IP systems now correctly build -- and verify -- AUTH payloads for RSA signatures and DSS, which should match other vendors and succeed, resulting in IKEv2 tunnels being created using certificates. The DSS signature is no longer DER encoded, and the RSA signature now includes the 15-byte DER prefix (mandated by RFC3447, page 42) before the 20-byte SHA1 digest is signed by RSA.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips