Bug ID 1250901: On rSeries FIPS system live upgrade, cavium_n3fips (liquidsec_pf_vf driver) may fail to come into operational state

Last Modified: Jan 23, 2024

Affected Product(s):
F5OS F5OS(all modules)

Known Affected Versions:
F5OS-A 1.4.0

Fixed In:
F5OS-A 1.5.0

Opened: Feb 22, 2023

Severity: 1-Blocking

Symptoms

After a reboot of the system in live upgrade, tenants that were running earlier might not change to a running state. This is due to the HSM board driver stuck in SAFE_STATE instead of OPERATIONAL_STATE. In some cases, the driver changes to an operational state after some amount of time (approximately 10 minutes). But this time might vary upon detection of reset/link failure in the hardware. In some other systems, the driver becomes stuck in SAFE_STATE indefinitely.

Impact

Running tenants goes to pending state when this issue occurs in a live upgrade.

Conditions

Live upgrade/reboot of the rSeries FIPS system with F5OS-A. You may observe the below logs in dmesg- [ 964.105021] liquidsec_pf_vf_driver 0000:ca:00.0: We might have a link issue... resetting [ 964.113688] liquidsec_pf_vf_driver 0000:ca:00.0: RESETTING FIRMWARE... CAUTION

Workaround

Check contents of cavium_n3fips file as shown below. [appliance]# cat /proc/cavium_n3fips/driver_state HSM 0:OPERATIONAL_STATE If the driver changes to an operational state, perform "docker restart fips-support-pod" to help in recovering. But if the driver state is still "HSM 0:SAFE_STATE", you may need to perform a power cycle reboot (but this will not guarantee recovery).

Fix Information

N/A

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips