Bug ID 739507: Improved recovery method for BIG-IP system that has halted from a failed FIPS integrity check

Last Modified: May 29, 2024

Affected Product(s):
BIG-IP TMOS(all modules)

Known Affected Versions:
13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 16.0.0, 16.0.0.1, 16.0.1, 16.0.1.1, 16.0.1.2

Fixed In:
16.1.0, 15.1.0.5, 14.1.4, 13.1.1.2

Opened: Aug 07, 2018

Severity: 2-Critical

Related Article: K25205233

Symptoms

After FIPS 140-2 license is installed on BIG-IP FIPS-certified hardware devices, the system halts while booting upon performing the FIPS integrity check. Console shows messages similar to: Starting System Logger Daemon... [ OK ] Started System Logger Daemon. [ 14.943495] System halted.

Impact

The device halts and cannot be used.

Conditions

-- The BIG-IP device has a license that includes the FIPS 140-2 option (FIPS full-box license). -- System element monitored by FIPS 140-2 integrity check has changed. -- The device is rebooted.

Workaround

Workaround: [1] Connect a terminal to the BIG-IP serial console port. [2] From the console, enter the GRUB menu and boot into a partition that does not have a FIPS 140-2-enabled license, or into TMOS Maintenance. [3] Mount config from the inactive partition (see K51222154: Mounting the filesystem of an inactive partition :: https://support.f5.com/csp/article/K51222154) that was halted, and examine the contents of /config/f5_public/fipserr, which shows the files that were changed, leading to failure of the FIPS 140-2 license-enabled partition. [4] Restore those files to their original ones. [5] Truncate the inactive partition's /config/f5_public/fipserr, e.g., by running: cat /dev/null > /mnt/test/f5_public/fipserr [6] Reboot. If the system still halts, repeat from Step [1] above, until this no longer happens.

Fix Information

If your device is running a version where ID 739507 is fixed: [1] Connect a terminal to the BIG-IP serial console port [2] From the serial console, enter the GRUB menu. [3] Before the countdown expires, use the Up Arrow and Down Arrow keys to stop the countdown, and select the appropriate boot image. [4] Press the key 'E' to start the edit options. A new GRUB menu displays. [5] Use the Up Arrow and Down Arrow keys to navigate to the line that starts with 'linux', or the first line that starts with 'module'. [6] Add a space, followed by NO_FIPS_INTEGRITY=1 (do not press ENTER). [7] Press the Ctrl-X sequence or the F10 key to restart the system using the modified options. The machine boots into the partition containing FIPS 140-2-enabled license. [8] Examine the content of file /config/f5_public/fipserr to ascertain the cause of the FIPS module startup error. [9] Fix the problem reported in the aforementioned error file. [10] Run the test tool /usr/libexec/sys-eicheck.py to ensure that no fatal error is reported, such as: Integrity Check Result: [ FAIL ] If fatal errors persist, do not reboot (otherwise the system foes into the halt state, and the steps starting from Step [1] will need to be repeated). Instead, fix the problematic files reported. Rerun the test tool until no error is seen. Note: You can find information on the sys-eicheck (FIPS) utility in the AskF5 Non-Diagnostic Article K00029945: Using the sys-eicheck (FIPS) utility :: https://support.f5.com/csp/article/K00029945. [11] Truncate the file /config/f5_public/fipserr: cat /dev/null > /config/f5_public/fipserr

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips