Bug ID 658853: BMC (AOM) Firmware update fails on certain platforms after power-on for ~5 weeks after upgrade

Last Modified: Dec 27, 2018

Bug Tracker

Affected Product:  See more info
Atlantis AOM(all modules)

Known Affected Versions:
12.1.0, 12.1.0 HF1, 12.1.0 HF2, 12.1.1, 12.1.1 HF1, 12.1.1 HF2, 12.1.2, 12.1.2 HF1, 12.1.2 HF2, 12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1

Opened: Apr 17, 2017
Severity: 1-Blocking
Related AskF5 Article:
K00415052

Symptoms

The BMC (AOM) firmware intermittently gets into a state where the BMC log-cleaning services stops working and causes the logging subsystem to grow and use all available memory. When in this state, if a new version of BIG-IP software is installed, it attempts to update the BMC firmware, which fails because there is not enough memory available to support firmware upgrade. As a result of this issue, you may encounter one or more of the following symptoms: -- You see the following output on the serial console: Retrigger udev events[ OK ] Starting sshd: [ OK ] Checking for firmware updates: Updating AOM to /usr/firmware/bmc_shuttle_2.51.ima_enc Updating AOM ... DO NOT POWER DOWN ... THIS MAY TAKE UP TO 12 MINUTES TO COMPLETE Updating AOM ... DO NOT POWER DOWN ... THIS MAY TAKE UP TO 12 MINUTES TO COMPLETE (elapsed seconds:1) -- The unit does not boot up to the host. -- The unit reports firmware upgrade in progress on the LCD. -- The Configuration utility is unavailable.

Impact

Once BMC upgrade attempt fails, BMC gets stuck in firmware update mode, which causes failure of all subsequent IPMI commands from the BIG-IP system. In addition, the BIG-IP system cannot read the inventory information, which causes license check failure.

Conditions

This issue occurs when all of the following conditions are met: -- You have one of the following BIG-IP platforms: * i2x00 * i4x00 * i5x00 * i7x00 * i10x00 * HRC-i2x00 * HRC-i5x00 * HRC-i10x00 -- You perform an upgrade from any of the following BIG-IP versions: * 13.0.0 * 12.1.2 * 12.1.1 -- You have had the BIG-IP system powered on for approximately five weeks or more.

Workaround

Before new BIG-IP software is activated, reset BMC using one of these methods: -- Invoke AOM menu from system console port (ESC-Shift-9) and select A ( AOM Reset). -- Power cycle the system. -- From the BIG-IP command line, run the following command: ipmiutil reset -k. Note: If, after trying the above workarounds first, the problem still persists, then, you might need to power down the device and physically remove power for a few seconds. In most cases the above workaround will work, but if it does not then a power cycle may be necessary.

Fix Information

None

Behavior Change