Last Modified: Oct 16, 2023
Affected Product(s):
BIG-IP TMOS
Known Affected Versions:
12.1.0, 12.1.1, 12.1.2, 12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 12.1.4.1, 12.1.5, 12.1.5.1, 12.1.5.2, 12.1.5.3, 12.1.6, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 13.1.3.4, 13.1.3.5, 13.1.3.6, 13.1.4, 13.1.4.1, 13.1.5, 13.1.5.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1
Fixed In:
16.0.0, 15.1.0.2, 14.1.2.8
Opened: Nov 29, 2019 Severity: 2-Critical
Despite having free memory, the BIG-IP system frequently logs kernel page allocation failures to the /var/log/kern.log file. The first line of the output appears similar to the following example: swapper/16: page allocation failure: order:2, mode:0x104020 After that, a stack trace follows. Note that the process name in the line ('swapper/16', in this example) varies. You may see generic Linux processes or processes specific to F5 in that line.
As different processes can experience this issue, the system may behave unpredictably. For example, it is possible for a TMOS installation to fail as a result of this issue. Other processes may not exhibit any side effect as a result of this issue. The exact impact depends on which process becomes affected and how this process is designed to handle such a failure to allocate memory.
This issue is known to occur on the following VIPRION blade models: - B2250 (A112) - B4300 (A108) - B4340N (A110) - B4450 (A114) Please note the issue is known to occur regardless of whether or not the system is running in vCMP mode, and regardless of whether the system is Active or Standby.
You can work around this issue by increasing the value of the min_free_kbytes kernel parameter. This controls the amount of memory that is kept free for use by special reserves. It is recommend to increase this as follows: -- 64 MB (65536 KB for 2250 blades) -- 48 MB (49152 KB for B4300 blades) -- 128 MB (131072 KB for 4450 blades) You must do this on each blade installed in the system. When instantiating this workaround, you must consider whether you want the workaround to survive only reboots, or to survive reboots, upgrades, RMAs, etc. This is an important consideration to make, as you should stop using this workaround when this issue is fixed in a future version of BIG-IP software. So consider the pros and cons of each approach before choosing one. -- If you want the workaround to survive reboots only, perform the following procedure: 1) Log on to the advanced shell (BASH) of the primary blade of the affected VIPRION system. 2) Run the following commands (with the desired amount in KB): # clsh "sysctl -w vm.min_free_kbytes=131072" # clsh "echo -e '\n# Workaround for ID753650' >> /etc/sysctl.conf" # clsh "echo 'vm.min_free_kbytes = 131072' >> /etc/sysctl.conf" -- If you want the workaround to survive reboots, upgrades, RMAs, etc., perform the following procedure: 1) Log on to the advanced shell (BASH) of the primary blade of the affected VIPRION system. 2) Run the following commands (with the desired amount in KB): # clsh "sysctl -w vm.min_free_kbytes=131072" # echo -e '\n# Workaround for ID753650' >> /config/startup # echo 'sysctl -w vm.min_free_kbytes=131072' >> /config/startup Note that the last two commands are not wrapped inside 'clsh' because the /config/startup file is already automatically synchronized across all blades. Once the issue is fixed in a future BIG-IP version, remove the workarounds: -- To remove the first workaround: 1) Edit the /etc/sysctl.conf file on all blades, and remove the added lines at the bottom. 2) Reboot the system by running 'clsh reboot'. This will restore the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running. -- To remove the second workaround: 1) Edit the /config/startup file on the primary blade only, and remove the extra lines at the bottom. 2) Reboot the system by running 'clsh reboot'. This restores the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running. To verify the workaround is in place, run the following command (this should return the desired amount in KB): # clsh "cat /proc/sys/vm/min_free_kbytes"
The BIG-IP system no longer experiences kernel page allocation failures.