Bug ID 603598: big3d memory under extreme load conditions

Last Modified: Sep 13, 2023

Affected Product(s):
BIG-IP GTM, LTM(all modules)

Known Affected Versions:
10.2.4, 11.0.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.5.1 HF1, 11.6.1 HF1, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.1 HF10, 11.5.1 HF11, 11.5.2 HF1, 11.5.3 HF1, 11.5.3 HF2, 11.5.4 HF1, 11.5.4 HF2, 11.1.0, 11.2.0, 11.2.1, 11.3.0, 11.4.0, 11.4.1, 11.6.0, 11.6.1, 11.6.2, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 11.6.5, 11.6.5.1, 11.6.5.2, 11.6.5.3, 12.0.0, 12.0.0 HF1, 12.1.0 HF1, 12.0.0 HF2, 12.1.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.1 HF1, 12.1.1 HF2, 12.1.2 HF1, 12.1.2 HF2, 12.1.0, 12.1.1

Fixed In:
13.0.0, 12.1.2, 11.6.1 HF2, 11.5.4 HF3

Opened: Jul 07, 2016

Severity: 2-Critical

Related Article: K81350440

Symptoms

big3d memory consumption can grow if big3d is unable to process monitor requests in a timely fashion. This can be seen by monitoring the memory consumption of big3d using standard OS tools such as top.

Impact

big3d memory consumption grows unbounded. This might result in a big3d restart or memory starvation of other processes.

Conditions

big3d maintains a queue for monitor requests. Incoming monitor requests are first placed in the Pending queue. Requests are moved from the Pending queue to the Active queue, if there is room in the Active queue. When the Pending queue is full, there is no room for the Monitor Request. big3d attempts to clean up the Monitor request, but fails to completely free the memory. This might result in a significant memory leak. For this to happen, the Active queue must be full as well as the Pending queue. One possible condition that might cause this is if multiple Monitors time out. This results in Monitors having long life times, which keeps the Active queue full. Thus the Pending queue might become full and the memory leak can occur. In BIG-IP 11.1.0 versions of big3d, the Active queue has 256 slots and the Pending queue has 4096 slots. In BIG-IP 11.1.0-hf3, the queue sizes were expanded to 2048 for the Active queue and 16384 for the Pending queue. Since the queues were smaller n versions prior to 11.1.0-hf3, this leaks is more likely to manifest itself. In later versions, the leak is still possible, but is less likely to occur.

Workaround

This can be partially mitigated by ensuring that monitors settings are reasonable and that big3d is not overloaded. This will minimize the chances that the Pending queue does not become full. There is no mechanism to resize the queues.

Fix Information

When a monitor request is unable to be placed in the queue, the memory for the request is freed properly.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips