Bug ID 548611: Memory protection strategies can conflict

Last Modified: Jun 20, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP LTM(all modules)

Known Affected Versions:
11.5.1, 11.5.1 HF1, 11.5.1 HF10, 11.5.1 HF11, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.2, 11.5.2 HF1, 11.5.3, 11.5.3 HF1, 11.5.3 HF2, 11.5.4, 11.5.4 HF1, 11.5.4 HF2, 11.5.4 HF3, 11.5.4 HF4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.6.1, 11.6.1 HF1, 11.6.1 HF2, 11.6.2, 11.6.2 HF1, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 12.0.0, 12.0.0 HF1, 12.0.0 HF2, 12.0.0 HF3, 12.0.0 HF4

Fixed In:
12.1.0

Opened: Sep 26, 2015
Severity: 3-Major

Symptoms

The TMM has three mechanisms to protect memory usage when under pressure: the sweeper responds to low memory with a variety of strategies such as killing idle flows; memory reaping is activated to restore memory to the system; and tcp random early drops are activated if configured. Since these are all targeting the same memory levels by default, it's possible that all three activate and victimize more flows than required. In addition, a flaw in the random early drop logic could cause unpredictable behavior.

Impact

More flows are victimized than necessary when under memory pressure. One symptom is a large number of random early drops, and hovering right near the sweeper's low-water mark causing new flows to encounter the random early drop limits nearly immediately.

Conditions

This occurs under normal operating conditions.

Workaround

The sweeper's low-water mark can be adjusted, along with the tm.tcpmemorypressure.hiwater and tm.tcpmemorypressure.lowater variables so that they are not all at the same location; this can alleviate most symptoms of this issue.

Fix Information

The built-in limits for random early drops and memory reaping are now aligned with the sweeper. The configured low-water mark is used to drive all three algorithms, and memory reaping targets five percent below the configured low-water mark. Along with this, the tcpmemorypressure variables are deprecated, and no longer used to control random early drops: these are now driven by the sweeper global memory constraints. The random early drop logic has been revised to provide more stable behavior across all pressures. The memory reaper will target the global low-water mark minus five percent, in order to prevent sticking right at the point where the aggressive sweeper will activate.

Behavior Change