Bug ID 477511: Very large GTM monitoring configs can cause monitors to fail

Last Modified: Mar 12, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP GTM(all modules)

Known Affected Versions:
11.0.0, 11.1.0, 11.2.0, 11.2.1, 11.3.0, 11.4.0, 11.4.1, 11.5.0, 11.5.1, 11.5.1 HF1, 11.5.1 HF10, 11.5.1 HF11, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.2, 11.5.2 HF1, 11.5.3, 11.5.3 HF1, 11.5.3 HF2, 11.5.4, 11.5.4 HF1, 11.5.4 HF2, 11.5.4 HF3, 11.5.4 HF4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.6.1, 11.6.1 HF1, 11.6.1 HF2, 11.6.2, 11.6.2 HF1, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4

Opened: Sep 02, 2014
Severity: 3-Major

Symptoms

When monitoring a very large number of virtual servers, GTM can mark objects down. This is more of a configuration issue. By setting Maximum Synchronous Monitor Requests to a value as high as 10,000 monitors at a time, GTMD is going to attempt to probe that many monitors at once. Because there is not enough processing power and memory for either GTMD or big3d to handle all of these requests and responses at once, the issue occurs.

Impact

Load balancing can be disrupted because the status of the virtual server is being marked unavailable. The log shows the following message, indicating that there was no reply from big3d (timed out): alert gtmd[xxxx]: xxxxxxxx:x: Monitor instance /Common/bigip xx.xx.xx.xx:80 UP -- DOWN from /Common/xxxx.

Conditions

GTM monitoring a very large number of virtuals with maximum synchronous monitor requests set very high.

Workaround

Set gtm.global-settings.metrics.max-synchronous-monitor-requests to a value that does not overwhelm the big3d on the other device.

Fix Information

None

Behavior Change