Bug ID 574052: GTM autoconf can cause high CPU usage for gtmd

Last Modified: Apr 10, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP GTM(all modules)

Known Affected Versions:
11.4.1, 11.5.0, 11.5.1, 11.5.1 HF1, 11.5.1 HF10, 11.5.1 HF11, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.2, 11.5.2 HF1, 11.5.3, 11.5.3 HF1, 11.5.3 HF2, 11.5.4, 11.5.4 HF1, 11.5.4 HF2, 11.5.4 HF3, 11.5.4 HF4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.6.1, 11.6.1 HF1, 12.0.0, 12.0.0 HF1, 12.0.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.0, 12.1.0 HF1, 12.1.0 HF2, 12.1.1, 12.1.1 HF1, 12.1.1 HF2

Fixed In:
13.0.0, 12.1.2, 11.6.1 HF2

Opened: Feb 16, 2016
Severity: 3-Major

Symptoms

The autoconf feature of GTM can cause high CPU utilization (~90%) under certain situations. In large configurations of LTM vses that contain "." (dot) in the name.

Impact

CPU usage is high, which may impact monitoring and LB decisions.

Conditions

Large configuration of LTM VS that contain "." in the name have the name converted ("." is replaced by "_") and the LTM VS name is saved to the config. This causes the matching algorithm in autoconf to spend many CPU cycles walking the list of VS to find a match. This problem is caused by large numbers of VSes on a GTM Server. (10k VSes on 10k Server is less of an issue than 10k VSes on 1 GTM Server)

Workaround

There are some mitigations. The preferable (for performance and stability) are listed first. 1. Rename the virtual servers on the LTM to remove the "." This would require deleting the GTM configuration and rediscovering it and recreating pools. 2. Turn off autoconf. Run autoconf once to populate the config, then turn it off. 3. Reduce the frequency of autoconf. It will still cause a high CPU usage scenario, but it will be less frequent. Versions 12.0.0 and higher do not convert the "." to "_". So that problem is eliminated for new configurations. If a customer upgrades to 12.0.0 and the config still contains VS names that were previously converted, they still may run into high CPU usage. Upgrading to 12.0.0 alone does not fix this issue, a reconfig would be necessary.

Fix Information

Change algorithm used to match LTM VS names to GTM VS to reduce linear walk of all VSes on a server.

Behavior Change