Bug ID 522241: Using tmsh to display the number of elements in a DNS cache may cause high CPU utilization, and the tmsh command may not complete

Last Modified: May 01, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP DNS, GTM(all modules)

Known Affected Versions:
11.5.1, 11.5.1 HF1, 11.5.1 HF10, 11.5.1 HF11, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.2, 11.5.2 HF1, 11.5.3, 11.5.3 HF1, 11.5.3 HF2, 11.5.4, 11.5.4 HF1, 11.5.4 HF2, 11.5.4 HF3, 11.5.4 HF4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.6.1, 11.6.1 HF1, 11.6.1 HF2, 11.6.2, 11.6.2 HF1, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 12.0.0, 12.0.0 HF1, 12.0.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.0, 12.1.0 HF1, 12.1.0 HF2, 12.1.1, 12.1.1 HF1, 12.1.1 HF2, 12.1.2, 12.1.2 HF1, 12.1.2 HF2, 12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 12.1.4.1, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.4

Opened: May 08, 2015
Severity: 3-Major

Symptoms

After running the tmsh command "show ltm dns cache records <key|msg|nameserver|rrset> cache <name> count-only" you may experience the following symptoms: - One of the TMM instances on the system climbs to 100% CPU utilization for a prolonged amount of time. - The odd-numbered hyperthread (i.e. 1) corresponding to the even-numbered hyperthread (i.e. 0) where the busy TMM instance is running is partially halted by the HT-Split feature (this will be observable in utilities such as "top" and by the presence of "Idle enforce starting" log messages in the /var/log/kern.log file). - After waiting for a very long time, the tmsh command may not actually return and display a record count. - The tmsh command does not respond to CTRL+C and continues running.

Impact

Due to the high CPU utilization, traffic handling is impaired. Control-plane processes can also become affected, leading to different issues (this depends on the size and load of the BIG-IP system). For example, the lacpd process can become descheduled causing trunks to flap.

Conditions

A DNS cache contains a large number of records and the BIG-IP Administrator runs the following tmsh command to determine the exact record count: "show ltm dns cache records <key|msg|nameserver|rrset> cache <name> count-only"

Workaround

Do not run the specified tmsh command. If you have run the specified tmsh command and this has not returned after a very long time and you want restore normal system operation, perform the following steps: 1) Press CTRL+Z to background execution of the command. 2) Enter the "killall -9 tmsh" command (if you have multiple tmsh commands running and only want to kill the affected one, you will have to identify the correct tmsh process using utilities such as ps and top). If your login shell is tmsh and not bash, simply close your SSH session to the BIG-IP system (as you won't be able to perform the aforementioned steps).

Fix Information

None

Behavior Change