Bug ID 1002969: Csyncd can consume excessive CPU time

Last Modified: Oct 04, 2024

Affected Product(s):
BIG-IP Install/Upgrade, TMOS(all modules)

Known Affected Versions:
13.1.5, 13.1.5.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 14.1.5.3, 14.1.5.4, 14.1.5.6, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 15.1.9, 15.1.9.1, 15.1.10, 15.1.10.2, 15.1.10.3, 15.1.10.4, 15.1.10.5, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 16.1.4, 16.1.4.1, 16.1.4.2, 16.1.4.3, 16.1.5, 16.1.5.1, 17.0.0, 17.0.0.1, 17.0.0.2, 17.1.0, 17.1.0.1, 17.1.0.2, 17.1.0.3, 17.1.1, 17.1.1.1, 17.1.1.2, 17.1.1.3, 17.1.1.4

Opened: Mar 16, 2021

Severity: 3-Major

Symptoms

Following a configuration change or software upgrade, the "csyncd" process becomes always busy, consuming excessive CPU.

Impact

The overuse of CPU resources by "csyncd" may starve other control-plane processes. Handling of payload network traffic by the data plane is not directly affected.

Conditions

-- occurs on a multi-blade VIPRION chassis or VELOS tenant -- may occur with or without vCMP -- may occur after configuring F5 Telemetry Streaming, but may also occur in other circumstances -- large numbers of files are contained in one or more of the directories being sync'ed between blades

Workaround

To mitigate the processing load, identify which directory or directories contain excessive numbers of files being replicated between blades by "csyncd". If this replication is not absolutely needed (see below), such a directory can be removed from the set of directories being sync'ed. For example: if there are too many files being generated in the "/run/pamcache" directory (same as "/var/run/pamcache"), remove this directory from the set being acted upon by "csyncd" by running the following commands to comment-out the associated lines in the configuration file. [Note it is better to follow the more complete workaround from ID 1103369, https://cdn.f5.com/product/bugtracker/ID1103369.html ] # clsh "cp /etc/csyncd.conf /etc/csyncd.conf.$(date +%Y%m%d_%H%M%S)" # clsh "sed -i '/run\/pamcache/,+2s/^/#/' /etc/csyncd.conf" # clsh "bigstart restart csyncd" If the problem was observed soon after the installation of F5 Telemetry Streaming, the configuration can be adjusted to make csyncd ignore the related files in a subdirectory of "/var/config/rest/iapps". Run the following commands: # clsh "cp /etc/csyncd.conf /etc/csyncd.conf.$(date +%Y%m%d_%H%M%S)" # clsh "sed -i '/\/var\/config\/rest\/iapps/a \ \ \ \ \ \ \ \ ignore f5-telemetry' /etc/csyncd.conf" # clsh "bigstart restart csyncd" ---- The impact of disabling replication for the pamcache folder is that in the event of a primary blade failover, the new primary blade would not be aware of the existing valid auth tokens, so the user (eg, a GUI user, or a REST script already in progress at the time of the failover) would need to authenticate again. The impact of disabling replication for a folder under the /var/config/rest/iapps is that in the event of a primary blade failover, the new primary blade would not be aware of the iApps LX package, so the user would need to install the iApps LX package on the new primary blade.

Fix Information

None

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips