Bug ID 499348: System statistics may fail to update, or report negative deltas due to delayed stats merging.

Last Modified: Apr 17, 2024

Affected Product(s):
BIG-IP All(all modules)

Known Affected Versions:
11.5.0, 11.5.1, 11.5.2, 11.5.3, 11.5.4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.5.10, 11.6.0, 11.6.1, 11.6.2, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 11.6.5, 11.6.5.1, 11.6.5.2, 11.6.5.3, 12.0.0, 12.0.0 HF1, 12.1.0 HF1, 12.0.0 HF2, 12.1.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.1 HF1, 12.1.1 HF2, 12.1.2 HF1, 12.1.2 HF2, 12.1.0, 12.1.1, 12.1.2, 12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 12.1.4.1, 12.1.5, 12.1.5.1, 12.1.5.2, 12.1.5.3, 12.1.6, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 13.1.3.4, 13.1.3.5, 13.1.3.6, 13.1.4, 13.1.4.1, 13.1.5, 13.1.5.1, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 14.1.5.3, 14.1.5.4, 14.1.5.6, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 15.1.9, 15.1.9.1, 15.1.10, 15.1.10.2, 15.1.10.3, 15.1.10.4, 16.0.0, 16.0.0.1, 16.0.1, 16.0.1.1, 16.0.1.2, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 16.1.4, 16.1.4.1, 16.1.4.2, 17.0.0, 17.0.0.1, 17.0.0.2

Opened: Jan 05, 2015

Severity: 3-Major

Related Article: K21811305

Symptoms

Under some conditions, the BIG-IP system might fail to report statistics over time. This can manifest as statistics reporting unchanging statistics (e.g., all zeroes (0)), or as sudden spikes in traffic, or as negative deltas in some counters. The system performance graphs will also appear to have gaps / be missing data at the times that this occurs.

Impact

Statistics fail to merge, which results in incorrect view of system behavior and operation.

Conditions

This occurs when there are frequent changes occurring to the underlying statistics data structures. This might occur under the following conditions: -- The system is spawning/reaping processes on a frequent basis (e.g., when there is a large number of external monitors). -- iRules are frequently using 'SSL::profile' to select different SSL profiles on a virtual server (this can cause per-virtual server, per-profile statistics to be created and deleted on a regular basis).

Workaround

This issue has two workarounds: 1. Reduce the frequency of changes in the statistics data structures. The specific action to take depends on what is triggering them. To do so, use any or all of the following: -- Reduce the frequency of configuration changes. -- Reduce the use of 'SSL::profile' in iRules. -- Reduce the number/frequency of processes being spawned by the system. 2. The second workaround has two parts: a) Switch statistics roll-ups to the 'slow_merge' method, which causes the system to spend more CPU merging statistics. To do so, set the 'merged.method' DB key to 'slow_merge' using the following command: tmsh modify sys db merged.method value slow_merge b) Change the merge-interval value to 2 to reduce CPU usage when merge-method is slow-merge. tmsh modify /sys db merged.merge.interval {value "2"} Note: Performing the second workaround has the side-effect of disabling tmstat snapshots on the device. The tmstat snapshots are intended for F5-internal use only: the lack of snapshots will have no bearing on the functionality of your system; however, F5 Support might be impacted in their ability to troubleshoot issues on your system.

Fix Information

None

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips