Bug ID 828937: Some systems can experience periodic high IO wait due to AVR data aggregation

Last Modified: Apr 18, 2024

Affected Product(s):
BIG-IP AVR(all modules)

Known Affected Versions:
13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4

Fixed In:
16.0.0, 15.1.0.5, 14.1.2.5, 13.1.3.4

Opened: Sep 19, 2019

Severity: 2-Critical

Related Article: K45725467

Symptoms

Systems with a large amount of statistics data collected in the local database (i.e., systems not working with BIG-IQ) can have high IO Wait CPU usage, peaking at 10 minutes, 1 hour, and 24 hours. This is caused by the data aggregation process that is running on the local database. Notice that large memory footprints, particularly for avrd might be a symptom for the phenomenon.

Impact

High IO can impact various processes on BIG-IP systems. Some of them can experience timeouts and might restart.

Conditions

-- The BIG-IP system is collecting statistics locally (i.e., not sending data to BIG-IQ or another external device). -- There is a large amount of statistics data. -- May occur even if AVR is not explicitly provisioned (in that case, ASM, APM, PEM, AFM, or AAM must be provisioned).

Workaround

The most effective workaround is to lower the amount of data collected by setting the 'avr.stats.internal.maxentitiespertable' DB variable to a lower value. The recommended values are 20000 (on larger, more powerful systems with more than 8 cores) or 2148 (on smaller systems). Note: After you lower the database value, continue to monitor the BIG-IP system for long I/O wait times and high CPU usage. If symptoms persist and the system continues to experience resource issues, you may need to reset the BIG-IP AVR statistics. For information about resetting BIG-IP AVR statistics, refer to K14956: Resetting BIG-IP AVR statistics :: https://support.f5.com/csp/article/K14956.

Fix Information

Set default value of avr.stats.internal.maxentitiespertable DB variable to 2148 on systems with the number of CPU cores fewer than or equal to 8.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips