Bug ID 828937: Some systems can experience periodic high IO wait due to AVR data aggregation

Last Modified: Nov 27, 2020

Bug Tracker

Affected Product:  See more info
BIG-IP AVR(all modules)

Known Affected Versions:
13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4

Fixed In:
16.0.0, 15.1.0.5, 14.1.2.5, 13.1.3.4

Opened: Sep 19, 2019
Severity: 2-Critical
Related AskF5 Article:
K45725467

Symptoms

Systems with a large amount of statistics data collected in the local database (i.e., systems not working with BIG-IQ) can have high IO Wait CPU usage, peaking at 10 minutes, 1 hour, and 24 hours. This is caused by the data aggregation process that is running on the local database. Notice that large memory footprints, particularly for avrd might be a symptom for the phenomenon.

Impact

High IO can impact various processes on BIG-IP systems. Some of them can experience timeouts and might restart.

Conditions

-- The BIG-IP system is collecting statistics locally (i.e., not sending data to BIG-IQ or another external device). -- There is a large amount of statistics data. -- May occur even if AVR is not explicitly provisioned.

Workaround

The most effective workaround is to lower the amount of data collected by setting the 'avr.stats.internal.maxentitiespertable' DB variable to a lower value. The recommended values are 20000 (on larger, more powerful systems with more than 16 cores) or 2148 (on smaller systems). Note: After you lower the database value, continue to monitor the BIG-IP system for long I/O wait times and high CPU usage. If symptoms persist and the system continues to experience resource issues, you may need to reset the BIG-IP AVR statistics. For information about resetting BIG-IP AVR statistics, refer to K14956: Resetting BIG-IP AVR statistics :: https://support.f5.com/csp/article/K14956.

Fix Information

Set default value of avr.stats.internal.maxentitiespertable DB variable to 20000. Set it to 2148 on systems with fewer than or equal to CPU 16 cores.

Behavior Change