Bug ID 913329: BIG-IQ analytics data retention policy and data aggregation might not work as expected

Last Modified: Sep 13, 2023

Affected Product(s):
BIG-IQ AppIQ(all modules)

Known Affected Versions:
7.0.0.1, 7.0.0.2, 7.1.0, 7.1.0.1, 7.1.0.2, 7.1.0.3, 7.1.6, 7.1.6.1, 7.1.7, 7.1.7.1, 7.1.7.2, 7.1.8, 7.1.8.1, 7.1.8.2, 7.1.8.3, 7.1.8.4, 7.1.8.5, 7.1.9, 7.1.9.7, 7.1.9.8, 7.1.9.9

Fixed In:
8.1.0.2

Opened: Jun 01, 2020

Severity: 3-Major

Symptoms

BIG-IQ logs for AppIQ/postaggregator are filled with memory exceptions (java.lang.OutOfMemoryError) Location of log: var/log/appiq/postaggregator.log Sample trace in log: ERROR c.f.a.a.TimeRangeAggregator [scheduling-1] Exception thrown while attempting to perform index aggregations java.lang.OutOfMemoryError: Java heap space

Impact

The BIG-IQ disk space might fill up quickly, resulting in overall instability.

Conditions

When BIG-IQ is configured to collect BIG-IP data.

Workaround

Expand JVM memory allocated to both post-aggregator service using the following steps from the BIG-IQ command line: 1. vi /etc/biq_daemon_provision.json 2. edit the restjavad memory setting: "big_iq": { "restjavad": { "active": true, "memory_allocation": { "SYS_4GB": "800m", "SYS_8GB": "3500m", "SYS_16GB": "6000m", "SYS_32GB": "12700m", -->>> change this to "9800m" "SYS_64GB": "20000m", "SYS_128GB": "20000m" }, "new_ratio": { "SYS_32GB": "1" } }, 3. edit the elasticsearch memory (the one under "big_iq") }, "elasticsearch": { "active": true, "memory_allocation": { "SYS_4GB": "100m", "SYS_8GB": "200m", "SYS_16GB": "500m", "SYS_32GB": "1600m", -->>> increase this to "4000m" "SYS_64GB": "3200m", "SYS_128GB": "6400m" } }, 4. bigstart restart restjavad 5. bigstart restart elasticsearch For post aggregator service: for versions < 8.0.0: 1. From BIG-IQ, start an SSH session 2. Edit the post-aggregator service config file /etc/bigstart/scripts/appiqpostaggregator 3. Change the line: From: MAX_JVM_HEAP=200m To: MAX_JVM_HEAP=1000m 4. Save the config file. 5. Restart post-aggregator service by typing the following command: bigstart restart appiqpostaggregator For BIG-IQ version later than 8.0.0: 1. vi /etc/biq_daemon_provision.json 2. edit the appiqpostaggregator memory (the one under "big_iq") "appiqpostaggregator": { "active": true, "memory_allocation": { "SYS_4GB": "100m", "SYS_8GB": "300m", "SYS_16GB": "500m", "SYS_32GB": "500m", -----> increase this to 1000m "SYS_64GB": "500m", "SYS_128GB": "500m" } }, 3. Restart post-aggregator service by typing the following command: bigstart restart appiqpostaggregator

Fix Information

Memory exceptions no longer happen if BIG-IQ has 64GB of memory. If BIG-IQ has 32GB or less, you can use the workaround to avoid this issue.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips