Bug ID 913329: BIG-IQ analytics data retention policy and data aggregation may not work as expected

Last Modified: Jan 25, 2021

Bug Tracker

Affected Product:  See more info
BIG-IQ AppIQ(all modules)

Known Affected Versions:
7.0.0, 7.0.0.1, 7.1.0, 7.1.0.1, 7.1.0.2

Opened: Jun 01, 2020
Severity: 3-Major

Symptoms

AppIQ/postaggregator logs are filled with memory exceptions (java.lang.OutOfMemoryError) Location of log: var/log/appiq/postaggregator.log Sample trace in log: 2020-05-07 00:10:15,010 ERROR c.f.a.a.TimeRangeAggregator [scheduling-1] Exception thrown while attempting to perform index aggregations java.lang.OutOfMemoryError: Java heap space

Impact

- The BIG-IQ disk space might fill up quickly, resulting in overall instability

Conditions

BIG-IQ configured to collect data.

Workaround

Please follow below steps to expand JVM memory allocated to both post-aggregator service and the Elasticsearch instance on the CM. For Elasticsearch instance on the CM: 1. vi /etc/biq_daemon_provision.json 2. edit the restjavad memory setting: "big_iq": { "restjavad": { "active": true, "memory_allocation": { "SYS_4GB": "800m", "SYS_8GB": "3500m", "SYS_16GB": "6000m", "SYS_32GB": "12700m", -->>> change this to "9800m" "SYS_64GB": "20000m", "SYS_128GB": "20000m" }, "new_ratio": { "SYS_32GB": "1" } }, 3. edit the elasticsearch memory (the one under "big_iq") }, "elasticsearch": { "active": true, "memory_allocation": { "SYS_4GB": "100m", "SYS_8GB": "200m", "SYS_16GB": "500m", "SYS_32GB": "1600m", -->>> increase this to "4000m" "SYS_64GB": "3200m", "SYS_128GB": "6400m" } }, 4. bigstart restart restjavad 5. bigstart restart elasticsearch For post aggregator service: 1. On the CM console start a SSH session 2. Edit the post-aggregator service config file /etc/bigstart/scripts/appiqpostaggregator 3. Change below line Old: MAX_JVM_HEAP=200m New: MAX_JVM_HEAP=1000m 4. Save the config file 5. Restart post-aggregator service bigstart restart appiqpostaggregator

Fix Information

None

Behavior Change