Bug ID 991617: Corrupted mapping prevents aggregation from all indexes and causing OOM and low disk space in the elasticsearch cluster

Last Modified: Nov 22, 2021

Bug Tracker

Affected Product:  See more info
BIG-IQ AppIQ(all modules)

Known Affected Versions:
7.0.0, 7.0.0.1, 7.0.0.2, 7.1.0, 7.1.0.1, 7.1.0.2, 7.1.0.3, 7.1.6, 7.1.6.1, 7.1.7, 7.1.7.1, 7.1.7.2, 7.1.8, 7.1.8.1, 7.1.8.2, 7.1.8.3, 7.1.8.4, 7.1.8.5, 7.1.9, 7.1.9.7, 7.1.9.8, 7.1.9.9

Fixed In:
8.0.0

Opened: Feb 09, 2021
Severity: 2-Critical

Symptoms

Indexes accumulate beyond their intended time and persist in Elasticsearch. Elasticsearch eventually becomes unresponsive due to lack of memory and disk space.

Impact

Although only one index is corrupted, all indexes are impacted. Information is lost from all indexes and it causes the whole elasticsearch cluster to be unresponsive due to no memory and disk space.

Conditions

In rare cases there is temporal corruption in the elasticsearch cluster, and an index might be created without a template. You will be able to identify the issue if the following message appears in /var/log/appiq/postaggregator.log on the device: Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: : Fielddata is disabled on text fields by default. Set fielddata=true on [dimensions.hash] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Workaround

N/A

Fix Information

Data aggregation no longer causes OOM for all indexes.

Behavior Change