Bug ID 745882: Tokumond service restarts due to exhausted memory heap

Last Modified: Jul 21, 2019

Bug Tracker

Affected Product:  See more info
BIG-IQ Platform(all modules)

Known Affected Versions:
6.0.0, 6.0.1

Opened: Oct 03, 2018
Severity: 3-Major

Symptoms

node daemon running script tokumon.js restarts every few minutes due to memory issues with the following error: Core was generated by `/usr/bin/node --max_old_space_size=440 /usr/share/rest/tokumon/src/tokumon.js' /var/log/tokumon/ FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Impact

You cannot access BIG-IQ when this happens.

Conditions

The issue happens when large audit logger documents exist, which causes memory issues.

Workaround

To work around this issue: 1. Remount /usr as read-write with the command: # mount -o remount,rw /usr 2. Go to the directory below: # cd /usr/share/rest/tokumon/config 3. Edit the file 'white-list.js' and remove the line that contains the string "audit-logger" (first, copy the line to the clipboard). 4. Edit 'black-list.js' and insert the copied line (which should be: '/audit-logger',) in line 3, right after the var definition. 5. Clean and restart tokumond by running these three commands: # bigstart stop tokumond # /usr/share/rest/tokumon/clean.sh # bigstart start tokumond 6. Remount /usr as read only # mount -o remount,ro /usr

Fix Information

None

Behavior Change