Bug ID 894593: High CPU usage caused by the restjavad daemon continually crashing and restarting

Last Modified: Nov 07, 2022

Bug Tracker

Affected Product:  See more info
BIG-IP All(all modules)

Known Affected Versions:
14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2

Opened: Mar 31, 2020
Severity: 3-Major

Symptoms

Restjavad may become unstable if the amount of memory required by the daemon exceeds the value allocated for its use.

Impact

The overall system performance is degraded during the continuous restart of the restjavad daemon due to a relatively high CPU usage.

Conditions

The memory required by the restjavad daemon may grow significantly in system configurations with either a high volume of device statistics collection (AVR provisioning), or a with relatively large number of LTM objects managed by the REST framework (SSL Orchestrator provisioning).

Workaround

Please don't apply the workarounds below if encountering issues after upgrade to 14.1.5.1-, 15.1.7-, 16.1.3.1- and 17.0.0.1 and you already have restjavad.useextramb set to true. If you have low restjavad memory under these conditions it is likely you are encountering a problem caused by the behaviour change introduced in ID 1025261 ( https://cdn.f5.com/product/bugtracker/ID1025261.html ). The linked article has suggestions on how to mitigate the issue. If you have restjavad.usextramb set to false and need more memory after upgrade to a version above you will also need to set provision.restjavad.extramb to a sensible value as well as the commands below - typically something like 384 + 80% of MIN (provision.extramb | 2500), so 1984 MB for example below. That's a high value and it may be possible to set it lower eg it may be worth trying 384 + 20% of MIN(provision.extramb|2500) which is 784 MB for example beneath. You can try different values quite quickly by changing provision.restjavad.extramb and restarting restjavad which should only effect availability of REST API for a few seconds. Generally 384 MB should be seen as the minimum. Increase the memory allocated for the restjavad daemon (e.g., 2 GB), by running the following commands in a BIG-IP terminal. tmsh modify sys db restjavad.useextramb value true tmsh modify sys db provision.extramb value 2000 bigstart restart restjavad Note changing provision.extramb is service effecting and systems may take several minutes to return to a state they could handle traffic. It also needs to be set on each peer of a service cluster. Note this may lead to impact on multi-module systems with ASM as approximately only 50-60% of provision.extramb value would be allocated as extra host memory and restjavad may take up to 80% of provision.extramb. It also lowers the ASM specific host allocation resulting in some tighter memory constraints on ASM daemons. Try to use the smallest value that works.

Fix Information

None

Behavior Change