Bug ID 776393: Restjavad restarts frequently due to insufficient memory with relatively large configurations

Last Modified: Sep 13, 2023

Affected Product(s):
BIG-IP TMOS(all modules)

Known Affected Versions:
13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 13.1.3.4, 13.1.3.5, 13.1.3.6, 13.1.4, 13.1.4.1, 13.1.5, 13.1.5.1, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1

Fixed In:
16.1.0, 16.0.1.1, 15.1.3, 14.1.4

Opened: Apr 25, 2019

Severity: 2-Critical

Symptoms

Restjavad restarts frequently -- approximately every 5 minutes -- due to the JVM heap running out of memory.

Impact

REST API intermittently unavailable.

Conditions

-- BIG-IP system with no extra memory given to restjavad. -- The configuration contains a large number of configuration items related to APM access-policies, APM policy-items, APM policy agents, LTM nodes, LTM rules, DNS Requests, sys application services, LTM data-groups, LTM profiles, security bot-defense profiles, and sys file ssl-certs.

Workaround

Give restjavad extra memory, using the following commands. The example below allocates 1600 MB of extra memory to restjavad: tmsh modify sys db restjavad.useextramb value true tmsh modify sys db provision.extramb value 2000 bigstart restart restjavad Note: The increase in restjavad size will be slightly greater than extra host memory from provision.extramb on systems using modules other than just LTM and GTM. To have this change persist past system reboots, save the configuration with: tmsh save sys config

Fix Information

Default restjavad heap memory has been increased to 384 MB. Minimum heap size has been set to the same value as maximum heap size.

Behavior Change

Extra memory is allocated to Java. The maximum heap size was increased to 384 MB from 192 MB previously. The minimum heap size has been set to same size as maximum heap size. This will be particularly apparent where restjavad.useextramb is set to the value true and provision.extramb is set to a high value but restjavad hadn't required much extra memory previously. Note though in this case the Linux kernel does not usually back all the memory requested with physical memory unless it is used. ID 1025261 ( https://cdn.f5.com/product/bugtracker/ID1025261.html ) introduces sys db variable provision.restjavad.extramb to allow finer grained control of restjavad memory size. See the Behavior Change text in that bug for more information.

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips