Bug ID 1153865: Restjavad OutOfMemoryError errors and restarts after upgrade

Last Modified: Nov 29, 2022

Bug Tracker

Affected Product:  See more info
BIG-IP Install/Upgrade, TMOS(all modules)

Known Affected Versions:
17.0.0.1, 16.1.3.2, 16.1.3.1, 15.1.8, 15.1.7, 14.1.5.2, 14.1.5.1

Opened: Sep 12, 2022
Severity: 3-Major

Symptoms

After upgrade to an affected version, restjavad restarts intermittently or frequently, and/or may use high CPU. The restjavad logs, /var/log/restjavad.X.log, may report the following errors: java.lang.OutOfMemoryError: Java heap space restjavad may instead, or as well, run many full garbage collection cycles one after another, causing high CPU. This will be shown by frequent logs with [FullGC] in /var/log/restjavad-gc.log.X.current

Impact

May have problems in TMUI with certain pages or tabs, such as network map with very config or SSLO or iLX related tabs. Other services that use REST API, internal and external to BIG-IP, may be impacted with low performance or service instability

Conditions

- Update to affected version: 14.1.5.1-, 15.1.7-, 16.1.3.1-, 17.0.0.1- or later versions. - Value of sys db restjavad.useextramb is true. - Value of sys db provision.restjavad.extramb is 192 or lower than previous restjavad heap size. - Use of REST API calls that need a lot of memory. Heavy users of REST API may be very affected such as SSLO.

Workaround

Before upgrade - if you set sys db restjavad.useextramb to value false before install of new version you will have more restjavad memory, the default 384MB, after upgrade. tmsh modify sys db restjavad.useextramb value false If you restart restjavad you can see if that value works before upgrade. If you don't restart then it will come into effect after reboot. If that no longer has issues after update then leave that setting at false. Otherwise set back to true (no restart) and increase provision.restjavad.extramb as in After upgrade section below. After upgrade: Set sys db provision.restjavad.extramb to an appropriate value and restart restjavad. Run the following command: tmsh modify sys db provision.restjavad.extramb value X bigstart restart restjavad Iterate as necessary. The value of X is derived by using one of the following formulae: - When updating from versions before 14.1.4 and 15.1.3, to affected versions, a value that preserves the maximum previous restjavad heap size is: 192MB + 80% of MIN(provision.extramb|2500) the minimum possible heap size was: 192MB + 20% of MIN(provision.extramb|2500) The actual restjavad heap size would be between those extremes. SSLO systems would typically need the maximum. - When updating from 14.1.4-14.1.5, from 15.1.3-15.1.6.1 or from 16.0.x to affected versions: 384MB + 80% of MIN(provision.extramb|2500) - When updating from 16.1.0-16.1.3 or from 17.0.0.0 to affected versions: 384MB + 90% of MIN(provision.extramb|4000)

Fix Information

None

Behavior Change