Last Modified: Apr 26, 2024
Affected Product(s):
BIG-IP Install/Upgrade, TMOS
Known Affected Versions:
17.0.0.2, 17.0.0.1, 16.1.3.5, 16.1.3.4, 16.1.3.3, 16.1.3.2, 16.1.3.1, 15.1.8.2, 15.1.8.1, 15.1.8, 15.1.7, 14.1.5.4, 14.1.5.3, 14.1.5.2, 14.1.5.1
Fixed In:
17.1.0, 16.1.4, 15.1.9
Opened: Sep 12, 2022 Severity: 3-Major
After upgrade to an affected version, restjavad restarts intermittently or frequently, and/or may use high CPU. The restjavad logs, /var/log/restjavad.X.log, may report the following errors: java.lang.OutOfMemoryError: Java heap space restjavad may instead, or as well, run many full garbage collection cycles one after another, causing high CPU. This will be shown by frequent logs with [FullGC] in /var/log/restjavad-gc.log.X.current
May have problems in the GUI with certain pages or tabs, such as network map with very large config or SSL Orchestrator or iLX related tabs. Other services that use REST API, internal and external to BIG-IP, may be impacted with low performance or service instability
- Update to affected version: 14.1.5.1-, 15.1.7-15.1.8.2, 16.1.3.1-16.1.3.5, 17.0.0.1-17.0.0.2 - Value of sys db restjavad.useextramb is true. - Value of sys db provision.restjavad.extramb is 192 or lower than previous restjavad heap size. - Use of REST API calls that need a lot of memory. Heavy users of REST API, such as SSL Orchestrator, may be very affected.
Before upgrade to an affected version - if you set sys db restjavad.useextramb to value false before install of new version you will have more restjavad memory, the default 384MB, after upgrade. tmsh modify sys db restjavad.useextramb value false If you restart restjavad you can see if that value works before upgrade. If you don't restart then it will come into effect after reboot. If that no longer has issues after update then leave that setting at false. Otherwise set back to true (no restart) and increase provision.restjavad.extramb as in After upgrade section below. After upgrade: Set sys db provision.restjavad.extramb to an appropriate value and restart restjavad. Run the following command: tmsh modify sys db provision.restjavad.extramb value X bigstart restart restjavad Iterate as necessary. The value of X is derived by using one of the following formulae: - When updating from versions before 14.1.4 and 15.1.3, to affected versions, a value that preserves the maximum previous restjavad heap size is: 192MB + 80% of MIN(provision.extramb|2500) the minimum possible heap size was: 192MB + 20% of MIN(provision.extramb|2500) The actual restjavad heap size would be between those extremes. SSL Orchestrator systems would typically need the maximum. - When updating from 14.1.4-14.1.5, from 15.1.3-15.1.6.1 or from 16.0.x to affected versions: 384MB + 80% of MIN(provision.extramb|2500) - When updating from 16.1.0-16.1.3 or from 17.0.0.0 to affected versions: 384MB + 90% of MIN(provision.extramb|4000)
After upgrade, the system now sets the default value for provision.restjavad.extramb variable to 384MB. This sets the maximum heap size to 384MB. For values of provision.restjavad.extramb of 384 and lower the starting heap size is set at 96MB. For values above 384MB the starting heap size is set to the same value as maximum heap size. When upgrading from a version that has provision.restjavad.extramb and it is explicitly set to a value*, rather than just takes default value, that value will be preserved and take affect in the new version. This will happen even if restjavad.useextramb had been set to false. Where sys db restjavad.useextramb was set to value true in the previously used version, and provision.restjavad.extramb doesn't already exist, the value for provision.restjavad.extramb is based on a calculation of the maximum restjavad heap size that could have been used. Usually this maintains the same or very similar restjavad heap size as used previously with more ability to fine tune it. The default size works in a wider range of settings. When upgrading from a version that had a smaller starting heap size than maximum heap size, so before 14.1.4 or 15.1.3, and restjavad.useextramb set to true, it's possible that restjavad will use more memory than required. That's because for values above the default size of 384MB for provision.restjavad.extramb starting heap size is set the same as maximum heap size to lower performance issues when large memory sizes are required. You can lower restjavad memory use by lowering the value of provision.restjavad.extramb and restarting it if needed. Note: This fix also removes the db variable restjavad.useextramb as it is no longer needed. *Note: The command 'tmsh list sys db X' will return the value for DB key where set, or the default value for X otherwise. printdb can be used to display both value and default. eg printdb -n provision.restjavad.extramb Name: Provision.Restjavad.extraMB Realm: common Type: integer Default: 384 Value: 600 SCF_Config: true Min: 192 Max: 8192 shows a default of 384 and an explicitly set value of 600 that would override the default. Where the value hadn't been set the 'Value:' line will not be present.