Bug ID 800185: Saving a large encrypted UCS archive may fail and might trigger failover

Last Modified: Sep 13, 2023

Affected Product(s):
BIG-IP All(all modules)

Known Affected Versions:
12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 12.1.4.1, 12.1.5, 12.1.5.1, 12.1.5.2, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3

Fixed In:
15.1.0, 15.0.1.4, 14.1.2.5, 13.1.3.4, 12.1.5.3

Opened: Jun 28, 2019

Severity: 2-Critical

Symptoms

-- When saving a very large encrypted UCS file, you may encounter an error: # tmsh save /sys ucs my_ucs passphrase <mysecret> Saving active configuration... Can't fork at /usr/local/bin/im line 305. /var/tmp/configsync.spec: Error creating package -- If saving UCS is automated you may find related errors in /var/log/audit: err scriptd[45342]: 014f0013:3: Script (/Common/f5.automated_backup__backup) generated this Tcl error: (script did not successfully complete: (UCS saving process failed. while executing "tmsh::save /sys ucs $fname )) -- Other services might be restarted due to lack of memory, which might result in failover. --System management via config utility or command line may be sluggish while UCS saves.

Impact

The operation uses at least 1.3 times the UCS file size of RAM. The UCS may not get saved correctly, and if not enough memory is available, low free-memory symptoms become apparent. The latter may result in services being killed to free memory, resulting in service impact and failover, though it is quite typical for the overly large process saving the UCS to be terminated with no other impact.

Conditions

-- Large encrypted UCS files and low free host memory. -- UCS file sizes in hundreds of MB are much more likely to encounter this issue, along with free memory less than 1 GB.

Workaround

A mitigation is to minimise UCS file size. UCS files large enough to encounter this issue typically contain very large files, some of which may not be needed or are no longer necessary. Remove unnecessary large files from directories that contribute to UCS archives, for example, stray, large files such as packet captures in /config or its subdirectories. (For help understanding what is in UCS archives, see K12278: Removing non-essential files from a UCS when disk space errors are encountered :: https://support.f5.com/csp/article/K12278.) If using APM, remove unnecessary EPSEC ISO files. (For more information, see K21175584: Removing unnecessary OPSWAT EPSEC packages from the BIG-IP APM system :: https://support.f5.com/csp/article/K21175584.

Fix Information

Saving a large UCS file no longer fails.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips