Bug ID 800185: Saving a large encrypted UCS archive may fail and might trigger failover

Last Modified: Sep 19, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP All(all modules)

Known Affected Versions:
12.1.0, 12.1.0 HF1, 12.1.0 HF2, 12.1.1, 12.1.1 HF1, 12.1.1 HF2, 12.1.2, 12.1.2 HF1, 12.1.2 HF2, 12.1.3, 12.1.3.1, 12.1.3.2, 12.1.3.3, 12.1.3.4, 12.1.3.5, 12.1.3.6, 12.1.3.7, 12.1.4, 12.1.4.1, 12.1.5, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.4, 14.1.0.5, 14.1.0.6, 14.1.2, 15.0.0, 15.0.1

Opened: Jun 28, 2019
Severity: 2-Critical

Symptoms

-- When saving a very large encrypted UCS file, you encounter an error: # tmsh save /sys ucs my_ucs passphrase <mysecret> Saving active configuration... Can't fork at /usr/local/bin/im line 305. /var/tmp/configsync.spec: Error creating package -- If saving UCS is automated you may find related errors in /var/log/audit: err scriptd[45342]: 014f0013:3: Script (/Common/f5.automated_backup__backup) generated this Tcl error: (script did not successfully complete: (UCS saving process failed. while executing "tmsh::save /sys ucs $fname )) -- Other services might be restarted due to lack of memory, which might result in failover.

Impact

The operation might consume as much as twice the UCS file size in RAM. The UCS may not get saved correctly, and if not enough memory is available, low free memory symptoms will be apparent. The latter may result in services being killed to free memory, resulting in service impact and failover, though it is quite typical for the overly large process saving the UCS to be terminated with no other impact.

Conditions

Large encrypted UCS files and low free host memory. UCS file sizes in hundreds of MB are much more likely to encounter this issue, along with free memory less than 1 GB.

Workaround

A mitigation is to minimise UCS file size. UCS files large enough to encounter this issue typically contain very large files, some of which may not be needed or are no longer necessary. Remove unnecessary large files from directories that contribute to UCS archives, for example, stray, large files such as packet captures in /config or its subdirectories. (For help understanding what is in UCS archives, see K12278: Removing non-essential files from a UCS when disk space errors are encountered :: https://support.f5.com/csp/article/K12278.) If using APM, remove unnecessary EPSEC ISO files. (For more information, see K21175584: Removing unnecessary OPSWAT EPSEC packages from the BIG-IP APM system :: https://support.f5.com/csp/article/K21175584.

Fix Information

None

Behavior Change