Bug ID 1284705: Appliance Orchestration Manager core file may consume entire root filesystem

Last Modified: Jul 25, 2024

Affected Product(s):
F5OS F5OS(all modules)

Known Affected Versions:
F5OS-A 1.3.0, F5OS-A 1.3.1, F5OS-A 1.3.2, F5OS-A 1.4.0

Fixed In:
F5OS-A 1.5.0

Opened: Apr 04, 2023

Severity: 1-Blocking

Symptoms

If the Appliance Orchestration Manager daemon crashes, it may write an extremely large (~80-90 GiB) core file that consumes all of the free space on the root filesystem and causes the system to become inoperative. For more information, see: - https://cdn.f5.com/product/bugtracker/ID1328977.html - https://cdn.f5.com/product/bugtracker/ID1296997.html

Impact

System is inoperative, and only accessible by logging in to the system command line as root: - Tenants are inoperative/inaccessible - System webUI is inaccessible - Attempts to log in to CLI as any user other than "root" fails

Conditions

Unknown.

Workaround

There is no way to avoid this issue. Once a system is affected, the system can be recovered by removing the large core file from /var/shared/core/container/ and then rebooting the system. 1. Log in to the system CLI as root and check the amount of free space on the root file system: df -h / 2. If that reports no free space, check whether there are large core files present in /var/shared core by running: du -h /var/shared/core ls -lSh /var/shared/core/container/ 3. To free up the space, remove the large "core.appliance_orch.appliance_orchestration_manager" core files from /var/shared/core/container/

Fix Information

For more information, see: - appliance_orchestration_manager free invalid pointer error and restart https://cdn.f5.com/product/bugtracker/ID1328977.html - Improve F5OS systems on dumping large core files that yields to out of disk space https://cdn.f5.com/product/bugtracker/ID1296997.html

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips