Bug ID 1750613: If a system controller PXE boots and reimages, partitions may not start correctly, and cause data loss

Last Modified: Jun 19, 2025

Affected Product(s):
F5OS F5OS-C, Install/Upgrade, Velos(all modules)

Known Affected Versions:
F5OS-C 1.5.0, F5OS-C 1.5.1, F5OS-C 1.6.0, F5OS-C 1.6.1, F5OS-C 1.6.2, F5OS-C 1.8.0

Fixed In:
F5OS-C 1.8.1

Opened: Nov 16, 2024

Severity: 3-Major

Symptoms

If a system controller PXE boots, the partition instance restart on that controller may not work and the partition instance will be left in the "failed"/not running state with no configuration database. If that instance later becomes "active" it will overwrite the correct partition configuration database with the empty database. Example failed partition instance state: syscon-1-active# show partitions RUNNING BLADE OS SERVICE PARTITION SERVICE STATUS NAME ID VERSION VERSION CONTROLLER STATUS VERSION AGE ---------------------------------------------------------------------------------------- none - - - default 1 1.6.2-22734 1.6.2-22734 1 running-active 1.6.2-22734 40m 2 failed - 11m Normally following a controller reimage, the partitions will complete restart after all the ISOs are replicated to the controller and reimported. This may take 15 to 30 minutes depending on how many images are present. The partitions will show as "failed" while this resync occurs, and then they will start up normally. In the failure case, the instance stays "failed" indefinitely. Do NOT attempt to enable/disable the partition while it is in this "failed" state, or perform a software upgrade (set-version). If that happens, the "wiped" partition instance may start up and become Active, and all partition configuration will be lost.

Impact

All partition and tenant configuration in that partition is lost.

Conditions

This problem occurs when the partition is running a "patch" version of partition-services rather than a "base" version. Patch versions have a version number (major.minor.patch) that ends in a number other than “0” (zero). A race condition may occur between the completion of the partition ISO import and the initiation of the partition, resulting in a potential declaration of success despite failure. In such cases, the operation will not be retried. In this scenario, the partition might never get started, so it has no opportunity to form an HA pair with the other partition instance and synchronize the configuration database and tenant images. If it does eventually become Active it will erase all partition configurations.

Workaround

Following a PXE boot or reimage of the controller, check the status of all partition ISOs using the "show image partition" command. For patch versions, the partitions may stay in the "failed" state. However, for base versions, the partition should automatically restart and become running-standby within approximately 5 minutes after the ISOs have been imported. No further corrective action is necessary in this scenario. To recover force the partition instance startup code to retry by changing the partition configuration in a minimally disruptive way. Recommend toggling the partition mgmt-ip to 'none' and then back, as this will force the retry but not permanently change any configuration. Example: syscon-1-active(config)# partitions partition default config mgmt-ip ipv4 address 0.0.0.0 ; exit syscon-1-active(config)# commit Commit complete. syscon-1-active(config)# partitions partition default config mgmt-ip ipv4 address <ip address>; exit syscon-1-active(config)# commit Commit complete. syscon-1-active(config)# Do NOT attempt to enable/disable the partition while an instance is in this "failed" state following a reimage or perform a software upgrade (set-version). If that happens, the "wiped" partition instance may become Active, and all partition configuration will be lost.

Fix Information

Partitions restart and form an HA pair correctly following system controller reimage/replacement, regardless of partition services version.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips