Last Modified: May 29, 2024
Affected Product(s):
F5OS Velos
Known Affected Versions:
F5OS-C 1.5.1
Fixed In:
F5OS-C 1.6.0
Opened: Dec 20, 2022 Severity: 2-Critical
Tenants not coming up correctly after upgrade or blade reboot. The tenants will be stuck in ContainerCreating in the "oc get pods --all-namespaces" output partition-2 virt-launcher-velos1-cf-gslb-2-gksrp 0/1 ContainerCreating 0 46m <none> blade-2.chassis.local <none> partition-2 virt-launcher-velos1-cf-rprxy1-1-jcw9k 0/1 ContainerCreating 0 46m <none> blade-1.chassis.local <none> partition-2 virt-launcher-velos1-cf-rprxy2-2-gl7kw 0/1 ContainerCreating 0 46m <none> blade-2.chassis.local <none> partition-2 virt-launcher-velos1-cloud-rprxy1-1-kwg4b 0/1 ContainerCreating 0 46m <none> blade-1.chassis.local <none> If this condition is hit, the token can validated to be bad from the CC shell with the following command: oc get pods -n partition-<#> --token="`cat /tmp/omd/tokens/partition-<#>/tokens/partition-<#>-saToken`" e.g. [root@controller-1 ~]# oc get pods -n partition-6 --token="`cat /tmp/omd/tokens/partition-6/tokens/partition-6-saToken`" NAME READY STATUS RESTARTS AGE lldpd-6d4458d967-xfs7d 0/1 Pending 0 7m stpd-6f844d8d65-wf6s8 0/1 Pending 0 7m tmstat-rsync-65c9cfb8b9-m2j7j 0/1 Pending 0 7m [root@controller-1 ~]# If the token is bad, an error will happen. [root@controller-1 ~]# oc get pods -n partition-3 --token="`cat /tmp/omd/tokens/partition-3/tokens/partition-3-saToken`" No resources found. error: You must be logged in to the server (Unauthorized) [root@controller-1 ~]#
Tenants will not start correctly, causing an outage.
This is related to deleting and re-creating partitions, and then upgrading or rebooting blades, but does not happen every time. There may be other conditions that can cause this.
The workaround is to remove the token files from the /tmp/omd/tokens/partition-<#>/tokens directory. e.g., rm /tmp/omd/tokens/partition-1/tokens/partition-1-saToken orchestration-manager will then regenerate the token file with the correct partition token.
N/A