Bug ID 1211465: Partition openshift tokens may go invalid, causing tenants to not start after configuration or reboot

Last Modified: May 29, 2024

Affected Product(s):
F5OS Velos(all modules)

Known Affected Versions:
F5OS-C 1.5.1

Fixed In:
F5OS-C 1.6.0

Opened: Dec 20, 2022

Severity: 2-Critical


Tenants not coming up correctly after upgrade or blade reboot. The tenants will be stuck in ContainerCreating in the "oc get pods --all-namespaces" output partition-2 virt-launcher-velos1-cf-gslb-2-gksrp 0/1 ContainerCreating 0 46m <none> blade-2.chassis.local <none> partition-2 virt-launcher-velos1-cf-rprxy1-1-jcw9k 0/1 ContainerCreating 0 46m <none> blade-1.chassis.local <none> partition-2 virt-launcher-velos1-cf-rprxy2-2-gl7kw 0/1 ContainerCreating 0 46m <none> blade-2.chassis.local <none> partition-2 virt-launcher-velos1-cloud-rprxy1-1-kwg4b 0/1 ContainerCreating 0 46m <none> blade-1.chassis.local <none> If this condition is hit, the token can validated to be bad from the CC shell with the following command: oc get pods -n partition-<#> --token="`cat /tmp/omd/tokens/partition-<#>/tokens/partition-<#>-saToken`" e.g. [root@controller-1 ~]# oc get pods -n partition-6 --token="`cat /tmp/omd/tokens/partition-6/tokens/partition-6-saToken`" NAME READY STATUS RESTARTS AGE lldpd-6d4458d967-xfs7d 0/1 Pending 0 7m stpd-6f844d8d65-wf6s8 0/1 Pending 0 7m tmstat-rsync-65c9cfb8b9-m2j7j 0/1 Pending 0 7m [root@controller-1 ~]# If the token is bad, an error will happen. [root@controller-1 ~]# oc get pods -n partition-3 --token="`cat /tmp/omd/tokens/partition-3/tokens/partition-3-saToken`" No resources found. error: You must be logged in to the server (Unauthorized) [root@controller-1 ~]#


Tenants will not start correctly, causing an outage.


This is related to deleting and re-creating partitions, and then upgrading or rebooting blades, but does not happen every time. There may be other conditions that can cause this.


The workaround is to remove the token files from the /tmp/omd/tokens/partition-<#>/tokens directory. e.g., rm /tmp/omd/tokens/partition-1/tokens/partition-1-saToken orchestration-manager will then regenerate the token file with the correct partition token.

Fix Information


Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips