Bug ID 1196017: Kube-flannel stuck in ImagePullBackOff status due to wrong port or tag

Last Modified: Aug 01, 2024

Affected Product(s):
F5OS None(all modules)

Known Affected Versions:
F5OS-A 1.3.2, F5OS-A 1.3.1, F5OS-A 1.3.0, F5OS-A 1.2.0, F5OS-A 1.1.1

Fixed In:
F5OS-A 1.7.0

Opened: Nov 29, 2022

Severity: 3-Major

Symptoms

On the rSeries appliance, the tenant deployment fails as the kube-flannel is stuck in ImagePullBackOff status due to the wrong port or tag.

Impact

Tenant deployment fails.

Conditions

The exact conditions are unclear but it is observed after the upgrade from F5OS-A 1.1.1.

Workaround

1) Check the expected registry port for the current running image. Example: On an appliance running with F5OS-A 1.4.0 registry port found as 2004 (I could change on other devices) [root@appliance-1 ~]# ls -l /var/docker/config/platform.yml lrwxrwxrwx. 1 root root 52 Feb 13 13:11 /var/docker/config/platform.yml -> /var/docker/config/appliance/1.4.0-7488/platform.yml [root@appliance-1 ~]# [root@appliance-1 ~]# mount | grep "1.4.0" | grep "registry" /var/export/chassis/import/.mounts/services/R5R10/1.4.0-7488/volume.img on /var/export/chassis/import/.volumes/appliance-services-registry-2004-volume type squashfs (ro,relatime,context=system_u:object_r:container_file_t:s0) 2) Check the expected flannel tag under the identified registry port Example: [root@appliance-1 ~]# crictl images | grep flannel | grep 2004 localhost:2004/appliance-flannel 0.13.1 0a69e5ee8f6ef 20.7MB [root@appliance-1 ~]# 3) Check the kube-flannel port under the DaemonSet in /tmp/omd/scripts/kube-flannel.yml Example: [root@appliance-1 ~]# grep -i "image:" /tmp/omd/scripts/kube-flannel.yml image: localhost:2004/appliance-flannel:0.13.1 image: localhost:2004/appliance-flannel:0.13.1 [root@appliance-1 ~]# [root@appliance-1 ~]# kubectl get ds -A NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system kube-flannel-ds 1 1 1 1 1 <none> 40d kube-system klipper-lb 1 1 1 1 1 <none> 40d kube-system kube-multus-ds-amd64 1 1 1 1 1 kubernetes.io/arch=amd64 40d kubevirt virt-handler 1 1 1 1 1 kubernetes.io/os=linux 40d [root@appliance-1 ~]# [root@appliance-1 ~]# kubectl describe ds kube-flannel-ds -n kube-system | grep -i "image" Image: localhost:2004/appliance-flannel:0.13.1 Image: localhost:2004/appliance-flannel:0.13.1 [root@appliance-1 ~]# 4) Edit the DaemonSet of kube-flannel and correct the registry port or tag as applicable Example: # kubectl edit ds kube-flannel-ds -n kube-system 5) After editing the port or tag on the flannel DaemonSet, reboot the system and check whether the pod is starting or not.

Fix Information

None

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips