Symptoms

When using the SOCK driver, the host-side (Azure) removal of the NIC Virtual Functions can cause a kernel deadlock on the pci_lock_rescan_remove() mutex

Impact

Kernel deadlock and subsequent reboot of the VE The kern.log file contains logs similar to these: info kernel: hv_netvsc 6045bdf2-e3b3-6045-bdf2-e3b36045bdf2 eth1: Data path switched from VF: nic1 info kernel: hv_netvsc 6045bdf2-e3b3-6045-bdf2-e3b36045bdf2 eth1: VF unregistering: nic1 info kernel: hv_netvsc 6045bdf2-e3b3-6045-bdf2-e3b36045bdf2 eth1: VF slot 2 removed info kernel: hv_netvsc 7c1e525d-cf8d-7c1e-525d-cf8d7c1e525d eth2: Data path switched from VF: nic2 info kernel: hv_netvsc 7c1e525d-cf8d-7c1e-525d-cf8d7c1e525d eth2: VF unregistering: nic2 info kernel: hv_netvsc 7c1e525d-cf8d-7c1e-525d-cf8d7c1e525d eth2: VF slot 3 removed info kernel: hv_netvsc 7ced8d13-39cf-7ced-8d13-39cf7ced8d13 eth3: Data path switched from VF: nic3 info kernel: hv_netvsc 7ced8d13-39cf-7ced-8d13-39cf7ced8d13 eth3: VF unregistering: nic3 ... info kernel: pci_hyperv: PCI dom# 0x8219 has collision, using 0x1 err kernel: hv_vmbus: probe failed for device 4d265dd7-8219-4002-86c3-5f88aa46fabc (-19) info kernel: pci_hyperv: PCI dom# 0xee32 has collision, using 0x1 ... err kernel: INFO: task kworker/u64:3:27429 blocked for more than 120 seconds. err kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. info kernel: kworker/u64:3 D ffff880abaa5e9a0 0 27429 2 0x00000080 info kernel: Workqueue: hv_pci_2 hv_eject_device_work [pci_hyperv] warning kernel: Call Trace: warning kernel: [<ffffffff816ac1c8>] ? klist_iter_exit+0x18/0x30 warning kernel: [<ffffffff81453c21>] ? bus_find_device+0x91/0xd0 warning kernel: [<ffffffff816c46d9>] schedule_preempt_disabled+0x29/0x70 warning kernel: [<ffffffff816c24d7>] __mutex_lock_slowpath+0xc7/0x1d0 warning kernel: [<ffffffff816c18bf>] mutex_lock+0x1f/0x2f ... err kernel: INFO: task kworker/22:3:28345 blocked for more than 120 seconds. err kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. info kernel: kworker/22:3 D ffff882046e9df10 0 28345 2 0x00000080 info kernel: Workqueue: events vmbus_onmessage_work [hv_vmbus] warning kernel: Call Trace: warning kernel: [<ffffffff816c46d9>] schedule_preempt_disabled+0x29/0x70 warning kernel: [<ffffffff816c24d7>] __mutex_lock_slowpath+0xc7/0x1d0 warning kernel: [<ffffffff816c18bf>] mutex_lock+0x1f/0x2f ... err kernel: INFO: task kworker/u64:2:26055 blocked for more than 120 seconds. err kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. info kernel: kworker/u64:2 D ffff880d73f71520 0 26055 2 0x00000080 info kernel: Workqueue: mlx5_health0004:00:02.0 health_recover_work [mlx5_core] warning kernel: Call Trace: warning kernel: [<ffffffff816c46d9>] schedule_preempt_disabled+0x29/0x70 warning kernel: [<ffffffff816c24d7>] __mutex_lock_slowpath+0xc7/0x1d0 warning kernel: [<ffffffff816c18bf>] mutex_lock+0x1f/0x2f ...

Conditions

- Big-IP VE running in Azure (Hyper-V hypervisor) - SOCK driver in use - Azure rescinds one or more Virtual Functions on the hypervisor (host maintenance, live migration, etc...)

Workaround

Replace the SOCK driver with the xnet/DPDK driver on all the interfaces in use: https://my.f5.com/manage/s/article/K33245002

Fix Information

None

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips