Bug ID 1319265: Tmm crash observed in GCP after a migration

Last Modified: May 29, 2024

Affected Product(s):
BIG-IP LTM(all modules)

Known Affected Versions:
14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 14.1.5.3, 14.1.5.4, 14.1.5.6, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 15.1.9, 15.1.9.1, 15.1.10, 15.1.10.2, 15.1.10.3, 15.1.10.4, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 16.1.4, 16.1.4.1, 16.1.4.2, 16.1.4.3, 17.0.0, 17.0.0.1, 17.0.0.2, 17.1.0, 17.1.0.1, 17.1.0.2, 17.1.0.3, 17.1.1, 17.1.1.1, 17.1.1.2, 17.1.1.3

Opened: Jul 11, 2023

Severity: 3-Major

Symptoms

Tmm may crash in Google Cloud Platform (GCP) after a migration. The following logs were observed in kern.log emerg kernel: NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [finish:5055] warning kernel: [<ffffffffa01a21ff>] virtnet_send_command.constprop.34+0x10f/0x160 [virtio_net] warning kernel: [<ffffffffa01a274f>] virtnet_set_queues+0x9f/0x100 [virtio_net] warning kernel: [<ffffffffa01a38bd>] virtnet_probe+0x77d/0x858 [virtio_net] warning kernel: [<ffffffffa002792f>] virtio_dev_probe+0x1cf/0x2d0 [virtio] warning kernel: [<ffffffff81456165>] driver_probe_device+0xc5/0x460 warning kernel: [<ffffffff81456500>] ? driver_probe_device+0x460/0x460 warning kernel: [<ffffffff81456543>] __device_attach+0x43/0x50 warning kernel: [<ffffffff81453de5>] bus_for_each_drv+0x75/0xc0 warning kernel: [<ffffffff81455fa0>] device_attach+0x90/0xb0 warning kernel: [<ffffffff814551c8>] bus_probe_device+0x98/0xd0 warning kernel: [<ffffffff81452a6f>] device_add+0x4ff/0x7c0 warning kernel: [<ffffffffa0070370>] ? vp_finalize_features+0x40/0x40 [virtio_pci] warning kernel: [<ffffffffa0070370>] ? vp_finalize_features+0x40/0x40 [virtio_pci] warning kernel: [<ffffffff81452d4a>] device_register+0x1a/0x20 warning kernel: [<ffffffffa00273c9>] register_virtio_device+0xb9/0x100 [virtio] warning kernel: [<ffffffffa006f8b7>] virtio_pci_probe+0xb7/0x140 [virtio_pci] warning kernel: [<ffffffff8137856a>] local_pci_probe+0x4a/0xb0 warning kernel: [<ffffffff81379ca9>] pci_device_probe+0x109/0x160 warning kernel: [<ffffffff81456165>] driver_probe_device+0xc5/0x460 warning kernel: [<ffffffff81456500>] ? driver_probe_device+0x460/0x460 warning kernel: [<ffffffff81456543>] __device_attach+0x43/0x50 warning kernel: [<ffffffff81453de5>] bus_for_each_drv+0x75/0xc0 warning kernel: [<ffffffff81455fa0>] device_attach+0x90/0xb0 warning kernel: [<ffffffff81454139>] bus_rescan_devices_helper+0x39/0x60 warning kernel: [<ffffffff81454542>] store_drivers_probe+0x32/0x70 warning kernel: [<ffffffff81453a69>] bus_attr_store+0x29/0x30 warning kernel: [<ffffffff81290d22>] sysfs_kf_write+0x42/0x50 warning kernel: [<ffffffff812902f3>] kernfs_fop_write+0xe3/0x160 warning kernel: [<ffffffff81207bf0>] vfs_write+0xc0/0x1f0 warning kernel: [<ffffffff81208a1f>] SyS_write+0x7f/0xf0 warning kernel: [<ffffffff816cf741>] system_call_fastpath+0x48/0x4d

Impact

Traffic disrupted while tmm restarts.

Conditions

-- Google Cloud Platform -- Virtio driver -- A GCP migration is performed

Workaround

-- Use the sock driver. For more information see K10142141: Configuring the BIG-IP VE system to use the SOCK network driver :: https://support.f5.com/csp/article/K10142141 Related Bug IDs: 1319265, 1322937, 1326721

Fix Information

None

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips