Bug ID 950201: Tmm core on GCP

Last Modified: Nov 06, 2023

Affected Product(s):
BIG-IP MA-VE(all modules)

Known Affected Versions:
14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 14.1.5.3, 14.1.5.4, 14.1.5.6, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 16.0.0, 16.0.0.1, 16.0.1, 16.0.1.1, 16.0.1.2, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 17.0.0, 17.0.0.1, 17.0.0.2, 17.1.0, 17.1.0.1, 17.1.0.2, 17.1.0.3

Fixed In:
17.1.1, 16.1.4, 15.1.9

Opened: Oct 01, 2020

Severity: 2-Critical

Symptoms

When BIG-IP Virtual Edition (VE) is running on Google Cloud Platform (GCP) with mergeable buffers enabled, tmm might core while passing traffic. Subsequently, the kernel locks up, which prevents the whole system from recovering. TMM panic with this message in a tmm log file: panic: ../dev/ndal/virtio/if_virtio.c:2038: Assertion "Valid num_buffers" failed.

Impact

Traffic disrupted while tmm restarts.

Conditions

-- VE running on GCP. -- Mergeable buffers (mrg_rxbuf) is enabled on the guest with direct descriptors.

Workaround

You can use either of the following workarounds: -- Use the sock driver. For more information see K10142141: Configuring the BIG-IP VE system to use the SOCK network driver :: https://support.f5.com/csp/article/K10142141 -- Request an Engineering Hotfix from F5, with mrg_rxbuf and lro turned off. Note: Using either workaround has a performance impact.

Fix Information

- Added error handling to prevent crashing when a bad packet gets received - Added a new column 'invalid_header' into tmm/virtio_rx_stats table to track incidents

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips