Symptoms

A openshift cluster rebuild kept failing and retrying do to timeouts while running the ansible-playbooks to rebuild the cluster. This caused the cluster rebuild to fail for 8 plus hours, during which time not tenants could be started.

Impact

While the ansible-playbook runs were timing out, it was not possible to launch tenants on the chassis.

Conditions

An openshift cluster rebuild was issued after upgrade the system. Unable to reproduce this issue locally.

Workaround

The playbooks stopped timing out after 8 plus hours, no workaround is known.

Fix Information

1.) Enhanced code that generates and corrects the /etc/hosts file to make sure all the necessary entries are always present and correct. 2.) Enhanced the code the handles the SSH connection caching to make sure it always cleared during ansible-playbook runs, so it won't get affected by a stale connection 3.) Playbook timeouts will be increased after a timeout failure up to 3x to try and allow the system to complete it's work even if something is slowing down the playbook runs.

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips