Bug ID 1010301: Long-Running iCall script commands can result in iCall script failures or ceasing to run

Last Modified: Dec 13, 2023

Affected Product(s):
BIG-IP TMOS(all modules)

Known Affected Versions:
15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 15.1.9, 15.1.9.1, 15.1.10, 15.1.10.2, 15.1.10.3

Opened: Apr 09, 2021

Severity: 3-Major

Symptoms

When an iCall script runs for at least 5 minutes (or the value of "tmsh list sys scriptd max-script-run-time", default 300), the Scriptd service attempts to terminate the script. However, iCall commands that result in external commands such as "tmsh::save sys ucs" (as used in the f5.automated_backup template) can block the termination signal until the command exits, and then block the parent Scriptd service. If this condition remains for 65 more seconds (for a total single iCall script time of at least 365 seconds), the BIG-IP system restarts the Scriptd service. If the already-running iCall script is running after Scriptd finishes restarting, there is an additional risk that the Scriptd service may be un-marked for high availability monitoring in the BIG-IP system. See the results of "tmsh list sys daemon-ha scriptd heartbeat" to understand the case. As a result, the next time a long-running iCall command blocks the Scriptd service may cause Scriptd to hang again, potentially preventing all further iCall script runs without manual intervention.

Impact

The iCall scripts repeatedly fail to finish or cease to run altogether.

Conditions

- An iCall script that takes at least 6 minutes 5 seconds to run, with individual command(s) that take at least 65 seconds to run. - For example, the f5.automated_backup template, when a UCS backups takes at least 6 minutes 5 seconds to finish on your BIG-IP system.

Workaround

Re-enable Scriptd HA daemon heartbeat check with the following command: tmsh modify sys daemon-ha scriptd heartbeat enabled If you believe your iCall scripts need more time to run normally, you can increase the maximum run time (with an example of 10 minutes) with the following command: tmsh modify sys scriptd max-script-run-time 600

Fix Information

None

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips