Last Modified: Nov 14, 2022
Affected Product(s):
BIG-IQ Device
Known Affected Versions:
7.1.0, 7.1.0.1, 7.1.0.2, 7.1.0.3, 7.1.6, 7.1.6.1, 7.1.7, 7.1.7.1, 7.1.7.2, 7.1.8, 7.1.8.1, 7.1.8.2, 7.1.8.3, 7.1.8.4, 7.1.8.5, 7.1.9, 7.1.9.7, 7.1.9.8, 7.1.9.9
Opened: Feb 24, 2020 Severity: 4-Minor
When this scenario occurs, the BIG-IQ UI’s GeoIP Database Management -> Update History page will show the active update task appearing to stall with the status "In progress", and indicating that one or more devices has not yet finished (e.g., "Updated 3/4 devices."). The details page for the update task will show results from other BIG-IP devices included in the update task that have completed their updates. Additionally, any other BIG-IQ tasks that use remote commands will fail to run against the affected BIG-IP device, reporting an error such as "The limit on the number of concurrent shells is exceeded".
GeoIP Database Management will not be able to start new update tasks. Any new update tasks that are created will appear with the state "Queued", but will not start processing until this issue is resolved.
In some environments, this scenario may never occur. In other environments, it may be more likely to occur when updating a BIG-IP device that is under heavy load.
To solve the immediate issue: 1. Use SSH to log in to the affected BIG-IP device. 2. At the shell prompt, run: "ps ax | grep geoip" 3. Identify the process that is stuck, and note the number at the start of its line (its process ID). - If "/usr/bin/tmsh load sys geoip" still appears, it is likely the stuck process. - If not, check whether any other commands stand out from the list. - As a fallback: Any command with a long command-line that starts with "/bin/bash". (That said: If one of these is waiting on a child process, that child process must be dealt with first.) 4. Run "kill -9 #####", using the process ID above. When the situation has been resolved, the relevant update task on BIG-IQ will show that another BIG-IP has finished, and its task details page will have a new row in the table showing a failure for the affected BIG-IP device. If this was the last BIG-IP device that the update task was waiting upon, the update task will now appear as finished; if any additional update tasks are queued, the next update task will start. To avoid the problem in the future: The device administrator can temporarily take one or more BIG-IP devices offline, before starting a GeoIP Database Management update task that includes those BIG-IP devices.
None
