lrm: fix getting stuck on restart

run_workers is responsible for updating the state after workers have
exited. if the current LRM state is 'active', but a shutdown_request was
issued in 'restart' mode (like on package upgrades), this call is the
only one made in the LRM work() loop.

skipping it if there are active services means the following sequence of
events effectively keeps the LRM from restarting or making any progress:

- start HA migration on node A
- reload LRM on node A while migration is still running

even once the migration is finished, the service count is still >= 1
since the LRM never calls run_workers (directly or via
manage_resources), so the service having been migrated is never noticed.

maintenance mode (i.e., rebooting the node with shutdown policy migrate)
does call manage_resources and thus run_workers, and will proceed once
the last worker has exited.

reported by a user:

https://forum.proxmox.com/threads/lrm-hangs-when-updating-while-migration-is-running.108628

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index 7e635e67a17d2ca9e6303c91c467e260a27372aa..8cbdb8266aa7fbeafe9e3efe15b0039024514664 100644 (file)
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -420,18 +420,17 @@ sub work {
            if ($self->{shutdown_request}) {
 
                if ($self->{mode} eq 'restart') {
-
+                   # catch exited workers to update service state
+                   my $workers = $self->run_workers();
                    my $service_count = $self->active_service_count();
 
-                   if ($service_count == 0) {
-                       if ($self->run_workers() == 0) {
-                           # safety: no active services or workers -> OK
-                           give_up_watchdog_protection($self);
-                           $shutdown = 1;
+                   if ($service_count == 0 && $workers == 0) {
+                       # safety: no active services or workers -> OK
+                       give_up_watchdog_protection($self);
+                       $shutdown = 1;
 
-                           # restart with no or freezed services, release the lock
-                           $haenv->release_ha_agent_lock();
-                       }
+                       # restart with no or freezed services, release the lock
+                       $haenv->release_ha_agent_lock();
                    }
                } else {