lrm: keep manual maintenance mode independent of shutdown policy

We did not handle being in maintenance mode explicitly with shutdown
policies, which is in practice not often an issue as most that use
the maintenance mode also switched over the shutdown policy to
'migrate', which keeps the maintenance mode, but for all those
evaluating HA or only using the manual maintenance mode it meant that
on shutdown the mode was set to 'restart' or 'shutdown', which made
the active manager think that the node got out of the maintenance
state again, and marked it as online – but as it wasn't really online
(and on the way to shutdown), this not only cleared the maintenance
mode by mistake, it also had a chance to cause fencing - if any
service was still on the node – i.e., maintenance mode wasn't reached
yet, but still in-progress of moving HA services (guests).

Fix that by checking if maintenance mode is requested, or already
active (we currently don't differ those two explicitly, but could be
determined from active service count if required), and avoid changing
the mode in the shutdown and restart case. Log that also explicitly
so admins can understand what happened and why.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>

diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index 604e0573ff59276605617231791832774ac7f30d..a283070b0ae42e7689efafd0d1e41228ba09130f 100644 (file)
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -104,7 +104,19 @@ sub shutdown_request {
 
     if ($shutdown) {
        my $shutdown_type = $reboot ? 'reboot' : 'shutdown';
-       if ($maintenance) {
+       if ($self->is_maintenance_requested()) {
+           if ($maintenance) {
+               $haenv->log(
+                   'info',
+                   "$shutdown_type LRM, ignore maintenance policy, already in maintenance mode",
+               );
+           } else {
+               $haenv->log(
+                   'info',
+                   "$shutdown_type LRM, ignore $shutdown_policy policy as manual maintenance mode is enabled",
+               );
+           }
+       } elsif ($maintenance) {
            $haenv->log('info', "$shutdown_type LRM, doing maintenance, removing this node from active list");
            $self->{mode} = 'maintenance';
        } elsif ($freeze_all) {
@@ -114,6 +126,9 @@ sub shutdown_request {
            $haenv->log('info', "shutdown LRM, stop all services");
            $self->{mode} = 'shutdown';
        }
+    } elsif ($self->is_maintenance_requested()) {
+       $haenv->log('
+           info', "Restarting LRM in maintenance mode may be delayed until all services are moved");
     } else {
        $haenv->log('info', "restart LRM, freeze all services");
        $self->{mode} = 'restart';

diff --git a/src/test/test-crm-maintenance-request3/log.expect b/src/test/test-crm-maintenance-request3/log.expect
index 5e023e9206720bfb4b12b9fddcc3b765c40ea658..0291a69a653df270804bb3d3d4b58e2fa5d5d92d 100644 (file)
--- a/src/test/test-crm-maintenance-request3/log.expect
+++ b/src/test/test-crm-maintenance-request3/log.expect
@@ -43,7 +43,7 @@ info    163    node2/lrm: starting service vm:101
 info    163    node2/lrm: service status vm:101 started
 info    220      cmdlist: execute shutdown node1
 info    220    node1/lrm: got shutdown request with shutdown policy 'migrate'
-info    220    node1/lrm: shutdown LRM, doing maintenance, removing this node from active list
+info    220    node1/lrm: shutdown LRM, ignore maintenance policy, already in maintenance mode
 info    222    node1/lrm: exit (loop end)
 info    222     shutdown: execute crm node1 stop
 info    221    node1/crm: server received shutdown request

diff --git a/src/test/test-crm-maintenance-request4/log.expect b/src/test/test-crm-maintenance-request4/log.expect
index 1fd2aff953f71485fe43ca953064b3fd88fdc3ec..a713ffb3b261ba2ad09941c2841aa8f3cee6c461 100644 (file)
--- a/src/test/test-crm-maintenance-request4/log.expect
+++ b/src/test/test-crm-maintenance-request4/log.expect
@@ -43,41 +43,17 @@ info    163    node2/lrm: starting service vm:101
 info    163    node2/lrm: service status vm:101 started
 info    220      cmdlist: execute shutdown node1
 info    220    node1/lrm: got shutdown request with shutdown policy 'conditional'
-info    220    node1/lrm: shutdown LRM, stop all services
-info    220    node1/crm: node 'node1': state changed from 'maintenance' => 'online'
-info    220    node1/crm: moving service 'vm:101' back to 'node1', node came back from maintenance.
-info    220    node1/crm: migrate service 'vm:101' to node 'node1' (running)
-info    220    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node2, target = node1)
-info    221    node1/lrm: got lock 'ha_agent_node1_lock'
-info    221    node1/lrm: status change maintenance => active
+info    220    node1/lrm: shutdown LRM, ignore conditional policy as manual maintenance mode is enabled
 info    222    node1/lrm: exit (loop end)
 info    222     shutdown: execute crm node1 stop
 info    221    node1/crm: server received shutdown request
-info    223    node2/lrm: service vm:101 - start migrate to node 'node1'
-info    223    node2/lrm: service vm:101 - end migrate to node 'node1'
 info    240    node1/crm: voluntary release CRM lock
 info    241    node1/crm: exit (loop end)
 info    241     shutdown: execute power node1 off
 info    241    node2/crm: got lock 'ha_manager_lock'
 info    241    node2/crm: status change slave => master
-info    241    node2/crm: node 'node1': state changed from 'online' => 'unknown'
-info    241    node2/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node1)
 info    320      cmdlist: execute power node1 on
 info    320    node1/crm: status change startup => wait_for_quorum
 info    320    node1/lrm: status change startup => wait_for_agent_lock
 info    320    node1/crm: status change wait_for_quorum => slave
-info    321    node1/lrm: got lock 'ha_agent_node1_lock'
-info    321    node1/lrm: status change wait_for_agent_lock => active
-info    321    node1/lrm: starting service vm:101
-info    321    node1/lrm: service status vm:101 started
-info    322    node2/crm: node 'node1': state changed from 'unknown' => 'online'
-info    341    node1/lrm: status change active => maintenance
-info    342    node2/crm: node 'node1': state changed from 'online' => 'maintenance'
-info    342    node2/crm: migrate service 'vm:101' to node 'node2' (running)
-info    342    node2/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node2)
-info    361    node1/lrm: service vm:101 - start migrate to node 'node2'
-info    361    node1/lrm: service vm:101 - end migrate to node 'node2'
-info    362    node2/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node2)
-info    363    node2/lrm: starting service vm:101
-info    363    node2/lrm: service status vm:101 started
 info    920     hardware: exit simulation - done