fix: shutdown: if lxc-stop fails, wait for socket closing with timeout

When trying to shutdown a hung container with `forceStop=0` (e.g. via
the Web UI), the shutdown task may run indefinitely while holding a lock
on the container config. The reason is that the shutdown subroutine
waits for the LXC command socket to close, even if the `lxc-stop`
command has failed due to timeout. This prevents other tasks (such as a
stop task) from acquiring the lock. In order to stop the container, the
shutdown task has to be explicitly killed first, which is inconvenient.
This occurs e.g. when trying to shutdown a hung CentOS 7 container (with
systemd <v232) in a cgroupv2 environment.

This fix imposes a timeout on the socket polling operation if the
`lxc-stop` command has failed. Behavior in case `lxc-stop` succeeds is
unchanged. This reintroduces some behavior from b1bad293. The timeout
duration is the given shutdown timeout, meaning that the final task
duration in the scenario above is twice the shutdown timeout.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>

diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index ce6d5a532aad98cd537c8b2d3cce830f9651b34d..7cf1dcf1e2f7d66c766b2608027cbbc85b5655ff 100644 (file)
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -13,6 +13,7 @@ use Cwd qw();
 use Fcntl qw(O_RDONLY O_WRONLY O_NOFOLLOW O_DIRECTORY);
 use Errno qw(ELOOP ENOTDIR EROFS ECONNREFUSED ENOSYS EEXIST);
 use IO::Socket::UNIX;
+use IO::Poll qw(POLLIN POLLHUP);
 
 use PVE::Exception qw(raise_perm_exc);
 use PVE::Storage;
@@ -2473,13 +2474,22 @@ sub vm_stop {
     }
 
     eval { run_command($cmd, timeout => $shutdown_timeout) };
+
+    # Wait until the command socket is closed.
+    # In case the lxc-stop call failed, reading from the command socket may block forever,
+    # so poll with another timeout to avoid freezing the shutdown task.
     if (my $err = $@) {
-       warn $@ if $@;
-    }
+       warn $err if $err;
 
-    my $result = <$sock>;
+       my $poll = IO::Poll->new();
+       $poll->mask($sock => POLLIN | POLLHUP); # watch for input and EOF events
+       $poll->poll($shutdown_timeout); # IO::Poll timeout is in seconds
+       return if ($poll->events($sock) & POLLHUP);
+    } else {
+       my $result = <$sock>;
+       return if !defined $result; # monitor is gone and the ct has stopped.
+    }
 
-    return if !defined $result; # monitor is gone and the ct has stopped.
     die "container did not stop\n";
 }