lrm: avoid job starvation on huge workloads

If a setup has a lot VMs we may run into the time limit from the
run_worker loop before processing all workers, which can easily
happen if an admin did not increased their default of max_workers in
the setup, but even with a bigger max_worker setting one can run into
it.

That combined with the fact that we sorted just by the $sid
alpha-numerically means that CTs where preferred over VMs (C comes
before V) and additionally lower VMIDs where preferred too.

That means that a set of SIDs had a lower chance of ever get actually
run, which is naturally not ideal at all.
Improve on that behavior by adding a counter to the queued worker and
preferring those that have a higher one, i.e., spent more time
waiting on getting actively run.

Note, due to the way the stop state is enforced, i.e., always
enqueued as new worker, its start-try counter will be reset every
round and thus have a lower priority compared to other request
states. We probably want to differ between a stop request when the
service is/was in another state just before and the time a stop is
just re-requested even if a service was already stopped for a while.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>

diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index b32c958bfc268de39970d7b56ddd56fe1ad3d8e1..f4c3e7671ea481b433c599b83b54c9c61d0b8858 100644 (file)
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -552,17 +552,24 @@ sub run_workers {
 
     # number of workers to start, if 0 we exec the command directly witouth forking
     my $max_workers = $haenv->get_max_workers();
-
     my $sc = $haenv->read_service_config();
 
+    my $worker = $self->{workers};
+    # we only got limited time but want to ensure that every queued worker is scheduled
+    # eventually, so sort by the count a worker was seen here in this loop
+    my $fair_sorter = sub {
+       $worker->{$b}->{start_tries} <=> $worker->{$a}->{start_tries} || $a cmp $b
+    };
+
     while (($haenv->get_time() - $starttime) < 5) {
        my $count = $self->check_active_workers();
 
-       foreach my $sid (sort keys %{$self->{workers}}) {
-           last if $count >= $max_workers && $max_workers > 0;
-
-           my $w = $self->{workers}->{$sid};
-           next if $w->{pid};
+       for my $sid (sort $fair_sorter grep { !$worker->{$_}->{pid} } keys %$worker) {
+           my $w = $worker->{$sid};
+           # higher try-count means higher priority especially compared to newly queued jobs, so
+           # count every try to avoid starvation
+           $w->{start_tries}++;
+           next if $count >= $max_workers && $max_workers > 0;
 
            # only fork if we may, else call exec_resource_agent directly (e.g. for tests)
            if ($max_workers > 0) {
@@ -666,6 +673,7 @@ sub queue_resource_command {
        sid => $sid,
        uid => $uid,
        state => $state,
+       start_tries => 0,
     };
 
     $self->{workers}->{$sid}->{params} = $params if $params;