Node Maintenance
----------------
-It is sometimes necessary to shutdown or reboot a node to do maintenance tasks,
-such as to replace hardware, or simply to install a new kernel image. This is
-also true when using the HA stack. The behaviour of the HA stack during a
-shutdown can be configured.
+Sometimes it is necessary to perform maintenance on a node, such as replacing
+hardware or simply installing a new kernel image. This also applies while the
+HA stack is in use.
+
+The HA stack can support you mainly in two types of maintenance:
+
+* for general shutdowns or reboots, the behavior can be configured, see
+ xref:ha_manager_shutdown_policy[Shutdown Policy].
+* for maintenance that does not require a shutdown or reboot, or that should
+ not be switched off automatically after only one reboot, you can enable the
+ manual maintenance mode.
+
+
+Maintenance Mode
+~~~~~~~~~~~~~~~~
+
+Enabling the manual maintenance mode will mark the node as unavailable for
+operation, this in turn will migrate away all services to other nodes, which
+are selected through the configured cluster resource scheduler (CRS) mode.
+During migration the original node will be recorded, so that the service can be
+moved back to to that node as soon as the maintenance mode is disabled, and it
+becomes online again.
+
+Currently you can enabled or disable the maintenance mode using the ha-manager
+CLI tool.
+
+.Enabling maintenance mode for a node
+----
+# ha-manager crm-command node-maintenance enable NODENAME
+----
+
+This will queue a CRM command, when the manager processes this command it will
+record the request for maintenance-mode in the manager status. This allows you
+to submit the command on any node, not just on the one you want to place in, or
+out of the maintenance mode.
+
+Once the LRM on the respective node picks the command up it will mark itself as
+unavailable, but still process all migration commands. This means that the LRM
+self-fencing watchdog will stay active until all active services got moved, and
+all running workers finished.
+
+Note that the LRM status will read `maintenance` mode as soon as the LRM
+picked the requested state up, not only when all services got moved away, this
+user experience is planned to be improved in the future.
+For now, you can check for any active HA service left on the node, or watching
+out for a log line like: `pve-ha-lrm[PID]: watchdog closed (disabled)` to know
+when the node finished its transition into the maintenance mode.
+
+NOTE: The manual maintenance mode is not automatically deleted on node reboot,
+but only if it is either manually deactivated using the `ha-manager` CLI or if
+the manager-status is manually cleared.
+
+.Disabling maintenance mode for a node
+----
+# ha-manager crm-command node-maintenance disable NODENAME
+----
+
+The process of disabling the manual maintenance mode is similar to enabling it.
+Using the `ha-manager` CLI command shown above will queue a CRM command that,
+once processed, marks the respective LRM node as available again.
+
+If you deactivate the maintenance mode, all services that were on the node when
+the maintenance mode was activated will be moved back.
[[ha_manager_shutdown_policy]]
Shutdown Policy
immediate node reboot or even reset.
+[[ha_manager_crs]]
Cluster Resource Scheduling
---------------------------
-The scheduler mode controls how HA selects nodes for the recovery of a service
-as well as for migrations that are triggered by a shutdown policy. The default
-mode is `basic`, you can change it in `datacenter.cfg`:
+The cluster resource scheduler (CRS) mode controls how HA selects nodes for the
+recovery of a service as well as for migrations that are triggered by a
+shutdown policy. The default mode is `basic`, you can change it in the Web UI
+(`Datacenter` -> `Options`), or directly in `datacenter.cfg`:
----
crs: ha=static
----
-The change will be in effect when a new master takes over. This can be triggered
-by executing the following on the current master's node:
+[thumbnail="screenshot/gui-datacenter-options-crs.png"]
-----
-systemctl reload-or-restart pve-ha-crm.service
-----
+The change will be in effect starting with the next manager round (after a few
+seconds).
For each service that needs to be recovered or migrated, the scheduler
iteratively chooses the best node among the nodes with the highest priority in
NOTE: There are plans to add modes for (static and dynamic) load-balancing in
the future.
-Basic
-^^^^^
+Basic Scheduler
+~~~~~~~~~~~~~~~
-The number of active HA serivces on each node is used to choose a recovery node.
+The number of active HA services on each node is used to choose a recovery node.
+Non-HA-managed services are currently not counted.
-Static
-^^^^^^
+Static-Load Scheduler
+~~~~~~~~~~~~~~~~~~~~~
IMPORTANT: The static mode is still a technology preview.
-Static usage information from HA serivces on each node is used to choose a
-recovery node.
+Static usage information from HA services on each node is used to choose a
+recovery node. Usage of non-HA-managed services is currently not considered.
For this selection, each node in turn is considered as if the service was
already running on it, using CPU and memory usage from the associated guest
services.
+CRS Scheduling Points
+~~~~~~~~~~~~~~~~~~~~~
+
+The CRS algorithm is not applied for every service in every round, since this
+would mean a large number of constant migrations. Depending on the workload,
+this could put more strain on the cluster than could be avoided by constant
+balancing.
+That's why the {pve} HA manager favors keeping services on their current node.
+
+The CRS is currently used at the following scheduling points:
+
+- Service recovery (always active). When a node with active HA services fails,
+ all its services need to be recovered to other nodes. The CRS algorithm will
+ be used here to balance that recovery over the remaining nodes.
+
+- HA group config changes (always active). If a node is removed from a group,
+ or its priority is reduced, the HA stack will use the CRS algorithm to find a
+ new target node for the HA services in that group, matching the adapted
+ priority constraints.
+
+- HA service stopped -> start transtion (opt-in). Requesting that a stopped
+ service should be started is an good opportunity to check for the best suited
+ node as per the CRS algorithm, as moving stopped services is cheaper to do
+ than moving them started, especially if their disk volumes reside on shared
+ storage. You can enable this by setting the **`ha-rebalance-on-start`**
+ CRS option in the datacenter config. You can change that option also in the
+ Web UI, under `Datacenter` -> `Options` -> `Cluster Resource Scheduling`.
+
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff