Node Maintenance
----------------
-It is sometimes necessary to shutdown or reboot a node to do maintenance tasks,
-such as to replace hardware, or simply to install a new kernel image. This is
-also true when using the HA stack. The behaviour of the HA stack during a
-shutdown can be configured.
+Sometimes it is necessary to perform maintenance on a node, such as replacing
+hardware or simply installing a new kernel image. This also applies while the
+HA stack is in use.
+
+The HA stack can support you mainly in two types of maintenance:
+
+* for general shutdowns or reboots, the behavior can be configured, see
+ xref:ha_manager_shutdown_policy[Shutdown Policy].
+* for maintenance that does not require a shutdown or reboot, or that should
+ not be switched off automatically after only one reboot, you can enable the
+ manual maintenance mode.
+
+
+Maintenance Mode
+~~~~~~~~~~~~~~~~
+
+Enabling the manual maintenance mode will mark the node as unavailable for
+operation, this in turn will migrate away all services to other nodes, which
+are selected through the configured cluster resource scheduler (CRS) mode.
+During migration the original node will be recorded, so that the service can be
+moved back to to that node as soon as the maintenance mode is disabled, and it
+becomes online again.
+
+Currently you can enabled or disable the maintenance mode using the ha-manager
+CLI tool.
+
+.Enabling maintenance mode for a node
+----
+# ha-manager crm-command node-maintenance enable NODENAME
+----
+
+This will queue a CRM command, when the manager processes this command it will
+record the request for maintenance-mode in the manager status. This allows you
+to submit the command on any node, not just on the one you want to place in, or
+out of the maintenance mode.
+
+Once the LRM on the respective node picks the command up it will mark itself as
+unavailable, but still process all migration commands. This means that the LRM
+self-fencing watchdog will stay active until all active services got moved, and
+all running workers finished.
+
+Note that the LRM status will read `maintenance` mode as soon as the LRM
+picked the requested state up, not only when all services got moved away, this
+user experience is planned to be improved in the future.
+For now, you can check for any active HA service left on the node, or watching
+out for a log line like: `pve-ha-lrm[PID]: watchdog closed (disabled)` to know
+when the node finished its transition into the maintenance mode.
+
+NOTE: The manual maintenance mode is not automatically deleted on node reboot,
+but only if it is either manually deactivated using the `ha-manager` CLI or if
+the manager-status is manually cleared.
+
+.Disabling maintenance mode for a node
+----
+# ha-manager crm-command node-maintenance disable NODENAME
+----
+
+The process of disabling the manual maintenance mode is similar to enabling it.
+Using the `ha-manager` CLI command shown above will queue a CRM command that,
+once processed, marks the respective LRM node as available again.
+
+If you deactivate the maintenance mode, all services that were on the node when
+the maintenance mode was activated will be moved back.
[[ha_manager_shutdown_policy]]
Shutdown Policy
Cluster Resource Scheduling
---------------------------
-The scheduler mode controls how HA selects nodes for the recovery of a service
-as well as for migrations that are triggered by a shutdown policy. The default
-mode is `basic`, you can change it in `datacenter.cfg`:
+The cluster resource scheduler (CRS) mode controls how HA selects nodes for the
+recovery of a service as well as for migrations that are triggered by a
+shutdown policy. The default mode is `basic`, you can change it in the Web UI
+(`Datacenter` -> `Options`), or directly in `datacenter.cfg`:
----
crs: ha=static
----
+[thumbnail="screenshot/gui-datacenter-options-crs.png"]
+
The change will be in effect starting with the next manager round (after a few
seconds).
NOTE: There are plans to add modes for (static and dynamic) load-balancing in
the future.
-Basic
-~~~~~
+Basic Scheduler
+~~~~~~~~~~~~~~~
The number of active HA services on each node is used to choose a recovery node.
Non-HA-managed services are currently not counted.
-Static
-~~~~~~
+Static-Load Scheduler
+~~~~~~~~~~~~~~~~~~~~~
IMPORTANT: The static mode is still a technology preview.
services.
+CRS Scheduling Points
+~~~~~~~~~~~~~~~~~~~~~
+
+The CRS algorithm is not applied for every service in every round, since this
+would mean a large number of constant migrations. Depending on the workload,
+this could put more strain on the cluster than could be avoided by constant
+balancing.
+That's why the {pve} HA manager favors keeping services on their current node.
+
+The CRS is currently used at the following scheduling points:
+
+- Service recovery (always active). When a node with active HA services fails,
+ all its services need to be recovered to other nodes. The CRS algorithm will
+ be used here to balance that recovery over the remaining nodes.
+
+- HA group config changes (always active). If a node is removed from a group,
+ or its priority is reduced, the HA stack will use the CRS algorithm to find a
+ new target node for the HA services in that group, matching the adapted
+ priority constraints.
+
+- HA service stopped -> start transtion (opt-in). Requesting that a stopped
+ service should be started is an good opportunity to check for the best suited
+ node as per the CRS algorithm, as moving stopped services is cheaper to do
+ than moving them started, especially if their disk volumes reside on shared
+ storage. You can enable this by setting the **`ha-rebalance-on-start`**
+ CRS option in the datacenter config. You can change that option also in the
+ Web UI, under `Datacenter` -> `Options` -> `Cluster Resource Scheduling`.
+
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff