* Eliminate single point of failure (redundant components)
** use an uninterruptible power supply (UPS)
-** use redundant power supplies on the main boards
+** use redundant power supplies in your servers
** use ECC-RAM
** use redundant network hardware
** use RAID for local storage
This section provides a short overview of common management tasks. The
first step is to enable HA for a resource. This is done by adding the
resource to the HA resource configuration. You can do this using the
-GUI, or simply use the command line tool, for example:
+GUI, or simply use the command-line tool, for example:
----
# ha-manager add vm:100
lock.
+[[ha_manager_service_states]]
Service States
~~~~~~~~~~~~~~
The CRM uses a service state enumeration to record the current service
state. This state is displayed on the GUI and can be queried using
-the `ha-manager` command line tool:
+the `ha-manager` command-line tool:
----
# ha-manager status
Service is stopped and marked as `disabled`
+[[ha_manager_lrm]]
Local Resource Manager
~~~~~~~~~~~~~~~~~~~~~~
`journalctl -u pve-ha-lrm` on the node(s) where the service is and
the same command for the pve-ha-crm on the node which is the current master.
+
+[[ha_manager_crm]]
Cluster Resource Manager
~~~~~~~~~~~~~~~~~~~~~~~~
-------------
The HA stack is well integrated into the {pve} API. So, for example,
-HA can be configured via the `ha-manager` command line interface, or
+HA can be configured via the `ha-manager` command-line interface, or
the {pve} web interface - both interfaces provide an easy way to
manage HA. Automation tools can use the API directly.
[thumbnail="screenshot/gui-ha-manager-add-resource.png"]
-The above config was generated using the `ha-manager` command line tool:
+The above config was generated using the `ha-manager` command-line tool:
----
# ha-manager add vm:501 --state started --max_relocate 2
case, may result in a reset triggered by the watchdog.
+[[ha_manager_node_maintenance]]
Node Maintenance
----------------
-It is sometimes necessary to shutdown or reboot a node to do maintenance tasks,
-such as to replace hardware, or simply to install a new kernel image. This is
-also true when using the HA stack. The behaviour of the HA stack during a
-shutdown can be configured.
+Sometimes it is necessary to perform maintenance on a node, such as replacing
+hardware or simply installing a new kernel image. This also applies while the
+HA stack is in use.
+
+The HA stack can support you mainly in two types of maintenance:
+
+* for general shutdowns or reboots, the behavior can be configured, see
+ xref:ha_manager_shutdown_policy[Shutdown Policy].
+* for maintenance that does not require a shutdown or reboot, or that should
+ not be switched off automatically after only one reboot, you can enable the
+ manual maintenance mode.
+
+
+Maintenance Mode
+~~~~~~~~~~~~~~~~
+
+You can use the manual maintenance mode to mark the node as unavailable for HA
+operation, prompting all services managed by HA to migrate to other nodes.
+
+The target nodes for these migrations are selected from the other currently
+available nodes, and determined by the HA group configuration and the configured
+cluster resource scheduler (CRS) mode.
+During each migration, the original node will be recorded in the HA managers'
+state, so that the service can be moved back again automatically once the
+maintenance mode is disabled and the node is back online.
+
+Currently you can enabled or disable the maintenance mode using the ha-manager
+CLI tool.
+
+.Enabling maintenance mode for a node
+----
+# ha-manager crm-command node-maintenance enable NODENAME
+----
+
+This will queue a CRM command, when the manager processes this command it will
+record the request for maintenance-mode in the manager status. This allows you
+to submit the command on any node, not just on the one you want to place in, or
+out of the maintenance mode.
+
+Once the LRM on the respective node picks the command up it will mark itself as
+unavailable, but still process all migration commands. This means that the LRM
+self-fencing watchdog will stay active until all active services got moved, and
+all running workers finished.
+
+Note that the LRM status will read `maintenance` mode as soon as the LRM
+picked the requested state up, not only when all services got moved away, this
+user experience is planned to be improved in the future.
+For now, you can check for any active HA service left on the node, or watching
+out for a log line like: `pve-ha-lrm[PID]: watchdog closed (disabled)` to know
+when the node finished its transition into the maintenance mode.
+
+NOTE: The manual maintenance mode is not automatically deleted on node reboot,
+but only if it is either manually deactivated using the `ha-manager` CLI or if
+the manager-status is manually cleared.
+
+.Disabling maintenance mode for a node
+----
+# ha-manager crm-command node-maintenance disable NODENAME
+----
+
+The process of disabling the manual maintenance mode is similar to enabling it.
+Using the `ha-manager` CLI command shown above will queue a CRM command that,
+once processed, marks the respective LRM node as available again.
+
+If you deactivate the maintenance mode, all services that were on the node when
+the maintenance mode was activated will be moved back.
[[ha_manager_shutdown_policy]]
Shutdown Policy
shutdown. Currently 'Conditional' is the default due to backward compatibility.
Some users may find that 'Migrate' behaves more as expected.
+The shutdown policy can be configured in the Web UI (`Datacenter` -> `Options`
+-> `HA Settings`), or directly in `datacenter.cfg`:
+
+----
+ha: shutdown_policy=<value>
+----
+
Migrate
^^^^^^^
immediate node reboot or even reset.
-Scheduler Mode
---------------
+[[ha_manager_crs]]
+Cluster Resource Scheduling
+---------------------------
-The scheduler mode controls how HA selects nodes for the recovery of a service
-as well as for migrations that are triggered by a shutdown policy. The default
-mode is `basic`, you can change it in `datacenter.cfg`:
+The cluster resource scheduler (CRS) mode controls how HA selects nodes for the
+recovery of a service as well as for migrations that are triggered by a
+shutdown policy. The default mode is `basic`, you can change it in the Web UI
+(`Datacenter` -> `Options`), or directly in `datacenter.cfg`:
----
crs: ha=static
----
-The change will be in effect when a new master takes over. This can be triggered
-by executing the following on the current master's node:
+[thumbnail="screenshot/gui-datacenter-options-crs.png"]
-----
-systemctl reload-or-restart pve-ha-crm.service
-----
+The change will be in effect starting with the next manager round (after a few
+seconds).
For each service that needs to be recovered or migrated, the scheduler
iteratively chooses the best node among the nodes with the highest priority in
NOTE: There are plans to add modes for (static and dynamic) load-balancing in
the future.
-Basic
-^^^^^
+Basic Scheduler
+~~~~~~~~~~~~~~~
-The number of active HA serivces on each node is used to choose a recovery node.
+The number of active HA services on each node is used to choose a recovery node.
+Non-HA-managed services are currently not counted.
-Static
-^^^^^^
+Static-Load Scheduler
+~~~~~~~~~~~~~~~~~~~~~
-Static usage information from HA serivces on each node is used to choose a
-recovery node.
+IMPORTANT: The static mode is still a technology preview.
+
+Static usage information from HA services on each node is used to choose a
+recovery node. Usage of non-HA-managed services is currently not considered.
For this selection, each node in turn is considered as if the service was
already running on it, using CPU and memory usage from the associated guest
(to still be able to distinguish in case there already is a more highly
committed node) are considered.
+IMPORTANT: The more services the more possible combinations there are, so it's
+currently not recommended to use it if you have thousands of HA managed
+services.
+
+
+CRS Scheduling Points
+~~~~~~~~~~~~~~~~~~~~~
+
+The CRS algorithm is not applied for every service in every round, since this
+would mean a large number of constant migrations. Depending on the workload,
+this could put more strain on the cluster than could be avoided by constant
+balancing.
+That's why the {pve} HA manager favors keeping services on their current node.
+
+The CRS is currently used at the following scheduling points:
+
+- Service recovery (always active). When a node with active HA services fails,
+ all its services need to be recovered to other nodes. The CRS algorithm will
+ be used here to balance that recovery over the remaining nodes.
+
+- HA group config changes (always active). If a node is removed from a group,
+ or its priority is reduced, the HA stack will use the CRS algorithm to find a
+ new target node for the HA services in that group, matching the adapted
+ priority constraints.
+
+- HA service stopped -> start transtion (opt-in). Requesting that a stopped
+ service should be started is an good opportunity to check for the best suited
+ node as per the CRS algorithm, as moving stopped services is cheaper to do
+ than moving them started, especially if their disk volumes reside on shared
+ storage. You can enable this by setting the **`ha-rebalance-on-start`**
+ CRS option in the datacenter config. You can change that option also in the
+ Web UI, under `Datacenter` -> `Options` -> `Cluster Resource Scheduling`.
+
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff