Service States
~~~~~~~~~~~~~~
+[thumbnail="gui-ha-manager-status.png"]
+
The CRM use a service state enumeration to record the current service
state. We display this state on the GUI and you can query it using
the `ha-manager` command line tool:
Resources
~~~~~~~~~
+[thumbnail="gui-ha-manager-resources-view.png"]
+
The resource configuration file `/etc/pve/ha/resources.cfg` stores
the list of resources managed by `ha-manager`. A resource configuration
inside that list look like this:
# Note: use default settings for everything
----
+[thumbnail="gui-ha-manager-add-resource.png"]
+
Above config was generated using the `ha-manager` command line tool:
----
Groups
~~~~~~
+[thumbnail="gui-ha-manager-groups-view.png"]
+
The HA group configuration file `/etc/pve/ha/groups.cfg` is used to
define groups of cluster nodes. A resource can be restricted to run
only on the members of such group. A group configuration look like
include::ha-groups-opts.adoc[]
+[thumbnail="gui-ha-manager-add-group.png"]
+
A commom requirement is that a resource should run on a specific
node. Usually the resource is able to run on other nodes, so you can define
an unrestricted group with a single member:
recovered services move straight back to the fenced node.
-Node Power Status
------------------
+[[ha_manager_fencing]]
+Fencing
+-------
-If a node needs maintenance you should migrate and or relocate all
-services which are required to run always on another node first.
-After that you can stop the LRM and CRM services. But note that the
-watchdog triggers if you stop it with active services.
+On node failures, fencing ensures that the erroneous node is
+guaranteed to be offline. This is required to make sure that no
+resource runs twice when it gets recovered on another node. This is a
+really important task, because without, it would not be possible to
+recover a resource on another node.
+If a node would not get fenced, it would be in an unknown state where
+it may have still access to shared resources. This is really
+dangerous! Imagine that every network but the storage one broke. Now,
+while not reachable from the public network, the VM still runs and
+writes to the shared storage.
-[[ha_manager_package_updates]]
-Package Updates
----------------
+If we then simply start up this VM on another node, we would get a
+dangerous race conditions because we write from both nodes. Such
+condition can destroy all VM data and the whole VM could be rendered
+unusable. The recovery could also fail if the storage protects from
+multiple mounts.
-When updating the ha-manager you should do one node after the other, never
-all at once for various reasons. First, while we test our software
-thoughtfully, a bug affecting your specific setup cannot totally be ruled out.
-Upgrading one node after the other and checking the functionality of each node
-after finishing the update helps to recover from an eventual problems, while
-updating all could render you in a broken cluster state and is generally not
-good practice.
-Also, the {pve} HA stack uses a request acknowledge protocol to perform
-actions between the cluster and the local resource manager. For restarting,
-the LRM makes a request to the CRM to freeze all its services. This prevents
-that they get touched by the Cluster during the short time the LRM is restarting.
-After that the LRM may safely close the watchdog during a restart.
-Such a restart happens on a update and as already stated a active master
-CRM is needed to acknowledge the requests from the LRM, if this is not the case
-the update process can be too long which, in the worst case, may result in
-a watchdog reset.
+How {pve} Fences
+~~~~~~~~~~~~~~~~
+There are different methods to fence a node, for example, fence
+devices which cut off the power from the node or disable their
+communication completely. Those are often quite expensive and bring
+additional critical components into a system, because if they fail you
+cannot recover any service.
-[[ha_manager_fencing]]
-Fencing
--------
+We thus wanted to integrate a simpler fencing method, which does not
+require additional external hardware. This can be done using
+watchdog timers.
-What is Fencing
-~~~~~~~~~~~~~~~
+.Possible Fencing Methods
+- external power switches
+- isolate nodes by disabling complete network traffic on the switch
+- self fencing using watchdog timers
-Fencing secures that on a node failure the dangerous node gets will be rendered
-unable to do any damage and that no resource runs twice when it gets recovered
-from the failed node. This is a really important task and one of the base
-principles to make a system Highly Available.
+Watchdog timers are widely used in critical and dependable systems
+since the beginning of micro controllers. They are often independent
+and simple integrated circuits which are used to detect and recover
+from computer malfunctions.
-If a node would not get fenced it would be in an unknown state where it may
-have still access to shared resources, this is really dangerous!
-Imagine that every network but the storage one broke, now while not
-reachable from the public network the VM still runs and writes on the shared
-storage. If we would not fence the node and just start up this VM on another
-Node we would get dangerous race conditions, atomicity violations the whole VM
-could be rendered unusable. The recovery could also simply fail if the storage
-protects from multiple mounts and thus defeat the purpose of HA.
+During normal operation, `ha-manager` regularly resets the watchdog
+timer to prevent it from elapsing. If, due to a hardware fault or
+program error, the computer fails to reset the watchdog, the timer
+will elapse and triggers a reset of the whole server (reboot).
-How {pve} Fences
-~~~~~~~~~~~~~~~~~
+Recent server motherboards often include such hardware watchdogs, but
+these need to be configured. If no watchdog is available or
+configured, we fall back to the Linux Kernel 'softdog'. While still
+reliable, it is not independent of the servers hardware, and thus has
+a lower reliability than a hardware watchdog.
-There are different methods to fence a node, for example fence devices which
-cut off the power from the node or disable their communication completely.
-Those are often quite expensive and bring additional critical components in
-a system, because if they fail you cannot recover any service.
+Configure Hardware Watchdog
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
-We thus wanted to integrate a simpler method in the HA Manager first, namely
-self fencing with watchdogs.
+By default, all hardware watchdog modules are blocked for security
+reasons. They are like a loaded gun if not correctly initialized. To
+enable a hardware watchdog, you need to specify the module to load in
+'/etc/default/pve-ha-manager', for example:
-Watchdogs are widely used in critical and dependable systems since the
-beginning of micro controllers, they are often independent and simple
-integrated circuit which programs can use to watch them. After opening they need to
-report periodically. If, for whatever reason, a program becomes unable to do
-so the watchdogs triggers a reset of the whole server.
+----
+# select watchdog module (default is softdog)
+WATCHDOG_MODULE=iTCO_wdt
+----
-Server motherboards often already include such hardware watchdogs, these need
-to be configured. If no watchdog is available or configured we fall back to the
-Linux Kernel softdog while still reliable it is not independent of the servers
-Hardware and thus has a lower reliability then a hardware watchdog.
+This configuration is read by the 'watchdog-mux' service, which load
+the specified module at startup.
-Configure Hardware Watchdog
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-By default all watchdog modules are blocked for security reasons as they are
-like a loaded gun if not correctly initialized.
-If you have a hardware watchdog available remove its kernel module from the
-blacklist, load it with insmod and restart the `watchdog-mux` service or reboot
-the node.
Recover Fenced Services
~~~~~~~~~~~~~~~~~~~~~~~
-After a node failed and its fencing was successful we start to recover services
-to other available nodes and restart them there so that they can provide service
-again.
+After a node failed and its fencing was successful, the CRM tries to
+move services from the failed node to nodes which are still online.
-The selection of the node on which the services gets recovered is influenced
-by the users group settings, the currently active nodes and their respective
-active service count.
-First we build a set out of the intersection between user selected nodes and
-available nodes. Then the subset with the highest priority of those nodes
-gets chosen as possible nodes for recovery. We select the node with the
-currently lowest active service count as a new node for the service.
-That minimizes the possibility of an overload, which else could cause an
-unresponsive node and as a result a chain reaction of node failures in the
-cluster.
+The selection of nodes, on which those services gets recovered, is
+influenced by the resource `group` settings, the list of currently active
+nodes, and their respective active service count.
+
+The CRM first builds a set out of the intersection between user selected
+nodes (from `group` setting) and available nodes. It then choose the
+subset of nodes with the highest priority, and finally select the node
+with the lowest active service count. This minimizes the possibility
+of an overloaded node.
+
+CAUTION: On node failure, the CRM distributes services to the
+remaining nodes. This increase the service count on those nodes, and
+can lead to high load, especially on small clusters. Please design
+your cluster so that it can handle such worst case scenarios.
[[ha_manager_start_failure_policy]]
* *after* you fixed all errors you may enable the service again
+Node Power Status
+-----------------
+
+If a node needs maintenance you should migrate and or relocate all
+services which are required to run always on another node first.
+After that you can stop the LRM and CRM services. But note that the
+watchdog triggers if you stop it with active services.
+
+
+[[ha_manager_package_updates]]
+Package Updates
+---------------
+
+When updating the ha-manager you should do one node after the other, never
+all at once for various reasons. First, while we test our software
+thoughtfully, a bug affecting your specific setup cannot totally be ruled out.
+Upgrading one node after the other and checking the functionality of each node
+after finishing the update helps to recover from an eventual problems, while
+updating all could render you in a broken cluster state and is generally not
+good practice.
+
+Also, the {pve} HA stack uses a request acknowledge protocol to perform
+actions between the cluster and the local resource manager. For restarting,
+the LRM makes a request to the CRM to freeze all its services. This prevents
+that they get touched by the Cluster during the short time the LRM is restarting.
+After that the LRM may safely close the watchdog during a restart.
+Such a restart happens on a update and as already stated a active master
+CRM is needed to acknowledge the requests from the LRM, if this is not the case
+the update process can be too long which, in the worst case, may result in
+a watchdog reset.
+
+
[[ha_manager_service_operations]]
Service Operations
------------------
projects / pve-docs.git / blobdiff