+Above commands created the following group configuration fils:
+
+.Configuration Example (`/etc/pve/ha/groups.cfg`)
+----
+group: prefer_node1
+ nodes node1
+
+group: mygroup1
+ nodes node2:1,node4,node1:2,node3:1
+
+group: mygroup2
+ nodes node2,node1
+ restricted 1
+----
+
+
+The `nofailback` options is mostly useful to avoid unwanted resource
+movements during administration tasks. For example, if you need to
+migrate a service to a node which hasn't the highest priority in the
+group, you need to tell the HA manager to not move this service
+instantly back by setting the `nofailback` option.
+
+Another scenario is when a service was fenced and it got recovered to
+another node. The admin tries to repair the fenced node and brings it
+up online again to investigate the failure cause and check if it runs
+stable again. Setting the `nofailback` flag prevents that the
+recovered services move straight back to the fenced node.
+
+
+[[ha_manager_fencing]]
+Fencing
+-------
+
+On node failures, fencing ensures that the erroneous node is
+guaranteed to be offline. This is required to make sure that no
+resource runs twice when it gets recovered on another node. This is a
+really important task, because without, it would not be possible to
+recover a resource on another node.
+
+If a node did not get fenced, it would be in an unknown state where
+it may have still access to shared resources. This is really
+dangerous! Imagine that every network but the storage one broke. Now,
+while not reachable from the public network, the VM still runs and
+writes to the shared storage.
+
+If we then simply start up this VM on another node, we would get a
+dangerous race conditions because we write from both nodes. Such
+condition can destroy all VM data and the whole VM could be rendered
+unusable. The recovery could also fail if the storage protects from
+multiple mounts.
+
+
+How {pve} Fences
+~~~~~~~~~~~~~~~~
+
+There are different methods to fence a node, for example, fence
+devices which cut off the power from the node or disable their
+communication completely. Those are often quite expensive and bring
+additional critical components into a system, because if they fail you
+cannot recover any service.
+
+We thus wanted to integrate a simpler fencing method, which does not
+require additional external hardware. This can be done using
+watchdog timers.
+
+.Possible Fencing Methods
+- external power switches
+- isolate nodes by disabling complete network traffic on the switch
+- self fencing using watchdog timers
+
+Watchdog timers are widely used in critical and dependable systems
+since the beginning of micro controllers. They are often independent
+and simple integrated circuits which are used to detect and recover
+from computer malfunctions.
+
+During normal operation, `ha-manager` regularly resets the watchdog
+timer to prevent it from elapsing. If, due to a hardware fault or
+program error, the computer fails to reset the watchdog, the timer
+will elapse and triggers a reset of the whole server (reboot).
+
+Recent server motherboards often include such hardware watchdogs, but
+these need to be configured. If no watchdog is available or
+configured, we fall back to the Linux Kernel 'softdog'. While still
+reliable, it is not independent of the servers hardware, and thus has
+a lower reliability than a hardware watchdog.
+
+
+Configure Hardware Watchdog
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, all hardware watchdog modules are blocked for security
+reasons. They are like a loaded gun if not correctly initialized. To
+enable a hardware watchdog, you need to specify the module to load in
+'/etc/default/pve-ha-manager', for example:
+
+----
+# select watchdog module (default is softdog)
+WATCHDOG_MODULE=iTCO_wdt
+----
+
+This configuration is read by the 'watchdog-mux' service, which load
+the specified module at startup.
+
+
+Recover Fenced Services
+~~~~~~~~~~~~~~~~~~~~~~~
+
+After a node failed and its fencing was successful, the CRM tries to
+move services from the failed node to nodes which are still online.
+
+The selection of nodes, on which those services gets recovered, is
+influenced by the resource `group` settings, the list of currently active
+nodes, and their respective active service count.
+
+The CRM first builds a set out of the intersection between user selected
+nodes (from `group` setting) and available nodes. It then choose the
+subset of nodes with the highest priority, and finally select the node
+with the lowest active service count. This minimizes the possibility
+of an overloaded node.
+
+CAUTION: On node failure, the CRM distributes services to the
+remaining nodes. This increase the service count on those nodes, and
+can lead to high load, especially on small clusters. Please design
+your cluster so that it can handle such worst case scenarios.
+
+
+[[ha_manager_start_failure_policy]]
+Start Failure Policy
+---------------------
+
+The start failure policy comes in effect if a service failed to start on a
+node one or more times. It can be used to configure how often a restart
+should be triggered on the same node and how often a service should be
+relocated so that it gets a try to be started on another node.
+The aim of this policy is to circumvent temporary unavailability of shared
+resources on a specific node. For example, if a shared storage isn't available
+on a quorate node anymore, e.g. network problems, but still on other nodes,
+the relocate policy allows then that the service gets started nonetheless.
+
+There are two service start recover policy settings which can be configured