times of about 2 minutes, so you can get no more than 99.999%
availability.
+
Requirements
------------
+You must meet the following requirements before you start with HA:
+
* at least three cluster nodes (to get reliable quorum)
* shared storage for VMs and containers
* hardware redundancy (everywhere)
+* use reliable “server” components
+
* hardware watchdog - if not available we fall back to the
linux kernel software watchdog (`softdog`)
How It Works
------------
-This section provides an in detail description of the {PVE} HA-manager
-internals. It describes how the CRM and the LRM work together.
-
-To provide High Availability two daemons run on each node:
+This section provides a detailed description of the {PVE} HA manager
+internals. It describes all involved daemons and how they work
+together. To provide HA, two daemons run on each node:
`pve-ha-lrm`::
This all gets supervised by the CRM which holds currently the manager master
lock.
+
+Service States
+~~~~~~~~~~~~~~
+
+The CRM use a service state enumeration to record the current service
+state. We display this state on the GUI and you can query it using
+the `ha-manager` command line tool:
+
+----
+# ha-manager status
+quorum OK
+master elsa (active, Mon Nov 21 07:23:29 2016)
+lrm elsa (active, Mon Nov 21 07:23:22 2016)
+service ct:100 (elsa, stopped)
+service ct:102 (elsa, started)
+service vm:501 (elsa, started)
+----
+
+Here is the list of possible states:
+
+stopped::
+
+Service is stopped (confirmed by LRM). If the LRM detects a stopped
+service is still running, it will stop it again.
+
+request_stop::
+
+Service should be stopped. The CRM waits for confirmation from the
+LRM.
+
+started::
+
+Service is active an LRM should start it ASAP if not already running.
+If the Service fails and is detected to be not running the LRM
+restarts it
+(see xref:ha_manager_start_failure_policy[Start Failure Policy]).
+
+fence::
+
+Wait for node fencing (service node is not inside quorate cluster
+partition). As soon as node gets fenced successfully the service will
+be recovered to another node, if possible
+(see xref:ha_manager_fencing[Fencing]).
+
+freeze::
+
+Do not touch the service state. We use this state while we reboot a
+node, or when we restart the LRM daemon
+(see xref:ha_manager_package_updates[Package Updates]).
+
+migrate::
+
+Migrate service (live) to other node.
+
+error::
+
+Service is disabled because of LRM errors. Needs manual intervention
+(see xref:ha_manager_error_recovery[Error Recovery]).
+
+
Local Resource Manager
~~~~~~~~~~~~~~~~~~~~~~
inside that list look like this:
----
-<sid>:
+<type>: <name>
<property> <value>
...
----
-It starts with the service ID followed by a collon. The next lines
-contain additional properties:
+It starts with a resource type followed by a resource specific name,
+separated with colon. Together this forms the HA resource ID, which is
+used by all `ha-manager` commands to uniquely identify a resource
+(example: `vm:100` or `ct:101`). The next lines contain additional
+properties:
include::ha-resources-opts.adoc[]
+Here is a real world example with one VM and one container. As you see,
+the syntax of those files is really simple, so it is even posiible to
+read or edit those files using your favorite editor:
+.Configuration Example (`/etc/pve/ha/resources.cfg`)
+----
+vm: 501
+ state started
+ max_relocate 2
+
+ct: 102
+ # Note: use default settings for everything
+----
+
+Above config was generated using the `ha-manager` command line tool:
+
+----
+# ha-manager add vm:501 --state started --max_relocate 2
+# ha-manager add ct:102
+----
+
+
+[[ha_manager_groups]]
Groups
~~~~~~
include::ha-groups-opts.adoc[]
+A commom requirement is that a resource should run on a specific
+node. Usually the resource is able to run on other nodes, so you can define
+an unrestricted group with a single member:
+
+----
+# ha-manager groupadd prefer_node1 --nodes node1
+----
+
+For bigger clusters, it makes sense to define a more detailed failover
+behavior. For example, you may want to run a set of services on
+`node1` if possible. If `node1` is not available, you want to run them
+equally splitted on `node2` and `node3`. If those nodes also fail the
+services should run on `node4`. To achieve this you could set the node
+list to:
+
+----
+# ha-manager groupadd mygroup1 -nodes "node1:2,node2:1,node3:1,node4"
+----
+
+Another use case is if a resource uses other resources only available
+on specific nodes, lets say `node1` and `node2`. We need to make sure
+that HA manager does not use other nodes, so we need to create a
+restricted group with said nodes:
+
+----
+# ha-manager groupadd mygroup2 -nodes "node1,node2" -restricted
+----
+
+Above commands created the following group configuration fils:
+
+.Configuration Example (`/etc/pve/ha/groups.cfg`)
+----
+group: prefer_node1
+ nodes node1
+
+group: mygroup1
+ nodes node2:1,node4,node1:2,node3:1
+
+group: mygroup2
+ nodes node2,node1
+ restricted 1
+----
+
+
+The `nofailback` options is mostly useful to avoid unwanted resource
+movements during administartion tasks. For example, if you need to
+migrate a service to a node which hasn't the highest priority in the
+group, you need to tell the HA manager to not move this service
+instantly back by setting the `nofailback` option.
+
+Another scenario is when a service was fenced and it got recovered to
+another node. The admin tries to repair the fenced node and brings it
+up online again to investigate the failure cause and check if it runs
+stable again. Setting the `nofailback` flag prevents that the
+recovered services move straight back to the fenced node.
+
Node Power Status
-----------------
After that you can stop the LRM and CRM services. But note that the
watchdog triggers if you stop it with active services.
+
+[[ha_manager_package_updates]]
Package Updates
---------------
Fencing
-------
-What is Fencing
-~~~~~~~~~~~~~~~
+On node failures, fencing ensures that the erroneous node is
+guaranteed to be offline. This is required to make sure that no
+resource runs twice when it gets recovered on another node. This is a
+really important task, because without, it would not be possible to
+recover a resource on another node.
-Fencing secures that on a node failure the dangerous node gets will be rendered
-unable to do any damage and that no resource runs twice when it gets recovered
-from the failed node. This is a really important task and one of the base
-principles to make a system Highly Available.
+If a node would not get fenced, it would be in an unknown state where
+it may have still access to shared resources. This is really
+dangerous! Imagine that every network but the storage one broke. Now,
+while not reachable from the public network, the VM still runs and
+writes to the shared storage.
+
+If we then simply start up this VM on another node, we would get a
+dangerous race conditions because we write from both nodes. Such
+condition can destroy all VM data and the whole VM could be rendered
+unusable. The recovery could also fail if the storage protects from
+multiple mounts.
-If a node would not get fenced it would be in an unknown state where it may
-have still access to shared resources, this is really dangerous!
-Imagine that every network but the storage one broke, now while not
-reachable from the public network the VM still runs and writes on the shared
-storage. If we would not fence the node and just start up this VM on another
-Node we would get dangerous race conditions, atomicity violations the whole VM
-could be rendered unusable. The recovery could also simply fail if the storage
-protects from multiple mounts and thus defeat the purpose of HA.
How {pve} Fences
-~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~
There are different methods to fence a node, for example fence devices which
cut off the power from the node or disable their communication completely.
unresponsive node and as a result a chain reaction of node failures in the
cluster.
-[[ha_manager_groups]]
-Groups
-------
-
-A group is a collection of cluster nodes which a service may be bound to.
-
-Group Settings
-~~~~~~~~~~~~~~
-
-nodes::
-
-List of group node members where a priority can be given to each node.
-A service bound to this group will run on the nodes with the highest priority
-available. If more nodes are in the highest priority class the services will
-get distributed to those node if not already there. The priorities have a
-relative meaning only.
- Example;;
- You want to run all services from a group on `node1` if possible. If this node
- is not available, you want them to run equally splitted on `node2` and `node3`, and
- if those fail it should use `node4`.
- To achieve this you could set the node list to:
-[source,bash]
- ha-manager groupset mygroup -nodes "node1:2,node2:1,node3:1,node4"
-
-restricted::
-
-Resources bound to this group may only run on nodes defined by the
-group. If no group node member is available the resource will be
-placed in the stopped state.
- Example;;
- Lets say a service uses resources only available on `node1` and `node2`,
- so we need to make sure that HA manager does not use other nodes.
- We need to create a 'restricted' group with said nodes:
-[source,bash]
- ha-manager groupset mygroup -nodes "node1,node2" -restricted
-
-nofailback::
-
-The resource won't automatically fail back when a more preferred node
-(re)joins the cluster.
- Examples;;
- * You need to migrate a service to a node which hasn't the highest priority
- in the group at the moment, to tell the HA manager to not move this service
- instantly back set the 'nofailback' option and the service will stay on
- the current node.
-
- * A service was fenced and it got recovered to another node. The admin
- repaired the node and brought it up online again but does not want that the
- recovered services move straight back to the repaired node as he wants to
- first investigate the failure cause and check if it runs stable. He can use
- the 'nofailback' option to achieve this.
-
+[[ha_manager_start_failure_policy]]
Start Failure Policy
---------------------
re-enabled without fixing the error only the restart policy gets
repeated.
+
+[[ha_manager_error_recovery]]
Error Recovery
--------------
service state (enabled, disabled).
-Service States
---------------
-
-stopped::
-
-Service is stopped (confirmed by LRM), if detected running it will get stopped
-again.
-
-request_stop::
-
-Service should be stopped. Waiting for confirmation from LRM.
-
-started::
-
-Service is active an LRM should start it ASAP if not already running.
-If the Service fails and is detected to be not running the LRM restarts it.
-
-fence::
-
-Wait for node fencing (service node is not inside quorate cluster
-partition).
-As soon as node gets fenced successfully the service will be recovered to
-another node, if possible.
-
-freeze::
-
-Do not touch the service state. We use this state while we reboot a
-node, or when we restart the LRM daemon.
-
-migrate::
-
-Migrate service (live) to other node.
-
-error::
-
-Service disabled because of LRM errors. Needs manual intervention.
-
-
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff