How It Works
------------
-This section provides an in detail description of the {PVE} HA-manager
-internals. It describes how the CRM and the LRM work together.
-
-To provide High Availability two daemons run on each node:
+This section provides a detailed description of the {PVE} HA manager
+internals. It describes all involved daemons and how they work
+together. To provide HA, two daemons run on each node:
`pve-ha-lrm`::
This all gets supervised by the CRM which holds currently the manager master
lock.
+
+Service States
+~~~~~~~~~~~~~~
+
+The CRM use a service state enumeration to record the current service
+state. We display this state on the GUI and you can query it using
+the `ha-manager` command line tool:
+
+----
+# ha-manager status
+quorum OK
+master elsa (active, Mon Nov 21 07:23:29 2016)
+lrm elsa (active, Mon Nov 21 07:23:22 2016)
+service ct:100 (elsa, stopped)
+service ct:102 (elsa, started)
+service vm:501 (elsa, started)
+----
+
+Here is the list of possible states:
+
+stopped::
+
+Service is stopped (confirmed by LRM). If the LRM detects a stopped
+service is still running, it will stop it again.
+
+request_stop::
+
+Service should be stopped. The CRM waits for confirmation from the
+LRM.
+
+started::
+
+Service is active an LRM should start it ASAP if not already running.
+If the Service fails and is detected to be not running the LRM
+restarts it
+(see xref:ha_manager_start_failure_policy[Start Failure Policy]).
+
+fence::
+
+Wait for node fencing (service node is not inside quorate cluster
+partition). As soon as node gets fenced successfully the service will
+be recovered to another node, if possible
+(see xref:ha_manager_fencing[Fencing]).
+
+freeze::
+
+Do not touch the service state. We use this state while we reboot a
+node, or when we restart the LRM daemon
+(see xref:ha_manager_package_updates[Package Updates]).
+
+migrate::
+
+Migrate service (live) to other node.
+
+error::
+
+Service is disabled because of LRM errors. Needs manual intervention
+(see xref:ha_manager_error_recovery[Error Recovery]).
+
+
Local Resource Manager
~~~~~~~~~~~~~~~~~~~~~~
max_relocate 2
ct: 102
- # use default settings for everything
+ # Note: use default settings for everything
----
+Above config was generated using the `ha-manager` command line tool:
+
+----
+# ha-manager add vm:501 --state started --max_relocate 2
+# ha-manager add ct:102
+----
+
+[[ha_manager_groups]]
Groups
~~~~~~
include::ha-groups-opts.adoc[]
+A commom requirement is that a resource should run on a specific
+node. Usually the resource is able to run on other nodes, so you can define
+an unrestricted group with a single member:
+
+----
+# ha-manager groupadd prefer_node1 --nodes node1
+----
+
+For bigger clusters, it makes sense to define a more detailed failover
+behavior. For example, you may want to run a set of services on
+`node1` if possible. If `node1` is not available, you want to run them
+equally splitted on `node2` and `node3`. If those nodes also fail the
+services should run on `node4`. To achieve this you could set the node
+list to:
+
+----
+# ha-manager groupadd mygroup1 -nodes "node1:2,node2:1,node3:1,node4"
+----
+
+Another use case is if a resource uses other resources only available
+on specific nodes, lets say `node1` and `node2`. We need to make sure
+that HA manager does not use other nodes, so we need to create a
+restricted group with said nodes:
+
+----
+# ha-manager groupadd mygroup2 -nodes "node1,node2" -restricted
+----
+
+Above commands created the following group configuration fils:
+
+.Configuration Example (`/etc/pve/ha/groups.cfg`)
+----
+group: prefer_node1
+ nodes node1
+
+group: mygroup1
+ nodes node2:1,node4,node1:2,node3:1
+
+group: mygroup2
+ nodes node2,node1
+ restricted 1
+----
+
+
+The `nofailback` options is mostly useful to avoid unwanted resource
+movements during administartion tasks. For example, if you need to
+migrate a service to a node which hasn't the highest priority in the
+group, you need to tell the HA manager to not move this service
+instantly back by setting the `nofailback` option.
+
+Another scenario is when a service was fenced and it got recovered to
+another node. The admin tries to repair the fenced node and brings it
+up online again to investigate the failure cause and check if it runs
+stable again. Setting the `nofailback` flag prevents that the
+recovered services move straight back to the fenced node.
+
Node Power Status
-----------------
After that you can stop the LRM and CRM services. But note that the
watchdog triggers if you stop it with active services.
+
+[[ha_manager_package_updates]]
Package Updates
---------------
unresponsive node and as a result a chain reaction of node failures in the
cluster.
-[[ha_manager_groups]]
-Groups
-------
-
-A group is a collection of cluster nodes which a service may be bound to.
-
-Group Settings
-~~~~~~~~~~~~~~
-
-nodes::
-
-List of group node members where a priority can be given to each node.
-A service bound to this group will run on the nodes with the highest priority
-available. If more nodes are in the highest priority class the services will
-get distributed to those node if not already there. The priorities have a
-relative meaning only.
- Example;;
- You want to run all services from a group on `node1` if possible. If this node
- is not available, you want them to run equally splitted on `node2` and `node3`, and
- if those fail it should use `node4`.
- To achieve this you could set the node list to:
-[source,bash]
- ha-manager groupset mygroup -nodes "node1:2,node2:1,node3:1,node4"
-
-restricted::
-
-Resources bound to this group may only run on nodes defined by the
-group. If no group node member is available the resource will be
-placed in the stopped state.
- Example;;
- Lets say a service uses resources only available on `node1` and `node2`,
- so we need to make sure that HA manager does not use other nodes.
- We need to create a 'restricted' group with said nodes:
-[source,bash]
- ha-manager groupset mygroup -nodes "node1,node2" -restricted
-
-nofailback::
-
-The resource won't automatically fail back when a more preferred node
-(re)joins the cluster.
- Examples;;
- * You need to migrate a service to a node which hasn't the highest priority
- in the group at the moment, to tell the HA manager to not move this service
- instantly back set the 'nofailback' option and the service will stay on
- the current node.
-
- * A service was fenced and it got recovered to another node. The admin
- repaired the node and brought it up online again but does not want that the
- recovered services move straight back to the repaired node as he wants to
- first investigate the failure cause and check if it runs stable. He can use
- the 'nofailback' option to achieve this.
-
+[[ha_manager_start_failure_policy]]
Start Failure Policy
---------------------
re-enabled without fixing the error only the restart policy gets
repeated.
+
+[[ha_manager_error_recovery]]
Error Recovery
--------------
service state (enabled, disabled).
-Service States
---------------
-
-stopped::
-
-Service is stopped (confirmed by LRM), if detected running it will get stopped
-again.
-
-request_stop::
-
-Service should be stopped. Waiting for confirmation from LRM.
-
-started::
-
-Service is active an LRM should start it ASAP if not already running.
-If the Service fails and is detected to be not running the LRM restarts it.
-
-fence::
-
-Wait for node fencing (service node is not inside quorate cluster
-partition).
-As soon as node gets fenced successfully the service will be recovered to
-another node, if possible.
-
-freeze::
-
-Do not touch the service state. We use this state while we reboot a
-node, or when we restart the LRM daemon.
-
-migrate::
-
-Migrate service (live) to other node.
-
-error::
-
-Service disabled because of LRM errors. Needs manual intervention.
-
-
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff