be possible to move to newest corosync, or even a totally different
cluster stack. So we want:
-- possible to run with any distributed key/value store which provides
- some kind of locking with timeouts.
+- possibility to run with any distributed key/value store which provides
+ some kind of locking with timeouts (zookeeper, consul, etcd, ..)
- self fencing using Linux watchdog device
- implemented in Perl, so that we can use PVE framework
-- only works with simply resources like VMs
+- only work with simply resources like VMs
+
+We dropped the idea to assemble complex, dependend services, because we think
+this is already done with the VM abstraction.
= Architecture =
The cluster stack must provide cluster wide locks with timeouts.
The Proxmox 'pmxcfs' implements this on top of corosync.
+=== Watchdog ===
+
+We need a reliable watchdog mechanism, which is able to provide hard
+timeouts. It must be guaranteed that the node reboots within the specified
+timeout if we do not update the watchdog. For me it looks that neither
+systemd nor the standard watchdog(8) daemon provides such guarantees.
+
+We could use the /dev/watchdog directly, but unfortunately this only
+allows one user. We need to protect at least two daemons, so we write
+our own watchdog daemon. This daemon work on /dev/watchdog, but
+provides that service to several other daemons using a local socket.
+
== Self fencing ==
-A node needs to aquire a special 'ha_agent_${node}_lock' (one separate
+A node needs to acquire a special 'ha_agent_${node}_lock' (one separate
lock for each node) before starting HA resources, and the node updates
the watchdog device once it get that lock. If the node loose quorum,
or is unable to get the 'ha_agent_${node}_lock', the watchdog is no
long as there are running services on that node.
The HA manger can assume that the watchdog triggered a reboot when he
-is able to aquire the 'ha_agent_${node}_lock' for that node.
+is able to acquire the 'ha_agent_${node}_lock' for that node.
+
+=== Problems with "two_node" Clusters ===
+
+This corosync options depends on a fence race condition, and only
+works using reliable HW fence devices.
+
+Above 'self fencing' algorithm does not work if you use this option!
== Testing requirements ==
So the LRM can migrate those services without stopping them (CRM
service state 'migrate'),
-Most other service types requires the service to be stopped, and
-then restarted at the other node. We use the following CRM service
-states transitions: 'relocate_stop' => 'relocate_move' => 'started'
+Most other service types requires the service to be stopped, and then
+restarted at the other node. Stopped services are moved by the CRM
+(usually by simply changing the service configuration).
+
+=== Service ordering and colocation constarints ===
-Stopped services are moved using service state 'move'. It has to be
-noted that service relocation is always done using the LRM (the LRM
-'owns' the service), unless a node is fenced. In that case the CRM
-is allowed to 'steal' the resource and mode it to another node.
+So far there are no plans to implement this (although it would be possible).
=== Possible CRM Service States ===
fence: Wait for node fencing (service node is not inside
quorate cluster partition).
+freeze: Do not touch. We use this state while we reboot a node,
+ or when we restart the LRM daemon.
+
migrate: Migrate (live) service to other node.
error: Service disabled because of LRM errors.
-
== Local Resource Manager (class PVE::HA::LRM) ==
The Local Resource Manager (LRM) daemon runs one each node, and
projects / pve-ha-manager.git / blobdiff