Expected votes: 4
Highest expected: 4
Total votes: 4
-Quorum: 2
+Quorum: 3
Flags: Quorate
Membership information
Expected votes: 3
Highest expected: 3
Total votes: 3
-Quorum: 3
+Quorum: 2
Flags: Quorate
Membership information
configured and the stop the corosync service on all nodes. After corosync is
stopped on all nodes start it one after the other again.
+Corosync External Vote Support
+------------------------------
+
+This section describes a way to deploy an external voter in a {pve} cluster.
+When configured, the cluster can sustain more node failures without
+violating safety properties of the cluster communication.
+
+For this to work there are two services involved:
+
+* a so called qdevice daemon which runs on each {pve} node
+
+* an external vote daemon which runs on an independent server.
+
+As a result you can achieve higher availability even in smaller setups (for
+example 2+1 nodes).
+
+QDevice Technical Overview
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Corosync Quroum Device (QDevice) is a daemon which runs on each cluster
+node. It provides a configured number of votes to the clusters quorum
+subsystem based on an external running third-party arbitrator's decision.
+Its primary use is to allow a cluster to sustain more node failures than
+standard quorum rules allow. This can be done safely as the external device
+can see all nodes and thus choose only one set of nodes to give its vote.
+This will only be done if said set of nodes can have quorum (again) when
+receiving the third-party vote.
+
+Currently only 'QDevice Net' is supported as a third-party arbitrator. It is
+a daemon which provides a vote to a cluster partition if it can reach the
+partition members over the network. It will give only votes to one partition
+of a cluster at any time.
+It's designed to support multiple clusters and is almost configuration and
+state free. New clusters are handled dynamically and no configuration file
+is needed on the host running a QDevice.
+
+The external host has the only requirement that it needs network access to the
+cluster and a corosync-qnetd package available. We provide such a package
+for Debian based hosts, other Linux distributions should also have a package
+available through their respective package manager.
+
+NOTE: In contrast to corosync itself, a QDevice connects to the cluster over
+TCP/IP and thus does not need a multicast capable network between itself and
+the cluster. In fact the daemon may run outside of the LAN and can have
+longer latencies than 2 ms.
+
+
+Supported Setups
+~~~~~~~~~~~~~~~~
+
+We support QDevices for clusters with an even number of nodes and recommend
+it for 2 node clusters, if they should provide higher availability.
+For clusters with an odd node count we discourage the use of QDevices
+currently. The reason for this, is the difference of the votes the QDevice
+provides for each cluster type. Even numbered clusters get single additional
+vote, with this we can only increase availability, i.e. if the QDevice
+itself fails we are in the same situation as with no QDevice at all.
+
+Now, with an odd numbered cluster size the QDevice provides '(N-1)' votes --
+where 'N' corresponds to the cluster node count. This difference makes
+sense, if we had only one additional vote the cluster can get into a split
+brain situation.
+This algorithm would allow that all nodes but one (and naturally the
+QDevice itself) could fail.
+There are two drawbacks with this:
+
+* If the QNet daemon itself fails, no other node may fail or the cluster
+ immediately loses quorum. For example, in a cluster with 15 nodes 7
+ could fail before the cluster becomes inquorate. But, if a QDevice is
+ configured here and said QDevice fails itself **no single node** of
+ the 15 may fail. The QDevice acts almost as a single point of failure in
+ this case.
+
+* The fact that all but one node plus QDevice may fail sound promising at
+ first, but this may result in a mass recovery of HA services that would
+ overload the single node left. Also ceph server will stop to provide
+ services after only '((N-1)/2)' nodes are online.
+
+If you understand the drawbacks and implications you can decide yourself if
+you should use this technology in an odd numbered cluster setup.
+
+
+QDevice-Net Setup
+~~~~~~~~~~~~~~~~~
+
+We recommend to run any daemon which provides votes to corosync-qdevice as an
+unprivileged user. {pve} and Debian provides a package which is already
+configured to do so.
+The traffic between the daemon and the cluster must be encrypted to ensure a
+safe and secure QDevice integration in {pve}.
+
+First install the 'corosync-qnetd' package on your external server and
+the 'corosync-qdevice' package on all cluster nodes.
+
+After that, ensure that all your nodes on the cluster are online.
+
+You can now easily set up your QDevice by running the following command on one
+of the {pve} nodes:
+
+----
+pve# pvecm qdevice setup <QDEVICE-IP>
+----
+
+The SSH key from the cluster will be automatically copied to the QDevice. You
+might need to enter an SSH password during this step.
+
+After you enter the password and all the steps are successfully completed, you
+will see "Done". You can check the status now:
+
+----
+pve# pvecm status
+
+...
+
+Votequorum information
+~~~~~~~~~~~~~~~~~~~~~
+Expected votes: 3
+Highest expected: 3
+Total votes: 3
+Quorum: 2
+Flags: Quorate Qdevice
+
+Membership information
+~~~~~~~~~~~~~~~~~~~~~~
+ Nodeid Votes Qdevice Name
+ 0x00000001 1 A,V,NMW 192.168.22.180 (local)
+ 0x00000002 1 A,V,NMW 192.168.22.181
+ 0x00000000 1 Qdevice
+
+----
+
+which means the QDevice is set up.
+
+
+Frequently Asked Questions
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tie Breaking
+^^^^^^^^^^^^
+
+In case of a tie, where two same-sized cluster partitions cannot see each other
+but the QDevice, the QDevice chooses randomly one of those partitions and
+provides a vote to it.
+
+Possible Negative Implications
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For clusters with an even node count there are no negative implications when
+setting up a QDevice. If it fails to work, you are as good as without QDevice at
+all.
+
+Adding/Deleting Nodes After QDevice Setup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to add a new node or remove an existing one from a cluster with a
+QDevice setup, you need to remove the QDevice first. After that, you can add or
+remove nodes normally. Once you have a cluster with an even node count again,
+you can set up the QDevice again as described above.
+
+Removing the QDevice
+^^^^^^^^^^^^^^^^^^^^
+
+If you used the official `pvecm` tool to add the QDevice, you can remove it
+trivially by running:
+
+----
+pve# pvecm qdevice remove
+----
+
+//Still TODO
+//^^^^^^^^^^
+//There ist still stuff to add here
+
+
Corosync Configuration
----------------------
projects / pve-docs.git / blobdiff