* All nodes must be in the same network as corosync uses IP Multicast
to communicate between nodes (also see
- http://www.corosync.org[Corosync Cluster Engine]). NOTE: Some
- switches do not support IP multicast by default and must be manually
- enabled first.
+ http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
+ ports 5404 and 5405 for cluster communication.
++
+NOTE: Some switches do not support IP multicast by default and must be
+manually enabled first.
* Date and time have to be synchronized.
-* SSH tunnel on port 22 between nodes is used.
+* SSH tunnel on TCP port 22 between nodes is used.
-* If you are interested in High Availability too, for reliable quorum
- you must have at least 3 nodes (all nodes should have the same
- version).
+* If you are interested in High Availability, you need to have at
+ least three nodes for reliable quorum. All nodes should have the
+ same version.
* We recommend a dedicated NIC for the cluster traffic, especially if
you use shared storage.
NOTE: It is not possible to mix Proxmox VE 3.x and earlier with
-Proxmox VE 4.0 cluster.
+Proxmox VE 4.0 cluster nodes.
-{PVE} Cluster Setup
--------------------
+Preparing Nodes
+---------------
First, install {PVE} on all nodes. Make sure that each node is
installed with the final hostname and IP configuration. Changing the
Currently the cluster creation has to be done on the console, so you
need to login via 'ssh'.
-
Create the Cluster
-~~~~~~~~~~~~~~~~~~
+------------------
Login via 'ssh' to the first Proxmox VE node. Use a unique name for
your cluster. This name cannot be changed later.
hp1# pvecm create YOUR-CLUSTER-NAME
+CAUTION: The cluster name is used to compute the default multicast
+address. Please use unique cluster names if you run more than one
+cluster inside your network.
+
To check the state of your cluster use:
hp1# pvecm status
Adding Nodes to the Cluster
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------------
Login via 'ssh' to the node you want to add.
For `IP-ADDRESS-CLUSTER` use the IP from an existing cluster node.
CAUTION: A new node cannot hold any VM´s, because you would get
-conflicts about identical VM IDs. To workaround, use vzdump to backup
-and to restore to a different VMID after adding the node to the
-cluster.
+conflicts about identical VM IDs. Also, all existing configuration in
+'/etc/pve' is overwritten when you join a new node to the cluster. To
+workaround, use vzdump to backup and restore to a different VMID after
+adding the node to the cluster.
To check the state of cluster:
# pvecm status
-.Check Cluster Status
+.Cluster status after adding 4 nodes
----
hp2# pvecm status
Quorum information
Remove a Cluster Node
-~~~~~~~~~~~~~~~~~~~~~
+---------------------
CAUTION: Read carefully the procedure before proceeding, as it could
not be what you want or need.
data or backups you want to keep, or save them accordingly.
Log in to one remaining node via ssh. Issue a 'pvecm nodes' command to
-identify the nodeID:
+identify the node ID:
----
hp1# pvecm status
* then join it, as explained in the previous section.
+Quorum
+------
+
+{pve} use a quorum-based technique to provide a consistent state among
+all cluster nodes.
+
+[quote, from Wikipedia, Quorum (distributed computing)]
+____
+A quorum is the minimum number of votes that a distributed transaction
+has to obtain in order to be allowed to perform an operation in a
+distributed system.
+____
+
+In case of network partitioning, state changes requires that a
+majority of nodes are online. The cluster switches to read-only mode
+if it loose quorum.
+
+NOTE: {pve} assigns a single vote to each node by default.
+
+
+Cluster Cold Start
+------------------
+
+It is obvious that a cluster is not quorate when all nodes are
+offline. This is a common case after a power failure.
+
+NOTE: It is always a good idea to use an uninterruptible power supply
+('UPS', also called 'battery backup') to avoid this state. Especially if
+you want HA.
+
+On node startup, service 'pve-manager' waits up to 60 seconds to reach
+quorum, and then starts all guests. If it fails to get quorum, that
+service simply aborts, and you need to start your guest manually once
+you have quorum.
+
+If you start all nodes at the same time (for example when power comes
+back), it is likely that you reach quorum within above timeout. But
+startup can fail if some nodes starts much faster than others, so you
+need to start your guest manually after reaching quorum. You can do
+that on the GUI, or on the command line with:
+
+ systemctl start pve-manager
+
+
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff