The {PVE} cluster manager `pvecm` is a tool to create a group of
physical servers. Such a group is called a *cluster*. We use the
http://www.corosync.org[Corosync Cluster Engine] for reliable group
-communication, and such clusters can consist of up to 32 physical nodes
-(probably more, dependent on network latency).
+communication. There's no explicit limit for the number of nodes in a cluster.
+In practice, the actual possible node count may be limited by the host and
+network performance. Currently (2021), there are reports of clusters (using
+high-end enterprise hardware) with over 50 nodes in production.
`pvecm` can be used to create a new cluster, join nodes to a cluster,
-leave the cluster, get status information and do various other cluster
-related tasks. The **P**rox**m**o**x** **C**luster **F**ile **S**ystem (``pmxcfs'')
+leave the cluster, get status information and do various other cluster-related
+tasks. The **P**rox**m**o**x** **C**luster **F**ile **S**ystem (``pmxcfs'')
is used to transparently distribute the cluster configuration to all cluster
nodes.
* Centralized, web based management
-* Multi-master clusters: each node can do all management task
+* Multi-master clusters: each node can do all management tasks
* `pmxcfs`: database-driven file system for storing configuration files,
replicated in real-time on all nodes using `corosync`.
* Date and time have to be synchronized.
-* SSH tunnel on TCP port 22 between nodes is used.
+* SSH tunnel on TCP port 22 between nodes is used.
* If you are interested in High Availability, you need to have at
least three nodes for reliable quorum. All nodes should have the
NOTE: It is not possible to mix {pve} 3.x and earlier with {pve} 4.X cluster
nodes.
-NOTE: While it's possible for {pve} 4.4 and {pve} 5.0 this is not supported as
-production configuration and should only used temporarily during upgrading the
-whole cluster from one to another major version.
+NOTE: While it's possible to mix {pve} 4.4 and {pve} 5.0 nodes, doing so is
+not supported as production configuration and should only used temporarily
+during upgrading the whole cluster from one to another major version.
NOTE: Running a cluster of {pve} 6.x with earlier versions is not possible. The
cluster protocol (corosync) between {pve} 6.x and earlier versions changed
installed with the final hostname and IP configuration. Changing the
hostname and IP is not possible after cluster creation.
-Currently the cluster creation can either be done on the console (login via
-`ssh`) or the API, which we have a GUI implementation for (__Datacenter ->
-Cluster__).
-
While it's common to reference all nodenames and their IPs in `/etc/hosts` (or
make their names resolvable through other means), this is not necessary for a
cluster to work. It may be useful however, as you can then connect from one node
[[pvecm_create_cluster]]
-Create the Cluster
-------------------
+Create a Cluster
+----------------
+
+You can either create a cluster on the console (login via `ssh`), or through
+the API using the {pve} Webinterface (__Datacenter -> Cluster__).
+
+NOTE: Use a unique name for your cluster. This name cannot be changed later.
+The cluster name follows the same rules as node names.
+
+[[pvecm_cluster_create_via_gui]]
+Create via Web GUI
+~~~~~~~~~~~~~~~~~~
+
+[thumbnail="screenshot/gui-cluster-create.png"]
+
+Under __Datacenter -> Cluster__, click on *Create Cluster*. Enter the cluster
+name and select a network connection from the dropdown to serve as the main
+cluster network (Link 0). It defaults to the IP resolved via the node's
+hostname.
+
+To add a second link as fallback, you can select the 'Advanced' checkbox and
+choose an additional network interface (Link 1, see also
+xref:pvecm_redundancy[Corosync Redundancy]).
+
+NOTE: Ensure the network selected for the cluster communication is not used for
+any high traffic loads like those of (network) storages or live-migration.
+While the cluster network itself produces small amounts of data, it is very
+sensitive to latency. Check out full
+xref:pvecm_cluster_network_requirements[cluster network requirements].
-Login via `ssh` to the first {pve} node. Use a unique name for your cluster.
-This name cannot be changed later. The cluster name follows the same rules as
-node names.
+[[pvecm_cluster_create_via_cli]]
+Create via Command Line
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Login via `ssh` to the first {pve} node and run the following command:
----
hp1# pvecm create CLUSTERNAME
----
-NOTE: It is possible to create multiple clusters in the same physical or logical
-network. Use unique cluster names if you do so. To avoid human confusion, it is
-also recommended to choose different names even if clusters do not share the
-cluster network.
-
-To check the state of your cluster use:
+To check the state of the new cluster use:
----
hp1# pvecm status
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is possible to create multiple clusters in the same physical or logical
-network. Each such cluster must have a unique name, this does not only helps
-admins to distinguish on which cluster they currently operate, it is also
-required to avoid possible clashes in the cluster communication stack.
+network. Each such cluster must have a unique name to avoid possible clashes in
+the cluster communication stack. This also helps avoid human confusion by making
+clusters clearly distinguishable.
While the bandwidth requirement of a corosync cluster is relatively low, the
latency of packages and the package per second (PPS) rate is the limiting
Adding Nodes to the Cluster
---------------------------
-Login via `ssh` to the node you want to add.
+CAUTION: A node that is about to be added to the cluster cannot hold any guests.
+All existing configuration in `/etc/pve` is overwritten when joining a cluster,
+since guest IDs could be conflicting. As a workaround create a backup of the
+guest (`vzdump`) and restore it as a different ID after the node has been added
+to the cluster.
+
+Join Node to Cluster via GUI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+[thumbnail="screenshot/gui-cluster-join-information.png"]
+
+Login to the web interface on an existing cluster node. Under __Datacenter ->
+Cluster__, click the button *Join Information* at the top. Then, click on the
+button *Copy Information*. Alternatively, copy the string from the 'Information'
+field manually.
+
+[thumbnail="screenshot/gui-cluster-join.png"]
+
+Next, login to the web interface on the node you want to add.
+Under __Datacenter -> Cluster__, click on *Join Cluster*. Fill in the
+'Information' field with the 'Join Information' text you copied earlier.
+Most settings required for joining the cluster will be filled out
+automatically. For security reasons, the cluster password has to be entered
+manually.
+
+NOTE: To enter all required data manually, you can disable the 'Assisted Join'
+checkbox.
+
+After clicking the *Join* button, the cluster join process will start
+immediately. After the node joined the cluster its current node certificate
+will be replaced by one signed from the cluster certificate authority (CA),
+that means the current session will stop to work after a few seconds. You might
+then need to force-reload the webinterface and re-login with the cluster
+credentials.
+
+Now your node should be visible under __Datacenter -> Cluster__.
+
+Join Node to Cluster via Command Line
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Login via `ssh` to the node you want to join into an existing cluster.
----
hp2# pvecm add IP-ADDRESS-CLUSTER
For `IP-ADDRESS-CLUSTER` use the IP or hostname of an existing cluster node.
An IP address is recommended (see xref:pvecm_corosync_addresses[Link Address Types]).
-CAUTION: A new node cannot hold any VMs, because you would get
-conflicts about identical VM IDs. Also, all existing configuration in
-`/etc/pve` is overwritten when you join a new node to the cluster. To
-workaround, use `vzdump` to backup and restore to a different VMID after
-adding the node to the cluster.
To check the state of the cluster use:
If you want to use the built-in xref:pvecm_redundancy[redundancy] of the
kronosnet transport layer, also use the 'link1' parameter.
+Using the GUI, you can select the correct interface from the corresponding 'Link 0'
+and 'Link 1' fields in the *Cluster Join* dialog.
Remove a Cluster Node
---------------------
----
hp1# pvecm delnode hp4
+ Killing node 4
----
-If the operation succeeds no output is returned, just check the node
-list again with `pvecm nodes` or `pvecm status`. You should see
-something like:
+Use `pvecm nodes` or `pvecm status` to check the node list again. It should
+look something like:
----
hp1# pvecm status
scratch. But after removing the node from the cluster it will still have
access to the shared storages! This must be resolved before you start removing
the node from the cluster. A {pve} cluster cannot share the exact same
-storage with another cluster, as storage locking doesn't work over cluster
+storage with another cluster, as storage locking doesn't work over the cluster
boundary. Further, it may also lead to VMID conflicts.
Its suggested that you create a new storage where only the node which you want
WARNING: Ensure all shared resources are cleanly separated! Otherwise you will
run into conflicts and problems.
-First stop the corosync and the pve-cluster services on the node:
+First, stop the corosync and the pve-cluster services on the node:
[source,bash]
----
systemctl stop pve-cluster
[source,bash]
----
rm /etc/pve/corosync.conf
-rm /etc/corosync/*
+rm -r /etc/corosync/*
----
You can now start the filesystem again as normal service:
Setting Up A New Network
^^^^^^^^^^^^^^^^^^^^^^^^
-First you have to set up a new network interface. It should be on a physically
+First, you have to set up a new network interface. It should be on a physically
separate network. Ensure that your network fulfills the
xref:pvecm_cluster_network_requirements[cluster network requirements].
which may lead to a situation where an address is changed without thinking
about implications for corosync.
-A seperate, static hostname specifically for corosync is recommended, if
+A separate, static hostname specifically for corosync is recommended, if
hostnames are preferred. Also, make sure that every node in the cluster can
resolve all hostnames correctly.
Nodes that joined the cluster on earlier versions likely still use their
unresolved hostname in `corosync.conf`. It might be a good idea to replace
-them with IPs or a seperate hostname, as mentioned above.
+them with IPs or a separate hostname, as mentioned above.
[[pvecm_redundancy]]
Corosync supports redundant networking via its integrated kronosnet layer by
default (it is not supported on the legacy udp/udpu transports). It can be
enabled by specifying more than one link address, either via the '--linkX'
-parameters of `pvecm` (while creating a cluster or adding a new node) or by
-specifying more than one 'ringX_addr' in `corosync.conf`.
+parameters of `pvecm`, in the GUI as **Link 1** (while creating a cluster or
+adding a new node) or by specifying more than one 'ringX_addr' in
+`corosync.conf`.
NOTE: To provide useful failover, every link should be on its own
physical network connection.
Links are used according to a priority setting. You can configure this priority
by setting 'knet_link_priority' in the corresponding interface section in
-`corosync.conf`, or, preferrably, using the 'priority' parameter when creating
+`corosync.conf`, or, preferably, using the 'priority' parameter when creating
your cluster with `pvecm`:
----
- # pvecm create CLUSTERNAME --link0 10.10.10.1,priority=20 --link1 10.20.20.1,priority=15
+ # pvecm create CLUSTERNAME --link0 10.10.10.1,priority=15 --link1 10.20.20.1,priority=20
----
-This would cause 'link1' to be used first, since it has the lower priority.
+This would cause 'link1' to be used first, since it has the higher priority.
If no priorities are configured manually (or two links have the same priority),
links will be used in order of their number, with the lower number having higher
If you see a healthy cluster state, it means that your new link is being used.
+Role of SSH in {PVE} Clusters
+-----------------------------
+
+{PVE} utilizes SSH tunnels for various features.
+
+* Proxying console/shell sessions (node and guests)
++
+When using the shell for node B while being connected to node A, connects to a
+terminal proxy on node A, which is in turn connected to the login shell on node
+B via a non-interactive SSH tunnel.
+
+* VM and CT memory and local-storage migration in 'secure' mode.
++
+During the migration one or more SSH tunnel(s) are established between the
+source and target nodes, in order to exchange migration information and
+transfer memory and disk contents.
+
+* Storage replication
+
+.Pitfalls due to automatic execution of `.bashrc` and siblings
+[IMPORTANT]
+====
+In case you have a custom `.bashrc`, or similar files that get executed on
+login by the configured shell, `ssh` will automatically run it once the session
+is established successfully. This can cause some unexpected behavior, as those
+commands may be executed with root permissions on any above described
+operation. That can cause possible problematic side-effects!
+
+In order to avoid such complications, it's recommended to add a check in
+`/root/.bashrc` to make sure the session is interactive, and only then run
+`.bashrc` commands.
+
+You can add this snippet at the beginning of your `.bashrc` file:
+
+----
+# Early exit if not running interactively to avoid side-effects!
+case $- in
+ *i*) ;;
+ *) return;;
+esac
+----
+====
+
+
Corosync External Vote Support
------------------------------
QDevice Technical Overview
~~~~~~~~~~~~~~~~~~~~~~~~~~
-The Corosync Quroum Device (QDevice) is a daemon which runs on each cluster
+The Corosync Quorum Device (QDevice) is a daemon which runs on each cluster
node. It provides a configured number of votes to the clusters quorum
subsystem based on an external running third-party arbitrator's decision.
Its primary use is to allow a cluster to sustain more node failures than
~~~~~~~~~~~~~~~~~
We recommend to run any daemon which provides votes to corosync-qdevice as an
-unprivileged user. {pve} and Debian provides a package which is already
+unprivileged user. {pve} and Debian provide a package which is already
configured to do so.
The traffic between the daemon and the cluster must be encrypted to ensure a
safe and secure QDevice integration in {pve}.
-First install the 'corosync-qnetd' package on your external server and
-the 'corosync-qdevice' package on all cluster nodes.
+First, install the 'corosync-qnetd' package on your external server
+
+----
+external# apt install corosync-qnetd
+----
+
+and the 'corosync-qdevice' package on all cluster nodes
+
+----
+pve# apt install corosync-qdevice
+----
After that, ensure that all your nodes on the cluster are online.
pve# pvecm qdevice setup <QDEVICE-IP>
----
-The SSH key from the cluster will be automatically copied to the QDevice. You
-might need to enter an SSH password during this step.
+The SSH key from the cluster will be automatically copied to the QDevice.
+
+NOTE: Make sure that the SSH configuration on your external server allows root
+login via password, if you are asked for a password during this step.
After you enter the password and all the steps are successfully completed, you
will see "Done". You can check the status now:
This sets the expected vote count to 1 and makes the cluster quorate. You can
now fix your configuration, or revert it back to the last working backup.
-This is not enough if corosync cannot start anymore. Here its best to edit the
+This is not enough if corosync cannot start anymore. Here it is best to edit the
local copy of the corosync configuration in '/etc/corosync/corosync.conf' so
that corosync can start again. Ensure that on all nodes this configuration has
the same content to avoid split brains. If you are not sure what went wrong
Therefore, we strongly recommend using the secure channel if you do
not have full control over the network and can not guarantee that no
-one is eavesdropping to it.
+one is eavesdropping on it.
NOTE: Storage migration does not follow this setting. Currently, it
always sends the storage content over a secure channel.
address 192.X.Y.57
netmask 255.255.250.0
gateway 192.X.Y.1
- bridge_ports eno1
- bridge_stp off
- bridge_fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
# cluster network
auto eno2
projects / pve-docs.git / blobdiff