+[[chapter_pvecm]]
ifdef::manvolnum[]
-PVE({manvolnum})
-================
-include::attributes.txt[]
+pvecm(1)
+========
+:pve-toplevel:
NAME
----
pvecm - Proxmox VE Cluster Manager
-SYNOPSYS
+SYNOPSIS
--------
include::pvecm.1-synopsis.adoc[]
ifndef::manvolnum[]
Cluster Manager
===============
-include::attributes.txt[]
+:pve-toplevel:
endif::manvolnum[]
The {PVE} cluster manager `pvecm` is a tool to create a group of
* We recommend a dedicated NIC for the cluster traffic, especially if
you use shared storage.
-NOTE: It is not possible to mix Proxmox VE 3.x and earlier with
-Proxmox VE 4.0 cluster nodes.
+* Root password of a cluster node is required for adding nodes.
+
+NOTE: It is not possible to mix {pve} 3.x and earlier with {pve} 4.X cluster
+nodes.
+
+NOTE: While it's possible for {pve} 4.4 and {pve} 5.0 this is not supported as
+production configuration and should only used temporarily during upgrading the
+whole cluster from one to another major version.
Preparing Nodes
installed with the final hostname and IP configuration. Changing the
hostname and IP is not possible after cluster creation.
-Currently the cluster creation has to be done on the console, so you
-need to login via `ssh`.
+Currently the cluster creation can either be done on the console (login via
+`ssh`) or the API, which we have a GUI implementation for (__Datacenter ->
+Cluster__).
+While it's often common use to reference all other nodenames in `/etc/hosts`
+with their IP this is not strictly necessary for a cluster, which normally uses
+multicast, to work. It maybe useful as you then can connect from one node to
+the other with SSH through the easier to remember node name.
+
+[[pvecm_create_cluster]]
Create the Cluster
------------------
Login via `ssh` to the first {pve} node. Use a unique name for your cluster.
-This name cannot be changed later.
+This name cannot be changed later. The cluster name follows the same rules as
+node names.
- hp1# pvecm create YOUR-CLUSTER-NAME
+----
+ hp1# pvecm create CLUSTERNAME
+----
-CAUTION: The cluster name is used to compute the default multicast
-address. Please use unique cluster names if you run more than one
-cluster inside your network.
+CAUTION: The cluster name is used to compute the default multicast address.
+Please use unique cluster names if you run more than one cluster inside your
+network. To avoid human confusion, it is also recommended to choose different
+names even if clusters do not share the cluster network.
To check the state of your cluster use:
+----
hp1# pvecm status
+----
+
+Multiple Clusters In Same Network
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+It is possible to create multiple clusters in the same physical or logical
+network. Each cluster must have a unique name, which is used to generate the
+cluster's multicast group address. As long as no duplicate cluster names are
+configured in one network segment, the different clusters won't interfere with
+each other.
+If multiple clusters operate in a single network it may be beneficial to setup
+an IGMP querier and enable IGMP Snooping in said network. This may reduce the
+load of the network significantly because multicast packets are only delivered
+to endpoints of the respective member nodes.
+
+
+[[pvecm_join_node_to_cluster]]
Adding Nodes to the Cluster
---------------------------
Login via `ssh` to the node you want to add.
+----
hp2# pvecm add IP-ADDRESS-CLUSTER
+----
For `IP-ADDRESS-CLUSTER` use the IP from an existing cluster node.
To check the state of cluster:
+----
# pvecm status
+----
.Cluster status after adding 4 nodes
----
If you only want the list of all nodes use:
+----
# pvecm nodes
+----
.List nodes in a cluster
----
4 1 hp4
----
+[[adding-nodes-with-separated-cluster-network]]
Adding Nodes With Separated Cluster Network
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
use the 'ringX_addr' parameters to set the nodes address on those networks:
[source,bash]
+----
pvecm add IP-ADDRESS-CLUSTER -ring0_addr IP-ADDRESS-RING0
+----
If you want to use the Redundant Ring Protocol you will also want to pass the
'ring1_addr' parameter.
Move all virtual machines from the node. Make sure you have no local
data or backups you want to keep, or save them accordingly.
+In the following example we will remove the node hp4 from the cluster.
-Log in to one remaining node via ssh. Issue a `pvecm nodes` command to
-identify the node ID:
-
-----
-hp1# pvecm status
-
-Quorum information
-~~~~~~~~~~~~~~~~~~
-Date: Mon Apr 20 12:30:13 2015
-Quorum provider: corosync_votequorum
-Nodes: 4
-Node ID: 0x00000001
-Ring ID: 1928
-Quorate: Yes
-
-Votequorum information
-~~~~~~~~~~~~~~~~~~~~~~
-Expected votes: 4
-Highest expected: 4
-Total votes: 4
-Quorum: 2
-Flags: Quorate
-
-Membership information
-~~~~~~~~~~~~~~~~~~~~~~
- Nodeid Votes Name
-0x00000001 1 192.168.15.91 (local)
-0x00000002 1 192.168.15.92
-0x00000003 1 192.168.15.93
-0x00000004 1 192.168.15.94
-----
-
-IMPORTANT: at this point you must power off the node to be removed and
-make sure that it will not power on again (in the network) as it
-is.
+Log in to a *different* cluster node (not hp4), and issue a `pvecm nodes`
+command to identify the node ID to remove:
----
hp1# pvecm nodes
4 1 hp4
----
-Log in to one remaining node via ssh. Issue the delete command (here
-deleting node `hp4`):
+At this point you must power off hp4 and
+make sure that it will not power on again (in the network) as it
+is.
+
+IMPORTANT: As said above, it is critical to power off the node
+*before* removal, and make sure that it will *never* power on again
+(in the existing cluster network) as it is.
+If you power on the node as it is, your cluster will be screwed up and
+it could be difficult to restore a clean cluster state.
+
+After powering off the node hp4, we can safely remove it from the cluster.
+
+----
hp1# pvecm delnode hp4
+----
If the operation succeeds no output is returned, just check the node
list again with `pvecm nodes` or `pvecm status`. You should see
0x00000003 1 192.168.15.92
----
-IMPORTANT: as said above, it is very important to power off the node
-*before* removal, and make sure that it will *never* power on again
-(in the existing cluster network) as it is.
-
-If you power on the node as it is, your cluster will be screwed up and
-it could be difficult to restore a clean cluster state.
-
If, for whatever reason, you want that this server joins the same
cluster again, you have to
* then join it, as explained in the previous section.
+[[pvecm_separate_node_without_reinstall]]
Separate A Node Without Reinstalling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
scratch. But after removing the node from the cluster it will still have
access to the shared storages! This must be resolved before you start removing
the node from the cluster. A {pve} cluster cannot share the exact same
-storage with another cluster, as it leads to VMID conflicts.
+storage with another cluster, as storage locking doesn't work over cluster
+boundary. Further, it may also lead to VMID conflicts.
Its suggested that you create a new storage where only the node which you want
to separate has access. This can be an new export on your NFS or a new Ceph
First stop the corosync and the pve-cluster services on the node:
[source,bash]
+----
systemctl stop pve-cluster
systemctl stop corosync
+----
Start the cluster filesystem again in local mode:
[source,bash]
+----
pmxcfs -l
+----
Delete the corosync configuration files:
[source,bash]
+----
rm /etc/pve/corosync.conf
rm /etc/corosync/*
+----
You can now start the filesystem again as normal service:
[source,bash]
+----
killall pmxcfs
systemctl start pve-cluster
+----
The node is now separated from the cluster. You can deleted it from a remaining
node of the cluster with:
[source,bash]
+----
pvecm delnode oldnode
+----
If the command failed, because the remaining node in the cluster lost quorum
when the now separate node exited, you may set the expected votes to 1 as a workaround:
[source,bash]
+----
pvecm expected 1
+----
And the repeat the 'pvecm delnode' command.
cluster again without problems.
[source,bash]
+----
rm /var/lib/corosync/*
+----
As the configuration files from the other nodes are still in the cluster
filesystem you may want to clean those up too. Remove simply the whole
* Ensure that multicast works in general and a high package rates. This can be
done with the `omping` tool. The final "%loss" number should be < 1%.
++
[source,bash]
----
omping -c 10000 -i 0.001 -F -q NODE1-IP NODE2-IP ...
----
* Ensure that multicast communication works over an extended period of time.
- This covers up problems where IGMP snooping is activated on the network but
+ This uncovers problems where IGMP snooping is activated on the network but
no multicast querier is active. This test has a duration of around 10
minutes.
++
[source,bash]
+----
omping -c 600 -i 1 -q NODE1-IP NODE2-IP ...
+----
Your network is not ready for clustering if any of these test fails. Recheck
your network configuration. Especially switches are notorious for having
This is possible through the 'ring0_addr' and 'bindnet0_addr' parameter of
the 'pvecm create' command used for creating a new cluster.
-If you have setup a additional NIC with a static address on 10.10.10.1/25
+If you have setup an additional NIC with a static address on 10.10.10.1/25
and want to send and receive all cluster communication over this interface
you would execute:
[source,bash]
+----
pvecm create test --ring0_addr 10.10.10.1 --bindnet0_addr 10.10.10.0
+----
To check if everything is working properly execute:
[source,bash]
+----
systemctl status corosync
+----
+
+Afterwards, proceed as descripted in the section to
+<<adding-nodes-with-separated-cluster-network,add nodes with a separated cluster network>>.
[[separate-cluster-net-after-creation]]
Separate After Cluster Creation
hostnames ensure that they are resolvable from all nodes.
In my example I want to switch my cluster communication to the 10.10.10.1/25
-network. So I replace all 'ring0_addr' respectively. I also set the bindetaddr
+network. So I replace all 'ring0_addr' respectively. I also set the bindnetaddr
in the totem section of the config to an address of the new network. It can be
any address from the subnet configured on the new network interface.
On a single node execute:
[source,bash]
+----
systemctl restart corosync
+----
Now check if everything is fine:
[source,bash]
+----
systemctl status corosync
+----
If corosync runs again correct restart corosync also on all other nodes.
They will then join the cluster membership one by one on the new network.
+[[pvecm_rrp]]
Redundant Ring Protocol
~~~~~~~~~~~~~~~~~~~~~~~
To avoid a single point of failure you should implement counter measurements.
10.10.20.1/24 subnet you would execute:
[source,bash]
+----
pvecm create CLUSTERNAME -bindnet0_addr 10.10.10.1 -ring0_addr 10.10.10.1 \
-bindnet1_addr 10.10.20.1 -ring1_addr 10.10.20.1
+----
-RRP On A Created Cluster
+RRP On Existing Clusters
~~~~~~~~~~~~~~~~~~~~~~~~
-When enabling an already running cluster to use RRP you will take similar steps
-as describe in <<separate-cluster-net-after-creation,separating the cluster
-network>>. You just do it on another ring.
+You will take similar steps as described in
+<<separate-cluster-net-after-creation,separating the cluster network>> to
+enable RRP on an already running cluster. The single difference is, that you
+will add `ring1` and use it instead of `ring0`.
First add a new `interface` subsection in the `totem` section, set its
`ringnumber` property to `1`. Set the interfaces `bindnetaddr` property to an
----
-Bring it in effect like described in the <<edit-corosync-conf,edit the
-corosync.conf file>> section.
+Bring it in effect like described in the
+<<edit-corosync-conf,edit the corosync.conf file>> section.
This is a change which cannot take live in effect and needs at least a restart
of corosync. Recommended is a restart of the whole cluster.
Corosync Configuration
----------------------
-The `/ect/pve/corosync.conf` file plays a central role in {pve} cluster. It
+The `/etc/pve/corosync.conf` file plays a central role in {pve} cluster. It
controls the cluster member ship and its network.
For reading more about it check the corosync.conf man page:
[source,bash]
+----
man corosync.conf
+----
For node membership you should always use the `pvecm` tool provided by {pve}.
You may have to edit the configuration file manually for other changes.
avoid triggering some unwanted changes by an in between safe.
[source,bash]
+----
cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new
+----
Then open the Config file with your favorite editor, `nano` and `vim.tiny` are
preinstalled on {pve} for example.
apply or makes problems in other ways.
[source,bash]
+----
cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak
+----
Then move the new configuration file over the old one:
[source,bash]
+----
mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf
+----
You may check with the commands
[source,bash]
+----
systemctl status corosync
journalctl -b -u corosync
+----
If the change could applied automatically. If not you may have to restart the
corosync service via:
[source,bash]
+----
systemctl restart corosync
+----
On errors check the troubleshooting section below.
If you need to change '/etc/pve/corosync.conf' on an node with no quorum, and you
know what you do, use:
[source,bash]
+----
pvecm expected 1
+----
This sets the expected vote count to 1 and makes the cluster quorate. You can
now fix your configuration, or revert it back to the last working backup.
(``UPS'', also called ``battery backup'') to avoid this state, especially if
you want HA.
-On node startup, service `pve-manager` is started and waits for
+On node startup, the `pve-guests` service is started and waits for
quorum. Once quorate, it starts all guests which have the `onboot`
flag set.
mind that guest startup is delayed until you reach quorum.
+Guest Migration
+---------------
+
+Migrating virtual guests to other nodes is a useful feature in a
+cluster. There are settings to control the behavior of such
+migrations. This can be done via the configuration file
+`datacenter.cfg` or for a specific migration via API or command line
+parameters.
+
+It makes a difference if a Guest is online or offline, or if it has
+local resources (like a local disk).
+
+For Details about Virtual Machine Migration see the
+xref:qm_migration[QEMU/KVM Migration Chapter]
+
+For Details about Container Migration see the
+xref:pct_migration[Container Migration Chapter]
+
+Migration Type
+~~~~~~~~~~~~~~
+
+The migration type defines if the migration data should be sent over an
+encrypted (`secure`) channel or an unencrypted (`insecure`) one.
+Setting the migration type to insecure means that the RAM content of a
+virtual guest gets also transferred unencrypted, which can lead to
+information disclosure of critical data from inside the guest (for
+example passwords or encryption keys).
+
+Therefore, we strongly recommend using the secure channel if you do
+not have full control over the network and can not guarantee that no
+one is eavesdropping to it.
+
+NOTE: Storage migration does not follow this setting. Currently, it
+always sends the storage content over a secure channel.
+
+Encryption requires a lot of computing power, so this setting is often
+changed to "unsafe" to achieve better performance. The impact on
+modern systems is lower because they implement AES encryption in
+hardware. The performance impact is particularly evident in fast
+networks where you can transfer 10 Gbps or more.
+
+
+Migration Network
+~~~~~~~~~~~~~~~~~
+
+By default, {pve} uses the network in which cluster communication
+takes place to send the migration traffic. This is not optimal because
+sensitive cluster traffic can be disrupted and this network may not
+have the best bandwidth available on the node.
+
+Setting the migration network parameter allows the use of a dedicated
+network for the entire migration traffic. In addition to the memory,
+this also affects the storage traffic for offline migrations.
+
+The migration network is set as a network in the CIDR notation. This
+has the advantage that you do not have to set individual IP addresses
+for each node. {pve} can determine the real address on the
+destination node from the network specified in the CIDR form. To
+enable this, the network must be specified so that each node has one,
+but only one IP in the respective network.
+
+
+Example
+^^^^^^^
+
+We assume that we have a three-node setup with three separate
+networks. One for public communication with the Internet, one for
+cluster communication and a very fast one, which we want to use as a
+dedicated network for migration.
+
+A network configuration for such a setup might look as follows:
+
+----
+iface eno1 inet manual
+
+# public network
+auto vmbr0
+iface vmbr0 inet static
+ address 192.X.Y.57
+ netmask 255.255.250.0
+ gateway 192.X.Y.1
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+# cluster network
+auto eno2
+iface eno2 inet static
+ address 10.1.1.1
+ netmask 255.255.255.0
+
+# fast network
+auto eno3
+iface eno3 inet static
+ address 10.1.2.1
+ netmask 255.255.255.0
+----
+
+Here, we will use the network 10.1.2.0/24 as a migration network. For
+a single migration, you can do this using the `migration_network`
+parameter of the command line tool:
+
+----
+# qm migrate 106 tre --online --migration_network 10.1.2.0/24
+----
+
+To configure this as the default network for all migrations in the
+cluster, set the `migration` property of the `/etc/pve/datacenter.cfg`
+file:
+
+----
+# use dedicated migration network
+migration: secure,network=10.1.2.0/24
+----
+
+NOTE: The migration type must always be set when the migration network
+gets set in `/etc/pve/datacenter.cfg`.
+
+
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]
projects / pve-docs.git / blobdiff