ifdef::manvolnum[]
-PVE({manvolnum})
-================
-include::attributes.txt[]
+pvecm(1)
+========
+:pve-toplevel:
NAME
----
pvecm - Proxmox VE Cluster Manager
-SYNOPSYS
+SYNOPSIS
--------
include::pvecm.1-synopsis.adoc[]
ifndef::manvolnum[]
Cluster Manager
===============
-include::attributes.txt[]
+:pve-toplevel:
endif::manvolnum[]
The {PVE} cluster manager `pvecm` is a tool to create a group of
use the 'ringX_addr' parameters to set the nodes address on those networks:
[source,bash]
+----
pvecm add IP-ADDRESS-CLUSTER -ring0_addr IP-ADDRESS-RING0
+----
If you want to use the Redundant Ring Protocol you will also want to pass the
'ring1_addr' parameter.
* then join it, as explained in the previous section.
+[[pvecm_separate_node_without_reinstall]]
Separate A Node Without Reinstalling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
the node from the cluster. A {pve} cluster cannot share the exact same
storage with another cluster, as it leads to VMID conflicts.
-Move the guests which you want to keep on this node now, after the removal you
-can do this only via backup and restore. Its suggested that you create a new
-storage where only the node which you want to separate has access. This can be
-an new export on your NFS or a new Ceph pool, to name a few examples. Its just
-important that the exact same storage does not gets accessed by multiple
-clusters. After setting this storage up move all data from the node and its VMs
-to it. Then you are ready to separate the node from the cluster.
+Its suggested that you create a new storage where only the node which you want
+to separate has access. This can be an new export on your NFS or a new Ceph
+pool, to name a few examples. Its just important that the exact same storage
+does not gets accessed by multiple clusters. After setting this storage up move
+all data from the node and its VMs to it. Then you are ready to separate the
+node from the cluster.
WARNING: Ensure all shared resources are cleanly separated! You will run into
conflicts and problems else.
First stop the corosync and the pve-cluster services on the node:
[source,bash]
+----
systemctl stop pve-cluster
systemctl stop corosync
+----
Start the cluster filesystem again in local mode:
[source,bash]
+----
pmxcfs -l
+----
Delete the corosync configuration files:
[source,bash]
+----
rm /etc/pve/corosync.conf
rm /etc/corosync/*
+----
You can now start the filesystem again as normal service:
[source,bash]
+----
killall pmxcfs
systemctl start pve-cluster
+----
The node is now separated from the cluster. You can deleted it from a remaining
node of the cluster with:
[source,bash]
+----
pvecm delnode oldnode
+----
If the command failed, because the remaining node in the cluster lost quorum
when the now separate node exited, you may set the expected votes to 1 as a workaround:
[source,bash]
+----
pvecm expected 1
+----
And the repeat the 'pvecm delnode' command.
cluster again without problems.
[source,bash]
+----
rm /var/lib/corosync/*
+----
As the configuration files from the other nodes are still in the cluster
filesystem you may want to clean those up too. Remove simply the whole
no multicast querier is active. This test has a duration of around 10
minutes.
[source,bash]
+----
omping -c 600 -i 1 -q NODE1-IP NODE2-IP ...
+----
Your network is not ready for clustering if any of these test fails. Recheck
your network configuration. Especially switches are notorious for having
you would execute:
[source,bash]
+----
pvecm create test --ring0_addr 10.10.10.1 --bindnet0_addr 10.10.10.0
+----
To check if everything is working properly execute:
[source,bash]
+----
systemctl status corosync
+----
[[separate-cluster-net-after-creation]]
Separate After Cluster Creation
On a single node execute:
[source,bash]
+----
systemctl restart corosync
+----
Now check if everything is fine:
[source,bash]
+----
systemctl status corosync
+----
If corosync runs again correct restart corosync also on all other nodes.
They will then join the cluster membership one by one on the new network.
10.10.20.1/24 subnet you would execute:
[source,bash]
+----
pvecm create CLUSTERNAME -bindnet0_addr 10.10.10.1 -ring0_addr 10.10.10.1 \
-bindnet1_addr 10.10.20.1 -ring1_addr 10.10.20.1
+----
RRP On A Created Cluster
~~~~~~~~~~~~~~~~~~~~~~~~
When enabling an already running cluster to use RRP you will take similar steps
-as describe in <<separate-cluster-net-after-creation,separating the cluster
-network>>. You just do it on another ring.
+as describe in
+<<separate-cluster-net-after-creation,separating the cluster network>>. You
+just do it on another ring.
First add a new `interface` subsection in the `totem` section, set its
`ringnumber` property to `1`. Set the interfaces `bindnetaddr` property to an
----
-Bring it in effect like described in the <<edit-corosync-conf,edit the
-corosync.conf file>> section.
+Bring it in effect like described in the
+<<edit-corosync-conf,edit the corosync.conf file>> section.
This is a change which cannot take live in effect and needs at least a restart
of corosync. Recommended is a restart of the whole cluster.
controls the cluster member ship and its network.
For reading more about it check the corosync.conf man page:
[source,bash]
+----
man corosync.conf
+----
For node membership you should always use the `pvecm` tool provided by {pve}.
You may have to edit the configuration file manually for other changes.
avoid triggering some unwanted changes by an in between safe.
[source,bash]
+----
cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new
+----
Then open the Config file with your favorite editor, `nano` and `vim.tiny` are
preinstalled on {pve} for example.
apply or makes problems in other ways.
[source,bash]
+----
cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak
+----
Then move the new configuration file over the old one:
[source,bash]
+----
mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf
+----
You may check with the commands
[source,bash]
+----
systemctl status corosync
journalctl -b -u corosync
+----
If the change could applied automatically. If not you may have to restart the
corosync service via:
[source,bash]
+----
systemctl restart corosync
+----
On errors check the troubleshooting section below.
If you need to change '/etc/pve/corosync.conf' on an node with no quorum, and you
know what you do, use:
[source,bash]
+----
pvecm expected 1
+----
This sets the expected vote count to 1 and makes the cluster quorate. You can
now fix your configuration, or revert it back to the last working backup.
mind that guest startup is delayed until you reach quorum.
+Guest Migration
+---------------
+
+Migrating virtual guests to other nodes is a useful feature in a
+cluster. There are settings to control the behavior of such
+migrations. This can be done via the configuration file
+`datacenter.cfg` or for a specific migration via API or command line
+parameters.
+
+
+Migration Type
+~~~~~~~~~~~~~~
+
+The migration type defines if the migration data should be sent over a
+encrypted ('secure') channel or an unencrypted ('insecure') one.
+Setting the migration type to insecure means that the RAM content of a
+Virtual Guest gets also transfered unencrypted, which can lead to
+information disclosure of critical data from inside the guest for
+example passwords or encryption keys.
+
+Therefore, we strongly recommend using the secure channel if you do
+not have full control over the network and can not guarantee that no
+one is eavesdropping to it.
+
+NOTE: Storage migration does not follow this setting. Currently, it
+always sends the storage content over a secure channel.
+
+Encryption requires a lot of computing power, so this setting is often
+changed to "unsafe" to achieve better performance. The impact on
+modern systems is lower because they implement AES encryption in
+hardware. But the influence is greater on fast networks, where you can
+transfer 10Gbps or more.
+
+
+Migration Network
+~~~~~~~~~~~~~~~~~
+
+By default {pve} uses the network where the cluster communication happens
+for sending the migration traffic. This is may be suboptimal, for one the
+sensible cluster traffic can be disturbed and on the other hand it may not
+have the best bandwidth available from all network interfaces on the node.
+
+Setting the migration network parameter allows using a dedicated network for
+sending all the migration traffic when migrating a guest system. This
+includes the traffic for offline storage migrations.
+
+The migration network is represented as a network in 'CIDR' notation. This
+has the advantage that you do not need to set a IP for each node, {pve} is
+able to figure out the real address from the given CIDR denoted network and
+the networks configured on the target node.
+To let this work the network must be specific enough, i.e. each node must
+have one and only one IP configured in the given network.
+
+Example
+^^^^^^^
+
+Lets assume that we have a three node setup with three networks, one for the
+public communication with the Internet, one for the cluster communication
+and one very fast one, which we want to use as an dedicated migration
+network. A network configuration for such a setup could look like:
+
+----
+iface eth0 inet manual
+
+# public network
+auto vmbr0
+iface vmbr0 inet static
+ address 192.X.Y.57
+ netmask 255.255.250.0
+ gateway 192.X.Y.1
+ bridge_ports eth0
+ bridge_stp off
+ bridge_fd 0
+
+# cluster network
+auto eth1
+iface eth1 inet static
+ address 10.1.1.1
+ netmask 255.255.255.0
+
+# fast network
+auto eth2
+iface eth2 inet static
+ address 10.1.2.1
+ netmask 255.255.255.0
+
+# [...]
+----
+
+Here we want to use the 10.1.2.1/24 network as migration network.
+For a single migration you can achieve this by using the 'migration_network'
+parameter:
+----
+# qm migrate 106 tre --online --migration_network 10.1.2.1/24
+----
+
+To set this up as default network for all migrations cluster wide you can use
+the migration property in '/etc/pve/datacenter.cfg':
+----
+# [...]
+migration: secure,network=10.1.2.1/24
+----
+
+Note that the migration type must be always set if the network gets set.
+
ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]