4 include::attributes.txt[]
9 pvecm - Proxmox VE Cluster Manager
14 include::pvecm.1-synopsis.adoc[]
23 include::attributes.txt[]
26 The {PVE} cluster manager `pvecm` is a tool to create a group of
27 physical servers. Such a group is called a *cluster*. We use the
28 http://www.corosync.org[Corosync Cluster Engine] for reliable group
29 communication, and such clusters can consist of up to 32 physical nodes
30 (probably more, dependent on network latency).
32 `pvecm` can be used to create a new cluster, join nodes to a cluster,
33 leave the cluster, get status information and do various other cluster
34 related tasks. The **P**rox**m**o**x** **C**luster **F**ile **S**ystem (``pmxcfs'')
35 is used to transparently distribute the cluster configuration to all cluster
38 Grouping nodes into a cluster has the following advantages:
40 * Centralized, web based management
42 * Multi-master clusters: each node can do all management task
44 * `pmxcfs`: database-driven file system for storing configuration files,
45 replicated in real-time on all nodes using `corosync`.
47 * Easy migration of virtual machines and containers between physical
52 * Cluster-wide services like firewall and HA
58 * All nodes must be in the same network as `corosync` uses IP Multicast
59 to communicate between nodes (also see
60 http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
61 ports 5404 and 5405 for cluster communication.
63 NOTE: Some switches do not support IP multicast by default and must be
64 manually enabled first.
66 * Date and time have to be synchronized.
68 * SSH tunnel on TCP port 22 between nodes is used.
70 * If you are interested in High Availability, you need to have at
71 least three nodes for reliable quorum. All nodes should have the
74 * We recommend a dedicated NIC for the cluster traffic, especially if
75 you use shared storage.
77 NOTE: It is not possible to mix Proxmox VE 3.x and earlier with
78 Proxmox VE 4.0 cluster nodes.
84 First, install {PVE} on all nodes. Make sure that each node is
85 installed with the final hostname and IP configuration. Changing the
86 hostname and IP is not possible after cluster creation.
88 Currently the cluster creation has to be done on the console, so you
89 need to login via `ssh`.
94 Login via `ssh` to the first {pve} node. Use a unique name for your cluster.
95 This name cannot be changed later.
97 hp1# pvecm create YOUR-CLUSTER-NAME
99 CAUTION: The cluster name is used to compute the default multicast
100 address. Please use unique cluster names if you run more than one
101 cluster inside your network.
103 To check the state of your cluster use:
108 Adding Nodes to the Cluster
109 ---------------------------
111 Login via `ssh` to the node you want to add.
113 hp2# pvecm add IP-ADDRESS-CLUSTER
115 For `IP-ADDRESS-CLUSTER` use the IP from an existing cluster node.
117 CAUTION: A new node cannot hold any VMs, because you would get
118 conflicts about identical VM IDs. Also, all existing configuration in
119 `/etc/pve` is overwritten when you join a new node to the cluster. To
120 workaround, use `vzdump` to backup and restore to a different VMID after
121 adding the node to the cluster.
123 To check the state of cluster:
127 .Cluster status after adding 4 nodes
132 Date: Mon Apr 20 12:30:13 2015
133 Quorum provider: corosync_votequorum
139 Votequorum information
140 ~~~~~~~~~~~~~~~~~~~~~~
147 Membership information
148 ~~~~~~~~~~~~~~~~~~~~~~
150 0x00000001 1 192.168.15.91
151 0x00000002 1 192.168.15.92 (local)
152 0x00000003 1 192.168.15.93
153 0x00000004 1 192.168.15.94
156 If you only want the list of all nodes use:
160 .List nodes in a cluster
164 Membership information
165 ~~~~~~~~~~~~~~~~~~~~~~
174 Remove a Cluster Node
175 ---------------------
177 CAUTION: Read carefully the procedure before proceeding, as it could
178 not be what you want or need.
180 Move all virtual machines from the node. Make sure you have no local
181 data or backups you want to keep, or save them accordingly.
183 Log in to one remaining node via ssh. Issue a `pvecm nodes` command to
184 identify the node ID:
191 Date: Mon Apr 20 12:30:13 2015
192 Quorum provider: corosync_votequorum
198 Votequorum information
199 ~~~~~~~~~~~~~~~~~~~~~~
206 Membership information
207 ~~~~~~~~~~~~~~~~~~~~~~
209 0x00000001 1 192.168.15.91 (local)
210 0x00000002 1 192.168.15.92
211 0x00000003 1 192.168.15.93
212 0x00000004 1 192.168.15.94
215 IMPORTANT: at this point you must power off the node to be removed and
216 make sure that it will not power on again (in the network) as it
222 Membership information
223 ~~~~~~~~~~~~~~~~~~~~~~
231 Log in to one remaining node via ssh. Issue the delete command (here
232 deleting node `hp4`):
234 hp1# pvecm delnode hp4
236 If the operation succeeds no output is returned, just check the node
237 list again with `pvecm nodes` or `pvecm status`. You should see
245 Date: Mon Apr 20 12:44:28 2015
246 Quorum provider: corosync_votequorum
252 Votequorum information
253 ~~~~~~~~~~~~~~~~~~~~~~
260 Membership information
261 ~~~~~~~~~~~~~~~~~~~~~~
263 0x00000001 1 192.168.15.90 (local)
264 0x00000002 1 192.168.15.91
265 0x00000003 1 192.168.15.92
268 IMPORTANT: as said above, it is very important to power off the node
269 *before* removal, and make sure that it will *never* power on again
270 (in the existing cluster network) as it is.
272 If you power on the node as it is, your cluster will be screwed up and
273 it could be difficult to restore a clean cluster state.
275 If, for whatever reason, you want that this server joins the same
276 cluster again, you have to
278 * reinstall {pve} on it from scratch
280 * then join it, as explained in the previous section.
282 Separate A Node Without Reinstalling
283 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
285 CAUTION: This is *not* the recommended method, proceed with caution. Use the
286 above mentioned method if you're unsure.
288 You can also separate a node from a cluster without reinstalling it from
289 scratch. But after removing the node from the cluster it will still have
290 access to the shared storages! This must be resolved before you start removing
291 the node from the cluster. A {pve} cluster cannot share the exact same
292 storage with another cluster, as it leads to VMID conflicts.
294 Move the guests which you want to keep on this node now, after the removal you
295 can do this only via backup and restore. Its suggested that you create a new
296 storage where only the node which you want to separate has access. This can be
297 an new export on your NFS or a new Ceph pool, to name a few examples. Its just
298 important that the exact same storage does not gets accessed by multiple
299 clusters. After setting this storage up move all data from the node and its VMs
300 to it. Then you are ready to separate the node from the cluster.
302 WARNING: Ensure all shared resources are cleanly separated! You will run into
303 conflicts and problems else.
305 First stop the corosync and the pve-cluster services on the node:
307 systemctl stop pve-cluster
308 systemctl stop corosync
310 Start the cluster filesystem again in local mode:
314 Delete the corosync configuration files:
316 rm /etc/pve/corosync.conf
319 You can now start the filesystem again as normal service:
322 systemctl start pve-cluster
324 The node is now separated from the cluster. You can deleted it from a remaining
325 node of the cluster with:
327 pvecm delnode oldnode
329 If the command failed, because the remaining node in the cluster lost quorum
330 when the now separate node exited, you may set the expected votes to 1 as a workaround:
334 And the repeat the 'pvecm delnode' command.
336 Now switch back to the separated node, here delete all remaining files left
337 from the old cluster. This ensures that the node can be added to another
338 cluster again without problems.
341 rm /var/lib/corosync/*
343 As the configuration files from the other nodes are still in the cluster
344 filesystem you may want to clean those up too. Remove simply the whole
345 directory recursive from '/etc/pve/nodes/NODENAME', but check three times that
346 you used the correct one before deleting it.
348 CAUTION: The nodes SSH keys are still in the 'authorized_key' file, this means
349 the nodes can still connect to each other with public key authentication. This
350 should be fixed by removing the respective keys from the
351 '/etc/pve/priv/authorized_keys' file.
356 {pve} use a quorum-based technique to provide a consistent state among
359 [quote, from Wikipedia, Quorum (distributed computing)]
361 A quorum is the minimum number of votes that a distributed transaction
362 has to obtain in order to be allowed to perform an operation in a
366 In case of network partitioning, state changes requires that a
367 majority of nodes are online. The cluster switches to read-only mode
370 NOTE: {pve} assigns a single vote to each node by default.
376 It is obvious that a cluster is not quorate when all nodes are
377 offline. This is a common case after a power failure.
379 NOTE: It is always a good idea to use an uninterruptible power supply
380 (``UPS'', also called ``battery backup'') to avoid this state, especially if
383 On node startup, service `pve-manager` is started and waits for
384 quorum. Once quorate, it starts all guests which have the `onboot`
387 When you turn on nodes, or when power comes back after power failure,
388 it is likely that some nodes boots faster than others. Please keep in
389 mind that guest startup is delayed until you reach quorum.
393 include::pve-copyright.adoc[]