4 include::attributes.txt[]
9 pvecm - Proxmox VE Cluster Manager
14 include::pvecm.1-synopsis.adoc[]
23 include::attributes.txt[]
26 The {PVE} cluster manager 'pvecm' is a tool to create a group of
27 physical servers. Such group is called a *cluster*. We use the
28 http://www.corosync.org[Corosync Cluster Engine] for reliable group
29 communication, and such cluster can consists of up to 32 physical nodes
30 (probably more, dependent on network latency).
32 'pvecm' can be used to create a new cluster, join nodes to a cluster,
33 leave the cluster, get status information and do various other cluster
34 related tasks. The Proxmox Cluster file system (pmxcfs) is used to
35 transparently distribute the cluster configuration to all cluster
38 Grouping nodes into a cluster has the following advantages:
40 * Centralized, web based management
42 * Multi-master clusters: Each node can do all management task
44 * Proxmox Cluster file system (pmxcfs): Database-driven file system
45 for storing configuration files, replicated in real-time on all
48 * Easy migration of Virtual Machines and Containers between physical
53 * Cluster-wide services like firewall and HA
59 * All nodes must be in the same network as corosync uses IP Multicast
60 to communicate between nodes (also see
61 http://www.corosync.org[Corosync Cluster Engine]). Corosync uses UDP
62 ports 5404 and 5405 for cluster communication.
64 NOTE: Some switches do not support IP multicast by default and must be
65 manually enabled first.
67 * Date and time have to be synchronized.
69 * SSH tunnel on TCP port 22 between nodes is used.
71 * If you are interested in High Availability, you need to have at
72 least three nodes for reliable quorum. All nodes should have the
75 * We recommend a dedicated NIC for the cluster traffic, especially if
76 you use shared storage.
78 NOTE: It is not possible to mix Proxmox VE 3.x and earlier with
79 Proxmox VE 4.0 cluster nodes.
85 First, install {PVE} on all nodes. Make sure that each node is
86 installed with the final hostname and IP configuration. Changing the
87 hostname and IP is not possible after cluster creation.
89 Currently the cluster creation has to be done on the console, so you
90 need to login via 'ssh'.
95 Login via 'ssh' to the first Proxmox VE node. Use a unique name for
96 your cluster. This name cannot be changed later.
98 hp1# pvecm create YOUR-CLUSTER-NAME
100 CAUTION: The cluster name is used to compute the default multicast
101 address. Please use unique cluster names if you run more than one
102 cluster inside your network.
104 To check the state of your cluster use:
109 Adding Nodes to the Cluster
110 ---------------------------
112 Login via 'ssh' to the node you want to add.
114 hp2# pvecm add IP-ADDRESS-CLUSTER
116 For `IP-ADDRESS-CLUSTER` use the IP from an existing cluster node.
118 CAUTION: A new node cannot hold any VM´s, because you would get
119 conflicts about identical VM IDs. Also, all existing configuration in
120 '/etc/pve' is overwritten when you join a new node to the cluster. To
121 workaround, use vzdump to backup and restore to a different VMID after
122 adding the node to the cluster.
124 To check the state of cluster:
128 .Cluster status after adding 4 nodes
133 Date: Mon Apr 20 12:30:13 2015
134 Quorum provider: corosync_votequorum
140 Votequorum information
141 ~~~~~~~~~~~~~~~~~~~~~~
148 Membership information
149 ~~~~~~~~~~~~~~~~~~~~~~
151 0x00000001 1 192.168.15.91
152 0x00000002 1 192.168.15.92 (local)
153 0x00000003 1 192.168.15.93
154 0x00000004 1 192.168.15.94
157 If you only want the list of all nodes use:
161 .List Nodes in a Cluster
165 Membership information
166 ~~~~~~~~~~~~~~~~~~~~~~
175 Remove a Cluster Node
176 ---------------------
178 CAUTION: Read carefully the procedure before proceeding, as it could
179 not be what you want or need.
181 Move all virtual machines from the node. Make sure you have no local
182 data or backups you want to keep, or save them accordingly.
184 Log in to one remaining node via ssh. Issue a 'pvecm nodes' command to
185 identify the node ID:
192 Date: Mon Apr 20 12:30:13 2015
193 Quorum provider: corosync_votequorum
199 Votequorum information
200 ~~~~~~~~~~~~~~~~~~~~~~
207 Membership information
208 ~~~~~~~~~~~~~~~~~~~~~~
210 0x00000001 1 192.168.15.91 (local)
211 0x00000002 1 192.168.15.92
212 0x00000003 1 192.168.15.93
213 0x00000004 1 192.168.15.94
216 IMPORTANT: at this point you must power off the node to be removed and
217 make sure that it will not power on again (in the network) as it
223 Membership information
224 ~~~~~~~~~~~~~~~~~~~~~~
232 Log in to one remaining node via ssh. Issue the delete command (here
235 hp1# pvecm delnode hp4
237 If the operation succeeds no output is returned, just check the node
238 list again with 'pvecm nodes' or 'pvecm status'. You should see
246 Date: Mon Apr 20 12:44:28 2015
247 Quorum provider: corosync_votequorum
253 Votequorum information
254 ~~~~~~~~~~~~~~~~~~~~~~
261 Membership information
262 ~~~~~~~~~~~~~~~~~~~~~~
264 0x00000001 1 192.168.15.90 (local)
265 0x00000002 1 192.168.15.91
266 0x00000003 1 192.168.15.92
269 IMPORTANT: as said above, it is very important to power off the node
270 *before* removal, and make sure that it will *never* power on again
271 (in the existing cluster network) as it is.
273 If you power on the node as it is, your cluster will be screwed up and
274 it could be difficult to restore a clean cluster state.
276 If, for whatever reason, you want that this server joins the same
277 cluster again, you have to
279 * reinstall pve on it from scratch
281 * then join it, as explained in the previous section.
287 {pve} use a quorum-based technique to provide a consistent state among
290 [quote, from Wikipedia, Quorum (distributed computing)]
292 A quorum is the minimum number of votes that a distributed transaction
293 has to obtain in order to be allowed to perform an operation in a
297 In case of network partitioning, state changes requires that a
298 majority of nodes are online. The cluster switches to read-only mode
301 NOTE: {pve} assigns a single vote to each node by default.
307 It is obvious that a cluster is not quorate when all nodes are
308 offline. This is a common case after a power failure.
310 NOTE: It is always a good idea to use an uninterruptible power supply
311 ('UPS', also called 'battery backup') to avoid this state. Especially if
314 On node startup, service 'pve-manager' is started and waits for
315 quorum. Once quorate, it starts all guests which have the 'onboot'
318 When you turn on nodes, or when power comes back after power failure,
319 it is likely that some nodes boots faster than others. Please keep in
320 mind that guest startup is delayed until you reach quorum.
324 include::pve-copyright.adoc[]