install gen-vzdump.conf.5-opts.pl
[pve-docs.git] / ha-manager.adoc
CommitLineData
22653ac8
DM
1[[chapter-ha-manager]]
2ifdef::manvolnum[]
3PVE({manvolnum})
4================
5include::attributes.txt[]
6
7NAME
8----
9
734404b4 10ha-manager - Proxmox VE HA Manager
22653ac8
DM
11
12SYNOPSYS
13--------
14
15include::ha-manager.1-synopsis.adoc[]
16
17DESCRIPTION
18-----------
19endif::manvolnum[]
20
21ifndef::manvolnum[]
22High Availability
23=================
24include::attributes.txt[]
25endif::manvolnum[]
26
b5266e9f
DM
27
28Our modern society depends heavily on information provided by
29computers over the network. Mobile devices amplified that dependency,
30because people can access the network any time from anywhere. If you
31provide such services, it is very important that they are available
32most of the time.
33
34We can mathematically define the availability as the ratio of (A) the
35total time a service is capable of being used during a given interval
36to (B) the length of the interval. It is normally expressed as a
37percentage of uptime in a given year.
38
39.Availability - Downtime per Year
40[width="60%",cols="<d,d",options="header"]
41|===========================================================
42|Availability % |Downtime per year
43|99 |3.65 days
44|99.9 |8.76 hours
45|99.99 |52.56 minutes
46|99.999 |5.26 minutes
47|99.9999 |31.5 seconds
48|99.99999 |3.15 seconds
49|===========================================================
50
04bde502
DM
51There are several ways to increase availability. The most elegant
52solution is to rewrite your software, so that you can run it on
53several host at the same time. The software itself need to have a way
54to detect erors and do failover. This is relatively easy if you just
55want to serve read-only web pages. But in general this is complex, and
56sometimes impossible because you cannot modify the software
57yourself. The following solutions works without modifying the
58software:
59
60* Use reliable "server" components
61
62NOTE: Computer components with same functionality can have varying
63reliability numbers, depending on the component quality. Most verdors
64sell components with higher reliability as "server" components -
65usually at higher price.
b5266e9f
DM
66
67* Eliminate single point of failure (redundant components)
68
69 - use an uniteruptable power supply (UPS)
70 - use redundant power supplies on the main boards
71 - use ECC-RAM
72 - use redundant network hardware
04bde502
DM
73 - use RAID for local storage
74 - use distributed, redundant storage for VM data
b5266e9f
DM
75
76* Reduce downtime
77
04bde502
DM
78 - rapidly accessible adminstrators (24/7)
79 - availability of spare parts (other nodes is a {pve} cluster)
80 - automatic error detection ('ha-manager')
81 - automatic failover ('ha-manager')
b5266e9f
DM
82
83Virtualization environments like {pve} makes it much easier to reach
04bde502
DM
84high availability because they remove the "hardware" dependency. They
85also support to setup and use redundant storage and network
86devices. So if one host fail, you can simply start those services on
43da8322
DM
87another host within your cluster.
88
89Even better, {pve} provides a software stack called 'ha-manager',
90which can do that automatically for you. It is able to automatically
91detect errors and do automatic failover.
92
93{pve} 'ha-manager' works like an "automated" administrator. First, you
94configure what resources (VMs, containers, ...) it should
95manage. 'ha-manager' then observes correct functionality, and handles
96service failover to another node in case of errors. 'ha-manager' can
97also handle normal user requests which may start, stop, relocate and
98migrate a service.
04bde502
DM
99
100But high availability comes at a price. High quality components are
101more expensive, and making them redundant duplicates the costs at
102least. Additional spare parts increase costs further. So you should
103carefully calculate the benefits, and compare with those additional
104costs.
105
106TIP: Increasing availability from 99% to 99.9% is relatively
107simply. But increasing availability from 99.9999% to 99.99999% is very
43da8322
DM
108hard and costly. 'ha-manager' has typical error detection and failover
109times of about 2 minutes, so you can get no more than 99.999%
110availability.
b5266e9f 111
5bd515d4
DM
112Requirements
113------------
3810ae1e 114
5bd515d4 115* at least three cluster nodes (to get reliable quorum)
43da8322 116
5bd515d4 117* shared storage for VMs and containers
43da8322 118
5bd515d4 119* hardware redundancy (everywhere)
3810ae1e 120
5bd515d4
DM
121* hardware watchdog - if not available we fall back to the
122 linux kernel software watchdog ('softdog')
3810ae1e 123
5bd515d4 124* optional hardware fencing devices
3810ae1e 125
3810ae1e 126
5bd515d4
DM
127Resources
128---------
129
130We call the primary management unit handled by 'ha-manager' a
131resource. A resource (also called "service") is uniquely
132identified by a service ID (SID), which consists of the resource type
133and an type specific ID, e.g.: 'vm:100'. That example would be a
134resource of type 'vm' (virtual machine) with the ID 100.
135
136For now we have two important resources types - virtual machines and
137containers. One basic idea here is that we can bundle related software
138into such VM or container, so there is no need to compose one big
139service from other services, like it was done with 'rgmanager'. In
140general, a HA enabled resource should not depend on other resources.
3810ae1e 141
22653ac8 142
2b52e195 143How It Works
22653ac8
DM
144------------
145
3810ae1e
TL
146This section provides an in detail description of the {PVE} HA-manager
147internals. It describes how the CRM and the LRM work together.
148
149To provide High Availability two daemons run on each node:
150
151'pve-ha-lrm'::
152
153The local resource manager (LRM), it controls the services running on
154the local node.
155It reads the requested states for its services from the current manager
156status file and executes the respective commands.
157
158'pve-ha-crm'::
159
160The cluster resource manager (CRM), it controls the cluster wide
161actions of the services, processes the LRM result includes the state
162machine which controls the state of each service.
163
164.Locks in the LRM & CRM
165[NOTE]
166Locks are provided by our distributed configuration file system (pmxcfs).
167They are used to guarantee that each LRM is active and working as a
168LRM only executes actions when he has its lock we can mark a failed node
169as fenced if we get its lock. This lets us then recover the failed HA services
170securely without the failed (but maybe still running) LRM interfering.
171This all gets supervised by the CRM which holds currently the manager master
172lock.
173
174Local Resource Manager
175~~~~~~~~~~~~~~~~~~~~~~
176
22653ac8 177The local resource manager ('pve-ha-lrm') is started as a daemon on
3810ae1e
TL
178boot and waits until the HA cluster is quorate and thus cluster wide
179locks are working.
180
181It can be in three states:
182
183* *wait for agent lock*: the LRM waits for our exclusive lock. This is
184 also used as idle sate if no service is configured
185* *active*: the LRM holds its exclusive lock and has services configured
186* *lost agent lock*: the LRM lost its lock, this means a failure happened
187 and quorum was lost.
188
189After the LRM gets in the active state it reads the manager status
190file in '/etc/pve/ha/manager_status' and determines the commands it
191has to execute for the service it owns.
192For each command a worker gets started, this workers are running in
193parallel and are limited to maximal 4 by default. This default setting
194may be changed through the datacenter configuration key "max_worker".
195
196.Maximal Concurrent Worker Adjustment Tips
197[NOTE]
198The default value of 4 maximal concurrent Workers may be unsuited for
199a specific setup. For example may 4 live migrations happen at the same
200time, which can lead to network congestions with slower networks and/or
201big (memory wise) services. Ensure that also in the worst case no congestion
202happens and lower the "max_worker" value if needed. In the contrary, if you
203have a particularly powerful high end setup you may also want to increase it.
204
205Each command requested by the CRM is uniquely identifiable by an UID, when
206the worker finished its result will be processed and written in the LRM
207status file '/etc/pve/nodes/<nodename>/lrm_status'. There the CRM may collect
208it and let its state machine - respective the commands output - act on it.
209
210The actions on each service between CRM and LRM are normally always synced.
211This means that the CRM requests a state uniquely marked by an UID, the LRM
212then executes this action *one time* and writes back the result, also
213identifiable by the same UID. This is needed so that the LRM does not
214executes an outdated command.
215With the exception of the 'stop' and the 'error' command,
216those two do not depend on the result produce and are executed
217always in the case of the stopped state and once in the case of
218the error state.
219
220.Read the Logs
221[NOTE]
222The HA Stack logs every action it makes. This helps to understand what
223and also why something happens in the cluster. Here its important to see
224what both daemons, the LRM and the CRM, did. You may use
225`journalctl -u pve-ha-lrm` on the node(s) where the service is and
226the same command for the pve-ha-crm on the node which is the current master.
227
228Cluster Resource Manager
229~~~~~~~~~~~~~~~~~~~~~~~~
22653ac8
DM
230
231The cluster resource manager ('pve-ha-crm') starts on each node and
232waits there for the manager lock, which can only be held by one node
233at a time. The node which successfully acquires the manager lock gets
3810ae1e
TL
234promoted to the CRM master.
235
236It can be in three states: TODO
237
238* *wait for agent lock*: the LRM waits for our exclusive lock. This is
239 also used as idle sate if no service is configured
240* *active*: the LRM holds its exclusive lock and has services configured
241* *lost agent lock*: the LRM lost its lock, this means a failure happened
242 and quorum was lost.
243
244It main task is to manage the services which are configured to be highly
245available and try to get always bring them in the wanted state, e.g.: a
246enabled service will be started if its not running, if it crashes it will
247be started again. Thus it dictates the LRM the wanted actions.
22653ac8
DM
248
249When an node leaves the cluster quorum, its state changes to unknown.
250If the current CRM then can secure the failed nodes lock, the services
251will be 'stolen' and restarted on another node.
252
253When a cluster member determines that it is no longer in the cluster
254quorum, the LRM waits for a new quorum to form. As long as there is no
255quorum the node cannot reset the watchdog. This will trigger a reboot
256after 60 seconds.
257
2b52e195 258Configuration
22653ac8
DM
259-------------
260
261The HA stack is well integrated int the Proxmox VE API2. So, for
262example, HA can be configured via 'ha-manager' or the PVE web
263interface, which both provide an easy to use tool.
264
265The resource configuration file can be located at
266'/etc/pve/ha/resources.cfg' and the group configuration file at
267'/etc/pve/ha/groups.cfg'. Use the provided tools to make changes,
268there shouldn't be any need to edit them manually.
269
3810ae1e
TL
270Node Power Status
271-----------------
272
273If a node needs maintenance you should migrate and or relocate all
274services which are required to run always on another node first.
275After that you can stop the LRM and CRM services. But note that the
276watchdog triggers if you stop it with active services.
277
278Fencing
279-------
280
281What Is Fencing
282~~~~~~~~~~~~~~~
283
284Fencing secures that on a node failure the dangerous node gets will be rendered
285unable to do any damage and that no resource runs twice when it gets recovered
286from the failed node.
287
288Configure Hardware Watchdog
289~~~~~~~~~~~~~~~~~~~~~~~~~~~
290By default all watchdog modules are blocked for security reasons as they are
291like a loaded gun if not correctly initialized.
292If you have a hardware watchdog available remove its module from the blacklist
293and restart 'the watchdog-mux' service.
294
295
2b52e195 296Groups
22653ac8
DM
297------
298
299A group is a collection of cluster nodes which a service may be bound to.
300
2b52e195 301Group Settings
22653ac8
DM
302~~~~~~~~~~~~~~
303
304nodes::
305
306list of group node members
307
308restricted::
309
310resources bound to this group may only run on nodes defined by the
311group. If no group node member is available the resource will be
312placed in the stopped state.
313
314nofailback::
315
316the resource won't automatically fail back when a more preferred node
317(re)joins the cluster.
318
319
2b52e195 320Recovery Policy
22653ac8
DM
321---------------
322
323There are two service recover policy settings which can be configured
324specific for each resource.
325
326max_restart::
327
328maximal number of tries to restart an failed service on the actual
329node. The default is set to one.
330
331max_relocate::
332
333maximal number of tries to relocate the service to a different node.
334A relocate only happens after the max_restart value is exceeded on the
335actual node. The default is set to one.
336
337Note that the relocate count state will only reset to zero when the
338service had at least one successful start. That means if a service is
339re-enabled without fixing the error only the restart policy gets
340repeated.
341
2b52e195 342Error Recovery
22653ac8
DM
343--------------
344
345If after all tries the service state could not be recovered it gets
346placed in an error state. In this state the service won't get touched
347by the HA stack anymore. To recover from this state you should follow
348these steps:
349
350* bring the resource back into an safe and consistent state (e.g:
351killing its process)
352
353* disable the ha resource to place it in an stopped state
354
355* fix the error which led to this failures
356
357* *after* you fixed all errors you may enable the service again
358
359
2b52e195 360Service Operations
22653ac8
DM
361------------------
362
363This are how the basic user-initiated service operations (via
364'ha-manager') work.
365
366enable::
367
368the service will be started by the LRM if not already running.
369
370disable::
371
372the service will be stopped by the LRM if running.
373
374migrate/relocate::
375
376the service will be relocated (live) to another node.
377
378remove::
379
380the service will be removed from the HA managed resource list. Its
381current state will not be touched.
382
383start/stop::
384
385start and stop commands can be issued to the resource specific tools
386(like 'qm' or 'pct'), they will forward the request to the
387'ha-manager' which then will execute the action and set the resulting
388service state (enabled, disabled).
389
390
2b52e195 391Service States
22653ac8
DM
392--------------
393
394stopped::
395
396Service is stopped (confirmed by LRM)
397
398request_stop::
399
400Service should be stopped. Waiting for confirmation from LRM.
401
402started::
403
404Service is active an LRM should start it ASAP if not already running.
405
406fence::
407
408Wait for node fencing (service node is not inside quorate cluster
409partition).
410
411freeze::
412
413Do not touch the service state. We use this state while we reboot a
414node, or when we restart the LRM daemon.
415
416migrate::
417
418Migrate service (live) to other node.
419
420error::
421
422Service disabled because of LRM errors. Needs manual intervention.
423
424
425ifdef::manvolnum[]
426include::pve-copyright.adoc[]
427endif::manvolnum[]
428