ha-manager: clarifications and various fixes
[pve-docs.git] / ha-manager.adoc
CommitLineData
22653ac8
DM
1[[chapter-ha-manager]]
2ifdef::manvolnum[]
3PVE({manvolnum})
4================
5include::attributes.txt[]
6
7NAME
8----
9
734404b4 10ha-manager - Proxmox VE HA Manager
22653ac8
DM
11
12SYNOPSYS
13--------
14
15include::ha-manager.1-synopsis.adoc[]
16
17DESCRIPTION
18-----------
19endif::manvolnum[]
20
21ifndef::manvolnum[]
22High Availability
23=================
24include::attributes.txt[]
25endif::manvolnum[]
26
b5266e9f
DM
27
28Our modern society depends heavily on information provided by
29computers over the network. Mobile devices amplified that dependency,
30because people can access the network any time from anywhere. If you
31provide such services, it is very important that they are available
32most of the time.
33
34We can mathematically define the availability as the ratio of (A) the
35total time a service is capable of being used during a given interval
36to (B) the length of the interval. It is normally expressed as a
37percentage of uptime in a given year.
38
39.Availability - Downtime per Year
40[width="60%",cols="<d,d",options="header"]
41|===========================================================
42|Availability % |Downtime per year
43|99 |3.65 days
44|99.9 |8.76 hours
45|99.99 |52.56 minutes
46|99.999 |5.26 minutes
47|99.9999 |31.5 seconds
48|99.99999 |3.15 seconds
49|===========================================================
50
04bde502
DM
51There are several ways to increase availability. The most elegant
52solution is to rewrite your software, so that you can run it on
53several host at the same time. The software itself need to have a way
2af6af05 54to detect errors and do failover. This is relatively easy if you just
04bde502
DM
55want to serve read-only web pages. But in general this is complex, and
56sometimes impossible because you cannot modify the software
57yourself. The following solutions works without modifying the
58software:
59
60* Use reliable "server" components
61
62NOTE: Computer components with same functionality can have varying
2af6af05 63reliability numbers, depending on the component quality. Most vendors
04bde502
DM
64sell components with higher reliability as "server" components -
65usually at higher price.
b5266e9f
DM
66
67* Eliminate single point of failure (redundant components)
68
2af6af05 69 - use an uninterruptible power supply (UPS)
b5266e9f
DM
70 - use redundant power supplies on the main boards
71 - use ECC-RAM
72 - use redundant network hardware
04bde502
DM
73 - use RAID for local storage
74 - use distributed, redundant storage for VM data
b5266e9f
DM
75
76* Reduce downtime
77
2af6af05
TL
78 - rapidly accessible administrators (24/7)
79 - availability of spare parts (other nodes in a {pve} cluster)
04bde502
DM
80 - automatic error detection ('ha-manager')
81 - automatic failover ('ha-manager')
b5266e9f 82
5771d9b0 83Virtualization environments like {pve} make it much easier to reach
04bde502
DM
84high availability because they remove the "hardware" dependency. They
85also support to setup and use redundant storage and network
86devices. So if one host fail, you can simply start those services on
43da8322
DM
87another host within your cluster.
88
89Even better, {pve} provides a software stack called 'ha-manager',
90which can do that automatically for you. It is able to automatically
91detect errors and do automatic failover.
92
93{pve} 'ha-manager' works like an "automated" administrator. First, you
94configure what resources (VMs, containers, ...) it should
95manage. 'ha-manager' then observes correct functionality, and handles
96service failover to another node in case of errors. 'ha-manager' can
97also handle normal user requests which may start, stop, relocate and
98migrate a service.
04bde502
DM
99
100But high availability comes at a price. High quality components are
101more expensive, and making them redundant duplicates the costs at
102least. Additional spare parts increase costs further. So you should
103carefully calculate the benefits, and compare with those additional
104costs.
105
106TIP: Increasing availability from 99% to 99.9% is relatively
107simply. But increasing availability from 99.9999% to 99.99999% is very
43da8322
DM
108hard and costly. 'ha-manager' has typical error detection and failover
109times of about 2 minutes, so you can get no more than 99.999%
110availability.
b5266e9f 111
5bd515d4
DM
112Requirements
113------------
3810ae1e 114
5bd515d4 115* at least three cluster nodes (to get reliable quorum)
43da8322 116
5bd515d4 117* shared storage for VMs and containers
43da8322 118
5bd515d4 119* hardware redundancy (everywhere)
3810ae1e 120
5bd515d4
DM
121* hardware watchdog - if not available we fall back to the
122 linux kernel software watchdog ('softdog')
3810ae1e 123
5bd515d4 124* optional hardware fencing devices
3810ae1e 125
3810ae1e 126
5bd515d4
DM
127Resources
128---------
129
130We call the primary management unit handled by 'ha-manager' a
131resource. A resource (also called "service") is uniquely
132identified by a service ID (SID), which consists of the resource type
133and an type specific ID, e.g.: 'vm:100'. That example would be a
134resource of type 'vm' (virtual machine) with the ID 100.
135
136For now we have two important resources types - virtual machines and
137containers. One basic idea here is that we can bundle related software
138into such VM or container, so there is no need to compose one big
139service from other services, like it was done with 'rgmanager'. In
140general, a HA enabled resource should not depend on other resources.
3810ae1e 141
22653ac8 142
2b52e195 143How It Works
22653ac8
DM
144------------
145
3810ae1e
TL
146This section provides an in detail description of the {PVE} HA-manager
147internals. It describes how the CRM and the LRM work together.
148
149To provide High Availability two daemons run on each node:
150
151'pve-ha-lrm'::
152
153The local resource manager (LRM), it controls the services running on
154the local node.
155It reads the requested states for its services from the current manager
156status file and executes the respective commands.
157
158'pve-ha-crm'::
159
160The cluster resource manager (CRM), it controls the cluster wide
2af6af05 161actions of the services, processes the LRM results and includes the state
3810ae1e
TL
162machine which controls the state of each service.
163
164.Locks in the LRM & CRM
165[NOTE]
166Locks are provided by our distributed configuration file system (pmxcfs).
5771d9b0
TL
167They are used to guarantee that each LRM is active once and working. As a
168LRM only executes actions when it holds its lock we can mark a failed node
169as fenced if we can acquire its lock. This lets us then recover any failed
170HA services securely without any interference from the now unknown failed Node.
3810ae1e
TL
171This all gets supervised by the CRM which holds currently the manager master
172lock.
173
174Local Resource Manager
175~~~~~~~~~~~~~~~~~~~~~~
176
22653ac8 177The local resource manager ('pve-ha-lrm') is started as a daemon on
3810ae1e
TL
178boot and waits until the HA cluster is quorate and thus cluster wide
179locks are working.
180
181It can be in three states:
182
183* *wait for agent lock*: the LRM waits for our exclusive lock. This is
184 also used as idle sate if no service is configured
185* *active*: the LRM holds its exclusive lock and has services configured
186* *lost agent lock*: the LRM lost its lock, this means a failure happened
187 and quorum was lost.
188
189After the LRM gets in the active state it reads the manager status
190file in '/etc/pve/ha/manager_status' and determines the commands it
2af6af05 191has to execute for the services it owns.
3810ae1e
TL
192For each command a worker gets started, this workers are running in
193parallel and are limited to maximal 4 by default. This default setting
194may be changed through the datacenter configuration key "max_worker".
2af6af05
TL
195When finished the worker process gets collected and its result saved for
196the CRM.
3810ae1e
TL
197
198.Maximal Concurrent Worker Adjustment Tips
199[NOTE]
200The default value of 4 maximal concurrent Workers may be unsuited for
201a specific setup. For example may 4 live migrations happen at the same
202time, which can lead to network congestions with slower networks and/or
203big (memory wise) services. Ensure that also in the worst case no congestion
204happens and lower the "max_worker" value if needed. In the contrary, if you
205have a particularly powerful high end setup you may also want to increase it.
206
207Each command requested by the CRM is uniquely identifiable by an UID, when
208the worker finished its result will be processed and written in the LRM
209status file '/etc/pve/nodes/<nodename>/lrm_status'. There the CRM may collect
210it and let its state machine - respective the commands output - act on it.
211
212The actions on each service between CRM and LRM are normally always synced.
213This means that the CRM requests a state uniquely marked by an UID, the LRM
214then executes this action *one time* and writes back the result, also
215identifiable by the same UID. This is needed so that the LRM does not
216executes an outdated command.
217With the exception of the 'stop' and the 'error' command,
c9aa5d47 218those two do not depend on the result produced and are executed
3810ae1e
TL
219always in the case of the stopped state and once in the case of
220the error state.
221
222.Read the Logs
223[NOTE]
224The HA Stack logs every action it makes. This helps to understand what
225and also why something happens in the cluster. Here its important to see
226what both daemons, the LRM and the CRM, did. You may use
227`journalctl -u pve-ha-lrm` on the node(s) where the service is and
228the same command for the pve-ha-crm on the node which is the current master.
229
230Cluster Resource Manager
231~~~~~~~~~~~~~~~~~~~~~~~~
22653ac8
DM
232
233The cluster resource manager ('pve-ha-crm') starts on each node and
234waits there for the manager lock, which can only be held by one node
235at a time. The node which successfully acquires the manager lock gets
3810ae1e
TL
236promoted to the CRM master.
237
2af6af05 238It can be in three states:
3810ae1e
TL
239
240* *wait for agent lock*: the LRM waits for our exclusive lock. This is
241 also used as idle sate if no service is configured
242* *active*: the LRM holds its exclusive lock and has services configured
243* *lost agent lock*: the LRM lost its lock, this means a failure happened
244 and quorum was lost.
245
246It main task is to manage the services which are configured to be highly
2af6af05 247available and try to always enforce them to the wanted state, e.g.: a
3810ae1e 248enabled service will be started if its not running, if it crashes it will
2af6af05 249be started again. Thus it dictates the LRM the actions it needs to execute.
22653ac8
DM
250
251When an node leaves the cluster quorum, its state changes to unknown.
252If the current CRM then can secure the failed nodes lock, the services
253will be 'stolen' and restarted on another node.
254
255When a cluster member determines that it is no longer in the cluster
256quorum, the LRM waits for a new quorum to form. As long as there is no
257quorum the node cannot reset the watchdog. This will trigger a reboot
2af6af05 258after the watchdog then times out, this happens after 60 seconds.
22653ac8 259
2b52e195 260Configuration
22653ac8
DM
261-------------
262
2af6af05 263The HA stack is well integrated in the Proxmox VE API2. So, for
22653ac8
DM
264example, HA can be configured via 'ha-manager' or the PVE web
265interface, which both provide an easy to use tool.
266
267The resource configuration file can be located at
268'/etc/pve/ha/resources.cfg' and the group configuration file at
269'/etc/pve/ha/groups.cfg'. Use the provided tools to make changes,
270there shouldn't be any need to edit them manually.
271
3810ae1e
TL
272Node Power Status
273-----------------
274
275If a node needs maintenance you should migrate and or relocate all
276services which are required to run always on another node first.
277After that you can stop the LRM and CRM services. But note that the
278watchdog triggers if you stop it with active services.
279
5771d9b0
TL
280Package Updates
281---------------
282
2af6af05 283When updating the ha-manager you should do one node after the other, never
5771d9b0
TL
284all at once for various reasons. First, while we test our software
285thoughtfully, a bug affecting your specific setup cannot totally be ruled out.
286Upgrading one node after the other and checking the functionality of each node
287after finishing the update helps to recover from an eventual problems, while
288updating all could render you in a broken cluster state and is generally not
289good practice.
290
291Also, the {pve} HA stack uses a request acknowledge protocol to perform
292actions between the cluster and the local resource manager. For restarting,
293the LRM makes a request to the CRM to freeze all its services. This prevents
294that they get touched by the Cluster during the short time the LRM is restarting.
295After that the LRM may safely close the watchdog during a restart.
296Such a restart happens on a update and as already stated a active master
297CRM is needed to acknowledge the requests from the LRM, if this is not the case
298the update process can be too long which, in the worst case, may result in
299a watchdog reset.
300
2af6af05 301
3810ae1e
TL
302Fencing
303-------
304
305What Is Fencing
306~~~~~~~~~~~~~~~
307
308Fencing secures that on a node failure the dangerous node gets will be rendered
309unable to do any damage and that no resource runs twice when it gets recovered
5771d9b0
TL
310from the failed node. This is a really important task and one of the base
311principles to make a system Highly Available.
312
313If a node would not get fenced it would be in an unknown state where it may
314have still access to shared resources, this is really dangerous!
315Imagine that every network but the storage one broke, now while not
316reachable from the public network the VM still runs and writes on the shared
317storage. If we would not fence the node and just start up this VM on another
318Node we would get dangerous race conditions, atomicity violations the whole VM
319could be rendered unusable. The recovery could also simply fail if the storage
320protects from multiple mounts and thus defeat the purpose of HA.
321
322How {pve} Fences
323~~~~~~~~~~~~~~~~~
324
325There are different methods to fence a node, for example fence devices which
326cut off the power from the node or disable their communication completely.
327
328Those are often quite expensive and bring additional critical components in
329a system, because if they fail you cannot recover any service.
330
331We thus wanted to integrate a simpler method in the HA Manager first, namely
332self fencing with watchdogs.
333
334Watchdogs are widely used in critical and dependable systems since the
335beginning of micro controllers, they are often independent and simple
336integrated circuit which programs can use to watch them. After opening they need to
337report periodically. If, for whatever reason, a program becomes unable to do
338so the watchdogs triggers a reset of the whole server.
339
340Server motherboards often already include such hardware watchdogs, these need
341to be configured. If no watchdog is available or configured we fall back to the
342Linux Kernel softdog while still reliable it is not independent of the servers
343Hardware and thus has a lower reliability then a hardware watchdog.
3810ae1e
TL
344
345Configure Hardware Watchdog
346~~~~~~~~~~~~~~~~~~~~~~~~~~~
347By default all watchdog modules are blocked for security reasons as they are
348like a loaded gun if not correctly initialized.
c9aa5d47
TL
349If you have a hardware watchdog available remove its kernel module from the
350blacklist, load it with insmod and restart the 'watchdog-mux' service or reboot
351the node.
3810ae1e 352
2b52e195 353Groups
22653ac8
DM
354------
355
356A group is a collection of cluster nodes which a service may be bound to.
357
2b52e195 358Group Settings
22653ac8
DM
359~~~~~~~~~~~~~~
360
361nodes::
362
c9aa5d47
TL
363List of group node members where a priority can be given to each node.
364A service bound to this group will run on the nodes with the highest priority
365available. If more nodes are in the highest priority class the services will
366get distributed to those node if not already there. The priorities have a
367relative meaning only.
22653ac8
DM
368
369restricted::
370
371resources bound to this group may only run on nodes defined by the
372group. If no group node member is available the resource will be
373placed in the stopped state.
374
375nofailback::
376
377the resource won't automatically fail back when a more preferred node
378(re)joins the cluster.
379
380
2b52e195 381Recovery Policy
22653ac8
DM
382---------------
383
384There are two service recover policy settings which can be configured
385specific for each resource.
386
387max_restart::
388
389maximal number of tries to restart an failed service on the actual
390node. The default is set to one.
391
392max_relocate::
393
394maximal number of tries to relocate the service to a different node.
395A relocate only happens after the max_restart value is exceeded on the
396actual node. The default is set to one.
397
0abc65b0 398NOTE: The relocate count state will only reset to zero when the
22653ac8
DM
399service had at least one successful start. That means if a service is
400re-enabled without fixing the error only the restart policy gets
401repeated.
402
2b52e195 403Error Recovery
22653ac8
DM
404--------------
405
406If after all tries the service state could not be recovered it gets
407placed in an error state. In this state the service won't get touched
408by the HA stack anymore. To recover from this state you should follow
409these steps:
410
411* bring the resource back into an safe and consistent state (e.g:
412killing its process)
413
414* disable the ha resource to place it in an stopped state
415
416* fix the error which led to this failures
417
418* *after* you fixed all errors you may enable the service again
419
420
2b52e195 421Service Operations
22653ac8
DM
422------------------
423
424This are how the basic user-initiated service operations (via
425'ha-manager') work.
426
427enable::
428
429the service will be started by the LRM if not already running.
430
431disable::
432
433the service will be stopped by the LRM if running.
434
435migrate/relocate::
436
437the service will be relocated (live) to another node.
438
439remove::
440
441the service will be removed from the HA managed resource list. Its
442current state will not be touched.
443
444start/stop::
445
446start and stop commands can be issued to the resource specific tools
447(like 'qm' or 'pct'), they will forward the request to the
448'ha-manager' which then will execute the action and set the resulting
449service state (enabled, disabled).
450
451
2b52e195 452Service States
22653ac8
DM
453--------------
454
455stopped::
456
c9aa5d47
TL
457Service is stopped (confirmed by LRM), if detected running it will get stopped
458again.
22653ac8
DM
459
460request_stop::
461
462Service should be stopped. Waiting for confirmation from LRM.
463
464started::
465
466Service is active an LRM should start it ASAP if not already running.
c9aa5d47 467If the Service fails and is detected to be not running the LRM restarts it.
22653ac8
DM
468
469fence::
470
471Wait for node fencing (service node is not inside quorate cluster
472partition).
c9aa5d47
TL
473As soon as node gets fenced successfully the service will be recovered to
474another node, if possible.
22653ac8
DM
475
476freeze::
477
478Do not touch the service state. We use this state while we reboot a
479node, or when we restart the LRM daemon.
480
481migrate::
482
483Migrate service (live) to other node.
484
485error::
486
487Service disabled because of LRM errors. Needs manual intervention.
488
489
490ifdef::manvolnum[]
491include::pve-copyright.adoc[]
492endif::manvolnum[]
493