ha-manager: error fixes and small additions
[pve-docs.git] / ha-manager.adoc
CommitLineData
22653ac8
DM
1[[chapter-ha-manager]]
2ifdef::manvolnum[]
3PVE({manvolnum})
4================
5include::attributes.txt[]
6
7NAME
8----
9
734404b4 10ha-manager - Proxmox VE HA Manager
22653ac8
DM
11
12SYNOPSYS
13--------
14
15include::ha-manager.1-synopsis.adoc[]
16
17DESCRIPTION
18-----------
19endif::manvolnum[]
20
21ifndef::manvolnum[]
22High Availability
23=================
24include::attributes.txt[]
25endif::manvolnum[]
26
b5266e9f
DM
27
28Our modern society depends heavily on information provided by
29computers over the network. Mobile devices amplified that dependency,
30because people can access the network any time from anywhere. If you
31provide such services, it is very important that they are available
32most of the time.
33
34We can mathematically define the availability as the ratio of (A) the
35total time a service is capable of being used during a given interval
36to (B) the length of the interval. It is normally expressed as a
37percentage of uptime in a given year.
38
39.Availability - Downtime per Year
40[width="60%",cols="<d,d",options="header"]
41|===========================================================
42|Availability % |Downtime per year
43|99 |3.65 days
44|99.9 |8.76 hours
45|99.99 |52.56 minutes
46|99.999 |5.26 minutes
47|99.9999 |31.5 seconds
48|99.99999 |3.15 seconds
49|===========================================================
50
04bde502
DM
51There are several ways to increase availability. The most elegant
52solution is to rewrite your software, so that you can run it on
53several host at the same time. The software itself need to have a way
2af6af05 54to detect errors and do failover. This is relatively easy if you just
04bde502
DM
55want to serve read-only web pages. But in general this is complex, and
56sometimes impossible because you cannot modify the software
57yourself. The following solutions works without modifying the
58software:
59
60* Use reliable "server" components
61
62NOTE: Computer components with same functionality can have varying
2af6af05 63reliability numbers, depending on the component quality. Most vendors
04bde502
DM
64sell components with higher reliability as "server" components -
65usually at higher price.
b5266e9f
DM
66
67* Eliminate single point of failure (redundant components)
68
2af6af05 69 - use an uninterruptible power supply (UPS)
b5266e9f
DM
70 - use redundant power supplies on the main boards
71 - use ECC-RAM
72 - use redundant network hardware
04bde502
DM
73 - use RAID for local storage
74 - use distributed, redundant storage for VM data
b5266e9f
DM
75
76* Reduce downtime
77
2af6af05
TL
78 - rapidly accessible administrators (24/7)
79 - availability of spare parts (other nodes in a {pve} cluster)
04bde502
DM
80 - automatic error detection ('ha-manager')
81 - automatic failover ('ha-manager')
b5266e9f
DM
82
83Virtualization environments like {pve} makes it much easier to reach
04bde502
DM
84high availability because they remove the "hardware" dependency. They
85also support to setup and use redundant storage and network
86devices. So if one host fail, you can simply start those services on
43da8322
DM
87another host within your cluster.
88
89Even better, {pve} provides a software stack called 'ha-manager',
90which can do that automatically for you. It is able to automatically
91detect errors and do automatic failover.
92
93{pve} 'ha-manager' works like an "automated" administrator. First, you
94configure what resources (VMs, containers, ...) it should
95manage. 'ha-manager' then observes correct functionality, and handles
96service failover to another node in case of errors. 'ha-manager' can
97also handle normal user requests which may start, stop, relocate and
98migrate a service.
04bde502
DM
99
100But high availability comes at a price. High quality components are
101more expensive, and making them redundant duplicates the costs at
102least. Additional spare parts increase costs further. So you should
103carefully calculate the benefits, and compare with those additional
104costs.
105
106TIP: Increasing availability from 99% to 99.9% is relatively
107simply. But increasing availability from 99.9999% to 99.99999% is very
43da8322
DM
108hard and costly. 'ha-manager' has typical error detection and failover
109times of about 2 minutes, so you can get no more than 99.999%
110availability.
b5266e9f 111
5bd515d4
DM
112Requirements
113------------
3810ae1e 114
5bd515d4 115* at least three cluster nodes (to get reliable quorum)
43da8322 116
5bd515d4 117* shared storage for VMs and containers
43da8322 118
5bd515d4 119* hardware redundancy (everywhere)
3810ae1e 120
5bd515d4
DM
121* hardware watchdog - if not available we fall back to the
122 linux kernel software watchdog ('softdog')
3810ae1e 123
5bd515d4 124* optional hardware fencing devices
3810ae1e 125
3810ae1e 126
5bd515d4
DM
127Resources
128---------
129
130We call the primary management unit handled by 'ha-manager' a
131resource. A resource (also called "service") is uniquely
132identified by a service ID (SID), which consists of the resource type
133and an type specific ID, e.g.: 'vm:100'. That example would be a
134resource of type 'vm' (virtual machine) with the ID 100.
135
136For now we have two important resources types - virtual machines and
137containers. One basic idea here is that we can bundle related software
138into such VM or container, so there is no need to compose one big
139service from other services, like it was done with 'rgmanager'. In
140general, a HA enabled resource should not depend on other resources.
3810ae1e 141
22653ac8 142
2b52e195 143How It Works
22653ac8
DM
144------------
145
3810ae1e
TL
146This section provides an in detail description of the {PVE} HA-manager
147internals. It describes how the CRM and the LRM work together.
148
149To provide High Availability two daemons run on each node:
150
151'pve-ha-lrm'::
152
153The local resource manager (LRM), it controls the services running on
154the local node.
155It reads the requested states for its services from the current manager
156status file and executes the respective commands.
157
158'pve-ha-crm'::
159
160The cluster resource manager (CRM), it controls the cluster wide
2af6af05 161actions of the services, processes the LRM results and includes the state
3810ae1e
TL
162machine which controls the state of each service.
163
164.Locks in the LRM & CRM
165[NOTE]
166Locks are provided by our distributed configuration file system (pmxcfs).
167They are used to guarantee that each LRM is active and working as a
168LRM only executes actions when he has its lock we can mark a failed node
169as fenced if we get its lock. This lets us then recover the failed HA services
170securely without the failed (but maybe still running) LRM interfering.
171This all gets supervised by the CRM which holds currently the manager master
172lock.
173
174Local Resource Manager
175~~~~~~~~~~~~~~~~~~~~~~
176
22653ac8 177The local resource manager ('pve-ha-lrm') is started as a daemon on
3810ae1e
TL
178boot and waits until the HA cluster is quorate and thus cluster wide
179locks are working.
180
181It can be in three states:
182
183* *wait for agent lock*: the LRM waits for our exclusive lock. This is
184 also used as idle sate if no service is configured
185* *active*: the LRM holds its exclusive lock and has services configured
186* *lost agent lock*: the LRM lost its lock, this means a failure happened
187 and quorum was lost.
188
189After the LRM gets in the active state it reads the manager status
190file in '/etc/pve/ha/manager_status' and determines the commands it
2af6af05 191has to execute for the services it owns.
3810ae1e
TL
192For each command a worker gets started, this workers are running in
193parallel and are limited to maximal 4 by default. This default setting
194may be changed through the datacenter configuration key "max_worker".
2af6af05
TL
195When finished the worker process gets collected and its result saved for
196the CRM.
3810ae1e
TL
197
198.Maximal Concurrent Worker Adjustment Tips
199[NOTE]
200The default value of 4 maximal concurrent Workers may be unsuited for
201a specific setup. For example may 4 live migrations happen at the same
202time, which can lead to network congestions with slower networks and/or
203big (memory wise) services. Ensure that also in the worst case no congestion
204happens and lower the "max_worker" value if needed. In the contrary, if you
205have a particularly powerful high end setup you may also want to increase it.
206
207Each command requested by the CRM is uniquely identifiable by an UID, when
208the worker finished its result will be processed and written in the LRM
209status file '/etc/pve/nodes/<nodename>/lrm_status'. There the CRM may collect
210it and let its state machine - respective the commands output - act on it.
211
212The actions on each service between CRM and LRM are normally always synced.
213This means that the CRM requests a state uniquely marked by an UID, the LRM
214then executes this action *one time* and writes back the result, also
215identifiable by the same UID. This is needed so that the LRM does not
216executes an outdated command.
217With the exception of the 'stop' and the 'error' command,
218those two do not depend on the result produce and are executed
219always in the case of the stopped state and once in the case of
220the error state.
221
222.Read the Logs
223[NOTE]
224The HA Stack logs every action it makes. This helps to understand what
225and also why something happens in the cluster. Here its important to see
226what both daemons, the LRM and the CRM, did. You may use
227`journalctl -u pve-ha-lrm` on the node(s) where the service is and
228the same command for the pve-ha-crm on the node which is the current master.
229
230Cluster Resource Manager
231~~~~~~~~~~~~~~~~~~~~~~~~
22653ac8
DM
232
233The cluster resource manager ('pve-ha-crm') starts on each node and
234waits there for the manager lock, which can only be held by one node
235at a time. The node which successfully acquires the manager lock gets
3810ae1e
TL
236promoted to the CRM master.
237
2af6af05 238It can be in three states:
3810ae1e
TL
239
240* *wait for agent lock*: the LRM waits for our exclusive lock. This is
241 also used as idle sate if no service is configured
242* *active*: the LRM holds its exclusive lock and has services configured
243* *lost agent lock*: the LRM lost its lock, this means a failure happened
244 and quorum was lost.
245
246It main task is to manage the services which are configured to be highly
2af6af05 247available and try to always enforce them to the wanted state, e.g.: a
3810ae1e 248enabled service will be started if its not running, if it crashes it will
2af6af05 249be started again. Thus it dictates the LRM the actions it needs to execute.
22653ac8
DM
250
251When an node leaves the cluster quorum, its state changes to unknown.
252If the current CRM then can secure the failed nodes lock, the services
253will be 'stolen' and restarted on another node.
254
255When a cluster member determines that it is no longer in the cluster
256quorum, the LRM waits for a new quorum to form. As long as there is no
257quorum the node cannot reset the watchdog. This will trigger a reboot
2af6af05 258after the watchdog then times out, this happens after 60 seconds.
22653ac8 259
2b52e195 260Configuration
22653ac8
DM
261-------------
262
2af6af05 263The HA stack is well integrated in the Proxmox VE API2. So, for
22653ac8
DM
264example, HA can be configured via 'ha-manager' or the PVE web
265interface, which both provide an easy to use tool.
266
267The resource configuration file can be located at
268'/etc/pve/ha/resources.cfg' and the group configuration file at
269'/etc/pve/ha/groups.cfg'. Use the provided tools to make changes,
270there shouldn't be any need to edit them manually.
271
3810ae1e
TL
272Node Power Status
273-----------------
274
275If a node needs maintenance you should migrate and or relocate all
276services which are required to run always on another node first.
277After that you can stop the LRM and CRM services. But note that the
278watchdog triggers if you stop it with active services.
279
2af6af05
TL
280Updates
281~~~~~~~
282When updating the ha-manager you should do one node after the other, never
283all at once. Further you have to ensure that no service located at the node
284is in the error state, a node with erroneous service is not able to be upgraded
285and if tried nonetheless it may even trigger a Node reset when doing so!
286When dealing with erroneous services first check what happened to them, then
287bring them in a secure state, after that disable or remove them from HA.
288Only after that you may start upgrading a Nodes LRM and CRM.
289
3810ae1e
TL
290Fencing
291-------
292
293What Is Fencing
294~~~~~~~~~~~~~~~
295
296Fencing secures that on a node failure the dangerous node gets will be rendered
297unable to do any damage and that no resource runs twice when it gets recovered
298from the failed node.
299
300Configure Hardware Watchdog
301~~~~~~~~~~~~~~~~~~~~~~~~~~~
302By default all watchdog modules are blocked for security reasons as they are
303like a loaded gun if not correctly initialized.
304If you have a hardware watchdog available remove its module from the blacklist
305and restart 'the watchdog-mux' service.
306
307
2b52e195 308Groups
22653ac8
DM
309------
310
311A group is a collection of cluster nodes which a service may be bound to.
312
2b52e195 313Group Settings
22653ac8
DM
314~~~~~~~~~~~~~~
315
316nodes::
317
318list of group node members
319
320restricted::
321
322resources bound to this group may only run on nodes defined by the
323group. If no group node member is available the resource will be
324placed in the stopped state.
325
326nofailback::
327
328the resource won't automatically fail back when a more preferred node
329(re)joins the cluster.
330
331
2b52e195 332Recovery Policy
22653ac8
DM
333---------------
334
335There are two service recover policy settings which can be configured
336specific for each resource.
337
338max_restart::
339
340maximal number of tries to restart an failed service on the actual
341node. The default is set to one.
342
343max_relocate::
344
345maximal number of tries to relocate the service to a different node.
346A relocate only happens after the max_restart value is exceeded on the
347actual node. The default is set to one.
348
0abc65b0 349NOTE: The relocate count state will only reset to zero when the
22653ac8
DM
350service had at least one successful start. That means if a service is
351re-enabled without fixing the error only the restart policy gets
352repeated.
353
2b52e195 354Error Recovery
22653ac8
DM
355--------------
356
357If after all tries the service state could not be recovered it gets
358placed in an error state. In this state the service won't get touched
359by the HA stack anymore. To recover from this state you should follow
360these steps:
361
362* bring the resource back into an safe and consistent state (e.g:
363killing its process)
364
365* disable the ha resource to place it in an stopped state
366
367* fix the error which led to this failures
368
369* *after* you fixed all errors you may enable the service again
370
371
2b52e195 372Service Operations
22653ac8
DM
373------------------
374
375This are how the basic user-initiated service operations (via
376'ha-manager') work.
377
378enable::
379
380the service will be started by the LRM if not already running.
381
382disable::
383
384the service will be stopped by the LRM if running.
385
386migrate/relocate::
387
388the service will be relocated (live) to another node.
389
390remove::
391
392the service will be removed from the HA managed resource list. Its
393current state will not be touched.
394
395start/stop::
396
397start and stop commands can be issued to the resource specific tools
398(like 'qm' or 'pct'), they will forward the request to the
399'ha-manager' which then will execute the action and set the resulting
400service state (enabled, disabled).
401
402
2b52e195 403Service States
22653ac8
DM
404--------------
405
406stopped::
407
408Service is stopped (confirmed by LRM)
409
410request_stop::
411
412Service should be stopped. Waiting for confirmation from LRM.
413
414started::
415
416Service is active an LRM should start it ASAP if not already running.
417
418fence::
419
420Wait for node fencing (service node is not inside quorate cluster
421partition).
422
423freeze::
424
425Do not touch the service state. We use this state while we reboot a
426node, or when we restart the LRM daemon.
427
428migrate::
429
430Migrate service (live) to other node.
431
432error::
433
434Service disabled because of LRM errors. Needs manual intervention.
435
436
437ifdef::manvolnum[]
438include::pve-copyright.adoc[]
439endif::manvolnum[]
440