pmgcm.adoc

   1 [[chapter_pmgcm]]
   2 ifdef::manvolnum[]
   3 pmgcm(1)
   4 ========
   5 :pmg-toplevel:
   6
   7 NAME
   8 ----
   9
  10 pmgcm - Proxmox Mail Gateway Cluster Management Toolkit
  11
  12
  13 SYNOPSIS
  14 --------
  15
  16 include::pmgcm.1-synopsis.adoc[]
  17
  18
  19 DESCRIPTION
  20 -----------
  21 endif::manvolnum[]
  22 ifndef::manvolnum[]
  23 Cluster Management
  24 ==================
  25 :pmg-toplevel:
  26 endif::manvolnum[]
  27
  28 We are living in a world where email becomes more and more important -
  29 failures in email systems are just not acceptable. To meet these
  30 requirements we developed the Proxmox HA (High Availability) Cluster.
  31
  32 The {pmg} HA Cluster consists of a master and several slave nodes
  33 (minimum one node). Configuration is done on the master. Configuration
  34 and data is synchronized to all cluster nodes over a VPN tunnel. This
  35 provides the following advantages:
  36
  37 * centralized configuration management
  38
  39 * fully redundant data storage
  40
  41 * high availability
  42
  43 * high performance
  44
  45 We use a unique application level clustering scheme, which provides
  46 extremely good performance. Special considerations where taken to make
  47 management as easy as possible. Complete Cluster setup is done within
  48 minutes, and nodes automatically reintegrate after temporary failures
  49 without any operator interaction.
  50
  51 image::images/Proxmox_HA_cluster_final_1024.png[]
  52
  53
  54 Hardware requirements
  55 ---------------------
  56
  57 There are no special hardware requirements, although it is highly
  58 recommended to use fast and reliable server with redundant disks on
  59 all cluster nodes (Hardware RAID with BBU and write cache enabled).
  60
  61 The HA Cluster can also run in virtualized environments.
  62
  63
  64 Subscriptions
  65 -------------
  66
  67 Each host in a cluster has its own subscription. If you want support
  68 for a cluster, each cluster node needs to have a valid
  69 subscription. All nodes must have the same subscription level.
  70
  71
  72 Load balancing
  73 --------------
  74
  75 It is usually advisable to distribute mail traffic among all cluster
  76 nodes. Please note that this is not always required, because it is
  77 also reasonable to use only one node to handle SMTP traffic. The
  78 second node is used as quarantine host, and only provides the web
  79 interface to the user quarantine.
  80
  81 The normal mail delivery process looks up DNS Mail Exchange (`MX`)
  82 records to determine the destination host. A `MX` record tells the
  83 sending system where to deliver mail for a certain domain. It is also
  84 possible to have several `MX` records for a single domain, they can have
  85 different priorities. For example, our `MX` record looks like that:
  86
  87 ----
  88 # dig -t mx proxmox.com
  89
  90 ;; ANSWER SECTION:
  91 proxmox.com.            22879   IN      MX      10 mail.proxmox.com.
  92
  93 ;; ADDITIONAL SECTION:
  94 mail.proxmox.com.       22879   IN      A       213.129.239.114
  95 ----
  96
  97 Please notice that there is one single `MX` record for the Domain
  98 `proxmox.com`, pointing to `mail.proxmox.com`. The `dig` command
  99 automatically puts out the corresponding address record if it
 100 exists. In our case it points to `213.129.239.114`. The priority of
 101 our `MX` record is set to 10 (preferred default value).
 102
 103
 104 Hot standby with backup `MX` records
 105 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 106
 107 Many people do not want to install two redundant mail proxies, instead
 108 they use the mail proxy of their ISP as fallback. This is simply done
 109 by adding an additional `MX` Record with a lower priority (higher
 110 number). With the example above this looks like that:
 111
 112 ----
 113 proxmox.com.            22879   IN      MX      100 mail.provider.tld.
 114 ----
 115
 116 In such a setup, your provider must accept mails for your domain and
 117 forward them to you. Please note that this is not advisable, because
 118 spam detection needs to be done by the backup `MX` server as well, and
 119 external servers provided by ISPs usually don't.
 120
 121 However, you will never lose mails with such a setup, because the sending Mail
 122 Transport Agent (MTA) will simply deliver the mail to the backup
 123 server (mail.provider.tld) if the primary server (mail.proxmox.com) is
 124 not available.
 125
 126 NOTE: Any reasonable mail server retries mail delivery if the target
 127 server is not available, i.e. {pmg} stores mail and retries delivery
 128 for up to one week. So you will not lose mail if your mail server is
 129 down, even if you run a single server setup.
 130
 131
 132 Load balancing with `MX` records
 133 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 134
 135 Using your ISPs mail server is not always a good idea, because many
 136 ISPs do not use advanced spam prevention techniques, or do not filter
 137 SPAM at all. It is often better to run a second server yourself to
 138 avoid lower spam detection rates.
 139
 140 Anyways, it’s quite simple to set up a high performance load balanced
 141 mail cluster using `MX` records. You just need to define two `MX` records
 142 with the same priority. Here is a complete example to make it clearer.
 143
 144 First, you need to have at least 2 working {pmg} servers
 145 (mail1.example.com and mail2.example.com) configured as cluster (see
 146 section xref:pmg_cluster_administration[Cluster administration]
 147 below), each having its own IP address. Let us assume the following
 148 addresses (DNS address records):
 149
 150 ----
 151 mail1.example.com.       22879   IN      A       1.2.3.4
 152 mail2.example.com.       22879   IN      A       1.2.3.5
 153 ----
 154
 155 It is always a good idea to add reverse lookup entries (PTR
 156 records) for those hosts. Many email systems nowadays reject mails
 157 from hosts without valid PTR records.  Then you need to define your `MX`
 158 records:
 159
 160 ----
 161 example.com.            22879   IN      MX      10 mail1.example.com.
 162 example.com.            22879   IN      MX      10 mail2.example.com.
 163 ----
 164
 165 This is all you need. You will receive mails on both hosts, more or
 166 less load-balanced using round-robin scheduling. If one host fails the
 167 other one is used.
 168
 169
 170 Other ways
 171 ~~~~~~~~~~
 172
 173 Multiple address records
 174 ^^^^^^^^^^^^^^^^^^^^^^^^
 175
 176 Using several DNS `MX` records is sometimes clumsy if you have many
 177 domains. It is also possible to use one `MX` record per domain, but
 178 multiple address records:
 179
 180 ----
 181 example.com.            22879   IN      MX      10 mail.example.com.
 182 mail.example.com.       22879   IN      A       1.2.3.4
 183 mail.example.com.       22879   IN      A       1.2.3.5
 184 ----
 185
 186
 187 Using firewall features
 188 ^^^^^^^^^^^^^^^^^^^^^^^
 189
 190 Many firewalls can do some kind of RR-Scheduling (round-robin) when
 191 using DNAT. See your firewall manual for more details.
 192
 193
 194 [[pmg_cluster_administration]]
 195 Cluster administration
 196 ----------------------
 197
 198 Cluster administration can be done on the GUI or using the command
 199 line utility `pmgcm`. The CLI tool is a bit more verbose, so we suggest
 200 to use that if you run into problems.
 201
 202 NOTE: Always setup the IP configuration before adding a node to the
 203 cluster. IP address, network mask, gateway address and hostname can’t
 204 be changed later.
 205
 206 Creating a Cluster
 207 ~~~~~~~~~~~~~~~~~~
 208
 209 [thumbnail="pmg-gui-cluster-panel.png", big=1]
 210
 211 You can create a cluster from any existing {pmg} host. All data is
 212 preserved.
 213
 214 * make sure you have the right IP configuration
 215   (IP/MASK/GATEWAY/HOSTNAME), because you cannot change that later
 216
 217 * press the create button on the GUI, or run the cluster creation command:
 218 +
 219 ----
 220 pmgcm create
 221 ----
 222
 223 NOTE: The node where you run the cluster create command will be the
 224 'master' node.
 225
 226
 227 Show Cluster Status
 228 ~~~~~~~~~~~~~~~~~~~
 229
 230 The GUI shows the status of all cluster nodes, and it is also possible
 231 to use the command line tool:
 232
 233 ----
 234 pmgcm status
 235 --NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
 236 pmg5(1)              192.168.2.127   master A       1 day 21:18   0.30    80%    41%
 237 ----
 238
 239
 240 [[pmgcm_join]]
 241 Adding Cluster Nodes
 242 ~~~~~~~~~~~~~~~~~~~~
 243
 244 [thumbnail="pmg-gui-cluster-join.png", big=1]
 245
 246 When you add a new node to a cluster (using `join`) all data on that node is
 247 destroyed. The whole database is initialized with cluster data from
 248 the master.
 249
 250 * make sure you have the right IP configuration
 251
 252 * run the cluster join command (on the new node):
 253 +
 254 ----
 255 pmgcm join <master_ip>
 256 ----
 257
 258 You need to enter the root password of the master host when asked for
 259 a password. When joining a cluster using the GUI, you also need to
 260 enter the 'fingerprint' of the master node. You get that information
 261 by pressing the `Add` button on the master node.
 262
 263 CAUTION: Node initialization deletes all existing databases, stops and
 264 then restarts all services accessing the database. So do not add nodes
 265 which are already active and receive mails.
 266
 267 Also, joining a cluster can take several minutes, because the new node
 268 needs to synchronize all data from the master (although this is done
 269 in the background).
 270
 271 NOTE: If you join a new node, existing quarantined items from the other nodes are not synchronized to the new node.
 272
 273
 274 Deleting Nodes
 275 ~~~~~~~~~~~~~~
 276
 277 Please detach nodes from the cluster network before removing them
 278 from the cluster configuration. Then run the following command on
 279 the master node:
 280
 281 ----
 282 pmgcm delete <cid>
 283 ----
 284
 285 Parameter `<cid>` is the unique cluster node ID, as listed with `pmgcm status`.
 286
 287
 288 Disaster Recovery
 289 ~~~~~~~~~~~~~~~~~
 290
 291 It is highly recommended to use redundant disks on all cluster nodes
 292 (RAID). So in almost any circumstances you just need to replace the
 293 damaged hardware or disk. {pmg} uses an asynchronous
 294 clustering algorithm, so you just need to reboot the repaired node,
 295 and everything will work again transparently.
 296
 297 The following scenarios only apply when you really lose the contents
 298 of the hard disk.
 299
 300
 301 Single Node Failure
 302 ^^^^^^^^^^^^^^^^^^^
 303
 304 * delete failed node on master
 305 +
 306 ----
 307 pmgcm delete <cid>
 308 ----
 309
 310 * add (re-join) a new node
 311 +
 312 ----
 313 pmgcm join <master_ip>
 314 ----
 315
 316
 317 Master Failure
 318 ^^^^^^^^^^^^^^
 319
 320 * force another node to be master
 321 +
 322 -----
 323 pmgcm promote
 324 -----
 325
 326 * tell other nodes that master has changed
 327 +
 328 ----
 329 pmgcm sync --master_ip <master_ip>
 330 ----
 331
 332
 333 Total Cluster Failure
 334 ^^^^^^^^^^^^^^^^^^^^^
 335
 336 * restore backup (Cluster and node information is not restored, you
 337   have to recreate master and nodes)
 338
 339 * tell it to become master
 340 +
 341 ----
 342 pmgcm create
 343 ----
 344
 345 * install new nodes
 346
 347 * add those new nodes to the cluster
 348 +
 349 ----
 350 pmgcm join <master_ip>
 351 ----
 352
 353
 354 ifdef::manvolnum[]
 355 include::pmg-copyright.adoc[]
 356 endif::manvolnum[]