10 pmgcm - Proxmox Mail Gateway Cluster Management Toolkit
16 include::pmgcm.1-synopsis.adoc[]
28 We are living in a world where email is becoming more and more important -
29 failures in email systems are not acceptable. To meet these
30 requirements, we developed the Proxmox HA (High Availability) Cluster.
32 The {pmg} HA Cluster consists of a master node and several slave nodes
33 (minimum one slave node). Configuration is done on the master,
34 and data is synchronized to all cluster nodes via a VPN tunnel. This
35 provides the following advantages:
37 * centralized configuration management
39 * fully redundant data storage
45 We use a unique application level clustering scheme, which provides
46 extremely good performance. Special considerations were taken to make
47 management as easy as possible. A complete cluster setup is done within
48 minutes, and nodes automatically reintegrate after temporary failures,
49 without any operator interaction.
51 image::images/Proxmox_HA_cluster_final_1024.png[]
57 There are no special hardware requirements, although it is highly
58 recommended to use fast and reliable server hardware, with redundant disks on
59 all cluster nodes (Hardware RAID with BBU and write cache enabled).
61 The HA Cluster can also run in virtualized environments.
67 Each node in a cluster has its own subscription. If you want support
68 for a cluster, each cluster node needs to have a valid
69 subscription. All nodes must have the same subscription level.
75 It is usually advisable to distribute mail traffic among all cluster
76 nodes. Please note that this is not always required, because it is
77 also reasonable to use only one node to handle SMTP traffic. The
78 second node can then be used as a quarantine host, that only provides the web
79 interface to the user quarantine.
81 The normal mail delivery process looks up DNS Mail Exchange (`MX`)
82 records to determine the destination host. An `MX` record tells the
83 sending system where to deliver mail for a certain domain. It is also
84 possible to have several `MX` records for a single domain, each of which can
85 have different priorities. For example, our `MX` record looks like this:
88 # dig -t mx proxmox.com
91 proxmox.com. 22879 IN MX 10 mail.proxmox.com.
93 ;; ADDITIONAL SECTION:
94 mail.proxmox.com. 22879 IN A 213.129.239.114
97 Notice that there is a single `MX` record for the domain
98 `proxmox.com`, pointing to `mail.proxmox.com`. The `dig` command
99 automatically outputs the corresponding address record, if it
100 exists. In our case it points to `213.129.239.114`. The priority of
101 our `MX` record is set to 10 (preferred default value).
104 Hot standby with backup `MX` records
105 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107 Many people do not want to install two redundant mail proxies. Instead
108 they use the mail proxy of their ISP as a fallback. This can be done
109 by adding an additional `MX` record with a lower priority (higher
110 number). Continuing from the example above, this would look like:
113 proxmox.com. 22879 IN MX 100 mail.provider.tld.
116 In such a setup, your provider must accept mails for your domain and
117 forward them to you. Please note that this is not advisable, because
118 spam detection needs to be done by the backup `MX` server as well, and
119 external servers provided by ISPs usually don't do this.
121 However, you will never lose mails with such a setup, because the sending Mail
122 Transport Agent (MTA) will simply deliver the mail to the backup
123 server (mail.provider.tld), if the primary server (mail.proxmox.com) is
126 NOTE: Any reasonable mail server retries mail delivery if the target
127 server is not available. {pmg} stores mail and retries delivery
128 for up to one week. Thus, you will not lose emails if your mail server is
129 down, even if you run a single server setup.
132 Load balancing with `MX` records
133 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
135 Using your ISP's mail server is not always a good idea, because many
136 ISPs do not use advanced spam prevention techniques, or do not filter
137 spam at all. It is often better to run a second server yourself to
138 avoid lower spam detection rates.
140 It’s quite simple to set up a high-performance, load-balanced
141 mail cluster using `MX` records. You just need to define two `MX`
142 records with the same priority. The rest of this section will provide
145 First, you need to have at least two working {pmg} servers
146 (mail1.example.com and mail2.example.com), configured as a cluster (see
147 section xref:pmg_cluster_administration[Cluster Administration]
148 below), with each having its own IP address. Let us assume the
149 following DNS address records:
152 mail1.example.com. 22879 IN A 1.2.3.4
153 mail2.example.com. 22879 IN A 1.2.3.5
156 It is always a good idea to add reverse lookup entries (PTR
157 records) for those hosts, as many email systems nowadays reject mails
158 from hosts without valid PTR records. Then you need to define your `MX`
162 example.com. 22879 IN MX 10 mail1.example.com.
163 example.com. 22879 IN MX 10 mail2.example.com.
166 This is all you need. Following this, you will receive mail on both
167 hosts, load-balanced using round-robin scheduling. If one host fails,
168 the other one is used.
174 Multiple address records
175 ^^^^^^^^^^^^^^^^^^^^^^^^
177 Using several DNS `MX` records can be tedious, if you have many
178 domains. It is also possible to use one `MX` record per domain, but
179 multiple address records:
182 example.com. 22879 IN MX 10 mail.example.com.
183 mail.example.com. 22879 IN A 1.2.3.4
184 mail.example.com. 22879 IN A 1.2.3.5
188 Using firewall features
189 ^^^^^^^^^^^^^^^^^^^^^^^
191 Many firewalls can do some kind of RR-Scheduling (round-robin) when
192 using DNAT. See your firewall manual for more details.
195 [[pmg_cluster_administration]]
196 Cluster Administration
197 ----------------------
199 Cluster administration can be done from the GUI or by using the command-line
200 utility `pmgcm`. The CLI tool is a bit more verbose, so we suggest
201 to use that if you run into any problems.
203 NOTE: Always set up the IP configuration, before adding a node to the
204 cluster. IP address, network mask, gateway address and hostname can’t
210 [thumbnail="pmg-gui-cluster-panel.png", big=1]
212 You can create a cluster from any existing {pmg} host. All data is
215 * make sure you have the right IP configuration
216 (IP/MASK/GATEWAY/HOSTNAME), because you cannot change that later
218 * press the create button on the GUI, or run the cluster creation command:
224 NOTE: The node where you run the cluster create command will be the
231 The GUI shows the status of all cluster nodes. You can also view this
232 using the command-line tool:
236 --NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
237 pmg5(1) 192.168.2.127 master A 1 day 21:18 0.30 80% 41%
245 [thumbnail="pmg-gui-cluster-join.png", big=1]
247 When you add a new node to a cluster (using `join`), all data on that node is
248 destroyed. The whole database is initialized with the cluster data from
251 * make sure you have the right IP configuration
253 * run the cluster join command (on the new node):
256 pmgcm join <master_ip>
259 You need to enter the root password of the master host, when asked for
260 a password. When joining a cluster using the GUI, you also need to
261 enter the 'fingerprint' of the master node. You can get this information
262 by pressing the `Add` button on the master node.
264 NOTE: Joining a cluster with two-factor authentication enabled for the `root`
265 user is not supported. Remove the second factor when joining the cluster.
267 CAUTION: Node initialization deletes all existing databases, stops all
268 services accessing the database and then restarts them. Therefore, do
269 not add nodes which are already active and receive mail.
271 Also note that joining a cluster can take several minutes, because the
272 new node needs to synchronize all data from the master (although this
273 is done in the background).
275 NOTE: If you join a new node, existing quarantined items from the
276 other nodes are not synchronized to the new node.
282 Please detach nodes from the cluster network, before removing them
283 from the cluster configuration. Only then you should run the following
284 command on the master node:
290 Parameter `<cid>` is the unique cluster node ID, as listed with `pmgcm status`.
296 It is highly recommended to use redundant disks on all cluster nodes
297 (RAID). So in almost any circumstance, you just need to replace the
298 damaged hardware or disk. {pmg} uses an asynchronous
299 clustering algorithm, so you just need to reboot the repaired node,
300 and everything will work again transparently.
302 The following scenarios only apply when you really lose the contents
309 * delete failed node on master
315 * add (re-join) a new node
318 pmgcm join <master_ip>
325 * force another node to be master
331 * tell other nodes that master has changed
334 pmgcm sync --master_ip <master_ip>
338 Total Cluster Failure
339 ^^^^^^^^^^^^^^^^^^^^^
341 * restore backup (Cluster and node information is not restored; you
342 have to recreate master and nodes)
344 * tell it to become master
352 * add those new nodes to the cluster
355 pmgcm join <master_ip>
360 include::pmg-copyright.adoc[]