10 pmgcm - Proxmox Mail Gateway Cluster Management Toolkit
16 include::pmgcm.1-synopsis.adoc[]
28 We are living in a world where email becomes more and more important -
29 failures in email systems are just not acceptable. To meet these
30 requirements we developed the Proxmox HA (High Availability) Cluster.
32 The {pmg} HA Cluster consists of a master and several slave nodes
33 (minimum one node). Configuration is done on the master. Configuration
34 and data is synchronized to all cluster nodes over a VPN tunnel. This
35 provides the following advantages:
37 * centralized configuration management
39 * fully redundant data storage
45 We use a unique application level clustering scheme, which provides
46 extremely good performance. Special considerations where taken to make
47 management as easy as possible. Complete Cluster setup is done within
48 minutes, and nodes automatically reintegrate after temporary failures
49 without any operator interaction.
51 image::images/Proxmox_HA_cluster_final_1024.png[]
57 There are no special hardware requirements, although it is highly
58 recommended to use fast and reliable server with redundant disks on
59 all cluster nodes (Hardware RAID with BBU and write cache enabled).
61 The HA Cluster can also run in virtualized environments.
67 Each host in a cluster has its own subscription. If you want support
68 for a cluster, each cluster node needs to have a valid
69 subscription. All nodes must have the same subscription level.
75 It is usually advisable to distribute mail traffic among all cluster
76 nodes. Please note that this is not always required, because it is
77 also reasonable to use only one node to handle SMTP traffic. The
78 second node is used as quarantine host, and only provides the web
79 interface to the user quarantine.
81 The normal mail delivery process looks up DNS Mail Exchange (`MX`)
82 records to determine the destination host. A `MX` record tells the
83 sending system where to deliver mail for a certain domain. It is also
84 possible to have several `MX` records for a single domain, they can have
85 different priorities. For example, our `MX` record looks like that:
88 # dig -t mx proxmox.com
91 proxmox.com. 22879 IN MX 10 mail.proxmox.com.
93 ;; ADDITIONAL SECTION:
94 mail.proxmox.com. 22879 IN A 213.129.239.114
97 Please notice that there is one single `MX` record for the Domain
98 `proxmox.com`, pointing to `mail.proxmox.com`. The `dig` command
99 automatically puts out the corresponding address record if it
100 exists. In our case it points to `213.129.239.114`. The priority of
101 our `MX` record is set to 10 (preferred default value).
104 Hot standby with backup `MX` records
105 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107 Many people do not want to install two redundant mail proxies, instead
108 they use the mail proxy of their ISP as fall-back. This is simply done
109 by adding an additional `MX` Record with a lower priority (higher
110 number). With the example above this looks like that:
113 proxmox.com. 22879 IN MX 100 mail.provider.tld.
116 Sure, your provider must accept mails for your domain and forward
117 received mails to you. Please note that such setup is not really
118 advisable, because spam detection needs to be done by that backup `MX`
119 server also, and external servers provided by ISPs usually don't do
122 You will never lose mails with such a setup, because the sending Mail
123 Transport Agent (MTA) will simply deliver the mail to the backup
124 server (mail.provider.tld) if the primary server (mail.proxmox.com) is
127 NOTE: Any resononable mail server retries mail devivery if the target
128 server is not available, i.e. {pmg} stores mail and retries delivery
129 for up to one week. So you will not loose mail if you mail server is
130 down, even if you run a single server setup.
133 Load balancing with `MX` records
134 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
136 Using your ISPs mail server is not always a good idea, because many
137 ISPs do not use advanced spam prevention techniques, or do not filter
138 SPAM at all. It is often better to run a second server yourself to
139 avoid lower spam detection rates.
141 Anyways, it’s quite simple to set up a high performance load balanced
142 mail cluster using `MX` records. You just need to define two `MX` records
143 with the same priority. I will explain this using a complete example
146 First, you need to have at least 2 working {pmg} servers
147 (mail1.example.com and mail2.example.com) configured as cluster (see
148 section xref:pmg_cluster_administration[Cluster administration]
149 below), each having its own IP address. Let us assume the following
150 addresses (DNS address records):
153 mail1.example.com. 22879 IN A 1.2.3.4
154 mail2.example.com. 22879 IN A 1.2.3.5
157 Btw, it is always a good idea to add reverse lookup entries (PTR
158 records) for those hosts. Many email systems nowadays reject mails
159 from hosts without valid PTR records. Then you need to define your `MX`
163 example.com. 22879 IN MX 10 mail1.example.com.
164 example.com. 22879 IN MX 10 mail2.example.com.
167 This is all you need. You will receive mails on both hosts, more or
168 less load-balanced using round-robin scheduling. If one host fails the
175 Multiple address records
176 ^^^^^^^^^^^^^^^^^^^^^^^^
178 Using several DNS `MX` record is sometime clumsy if you have many
179 domains. It is also possible to use one `MX` record per domain, but
180 multiple address records:
183 example.com. 22879 IN MX 10 mail.example.com.
184 mail.example.com. 22879 IN A 1.2.3.4
185 mail.example.com. 22879 IN A 1.2.3.5
189 Using firewall features
190 ^^^^^^^^^^^^^^^^^^^^^^^
192 Many firewalls can do some kind of RR-Scheduling (round-robin) when
193 using DNAT. See your firewall manual for more details.
196 [[pmg_cluster_administration]]
197 Cluster administration
198 ----------------------
200 Cluster administration can be done on the GUI or using the command
201 line utility `pmgcm`. The CLI tool is a bit more verbose, so we suggest
202 to use that if you run into problems.
204 NOTE: Always setup the IP configuration before adding a node to the
205 cluster. IP address, network mask, gateway address and hostname can’t
211 image::images/screenshot/pmg-gui-cluster-panel.png[]
213 You can create a cluster from any existing Proxmox host. All data is
216 * make sure you have the right IP configuration
217 (IP/MASK/GATEWAY/HOSTNAME), because you cannot change that later
219 * press the create button on the GUI, or run the cluster creation command:
225 NOTE: The node where you run the cluster create command will be the
232 The GUI shows the status of all cluster nodes, and it is also possible
233 to use the command line tool:
237 --NAME(CID)--------------IPADDRESS----ROLE-STATE---------UPTIME---LOAD----MEM---DISK
238 pmg5(1) 192.168.2.127 master A 1 day 21:18 0.30 80% 41%
246 image::images/screenshot/pmg-gui-cluster-join.png[]
248 When you add a new node to a cluster (join) all data on that node is
249 destroyed. The whole database is initialized with cluster data from
252 * make sure you have the right IP configuration
254 * run the cluster join command (on the new node):
257 pmgcm join <master_ip>
260 You need to enter the root password of the master host when asked for
261 a password. When joining a cluster using the GUI, you also need to
262 enter the 'fingerprint' of the master node. You get that information
263 by pressing the `Add` button on the master node.
265 CAUTION: Node initialization deletes all existing databases, stops and
266 then restarts all services accessing the database. So do not add nodes
267 which are already active and receive mails.
269 Also, joining a cluster can take several minutes, because the new node
270 needs to synchronize all data from the master (although this is done
273 NOTE: If you join a new node, existing quarantined items from the other nodes are not synchronized to the new node.
279 Please detach nodes from the cluster network before removing them
280 from the cluster configuration. Then run the following command on
287 Parameter `<cid>` is the unique cluster node ID, as listed with `pmgcm status`.
293 It is highly recommended to use redundant disks on all cluster nodes
294 (RAID). So in almost any circumstances you just need to replace the
295 damaged hardware or disk. {pmg} uses an asynchronous
296 clustering algorithm, so you just need to reboot the repaired node,
297 and everything will work again transparently.
299 The following scenarios only apply when you really loose the contents
306 * delete failed node on master
312 * add (re-join) a new node
315 pmgcm join <master_ip>
322 * force another node to be master
328 * tell other nodes that master has changed
331 pmgcm sync --master_ip <master_ip>
335 Total Cluster Failure
336 ^^^^^^^^^^^^^^^^^^^^^
338 * restore backup (Cluster and node information is not restored, you
339 have to recreate master and nodes)
341 * tell it to become master
349 * add those new nodes to the cluster
352 pmgcm join <master_ip>
357 include::pmg-copyright.adoc[]