1 [[chapter_notifications]]
11 {pve} will send notifications if case of noteworthy events in the system.
13 There are a number of different xref:notification_events[notification events],
14 each with their own set of metadata fields that can be used in
15 notification matchers.
17 A xref:notification_matchers[notification matcher] determines
18 _which_ notifications shall be sent _where_.
19 A matcher has _match rules_, that can be used to
20 match on certain notification properties (e.g. timestamp, severity,
22 If a matcher matches a notification, the notification will be routed
23 to a configurable set of notification targets.
25 A xref:notification_targets[notification target] is an abstraction for a
26 destination where a notification should be sent to - for instance,
27 a Gotify server instance, or a set of email addresses.
28 There are multiple types of notification targets, including
29 `sendmail`, which uses the system's sendmail command to send emails,
30 or `gotify`, which sends a notification to a Gotify instance.
32 The notification system can be configured in the GUI under
33 Datacenter -> Notifications. The configuration is stored in
34 `/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
35 the latter contains sensitive configuration options such as
36 passwords or authentication tokens for notification targets.
38 [[notification_targets]]
44 The sendmail binary is a program commonly found on Unix-like operating systems
45 that handles the sending of email messages.
46 It is a command-line utility that allows users and applications to send emails
47 directly from the command line or from within scripts.
49 The sendmail notification target uses the `sendmail` binary to send emails.
52 NOTE: In standard {pve} installations, the `sendmail` binary is provided by
53 Postfix. For this type of target to work correctly, it might be necessary to
54 change Postfix's configuration so that it can correctly deliver emails.
55 For cluster setups it is necessary to have a working Postfix configuration on
56 every single cluster node.
58 The configuration for Sendmail target plugins has the following options:
60 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
61 set multiple times to accomodate multiple recipients.
62 * `mailto-user`: Users to which emails shall be sent to. The user's email
63 address will be looked up in `users.cfg`. Can be set multiple times to
64 accomodate multiple recipients.
65 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
66 * `from-address`: Sets the from address of the E-Mail. If the parameter is not
67 set, the plugin will fall back to the `email_from` setting from
68 `datacenter.cfg`. If that is also not set, the plugin will default to
69 `root@$hostname`, where `$hostname` is the hostname of the node.
70 * `comment`: Comment for this target
71 The `From` header in the email will be set to `$author <$from-address>`.
73 Example configuration (`/etc/pve/notifications.cfg`):
78 mailto max@example.com
79 from-address pve1@example.com
80 comment Send to multiple users/addresses
86 SMTP notification targets can send emails directly to an SMTP mail relay.
88 The configuration for SMTP target plugins has the following options:
90 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
91 set multiple times to accomodate multiple recipients.
92 * `mailto-user`: Users to which emails shall be sent to. The user's email
93 address will be looked up in `users.cfg`. Can be set multiple times to
94 accomodate multiple recipients.
95 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
96 * `from-address`: Sets the From-addresss of the email. SMTP relays might require
97 that this address is owned by the user in order to avoid spoofing.
98 The `From` header in the email will be set to `$author <$from-address>`.
99 * `username`: Username to use during authentication. If no username is set,
100 no authentication will be performed. The PLAIN and LOGIN authentication methods
102 * `password`: Password to use when authenticating.
103 * `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
105 * `server`: Address/IP of the SMTP relay
106 * `port`: The port to connect to. If not set, the used port
107 defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
109 * `comment`: Comment for this target
111 Example configuration (`/etc/pve/notifications.cfg`):
115 mailto-user admin@pve
116 mailto max@example.com
117 from-address pve1@example.com
119 server mail.example.com
122 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
126 password somepassword
132 http://gotify.net[Gotify] is an open-source self-hosted notification server that
133 allows you to send and receive push notifications to various devices and
134 applications. It provides a simple API and web interface, making it easy to
135 integrate with different platforms and services.
137 The configuration for Gotify target plugins has the following options:
139 * `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
140 * `token`: The authentication token. Tokens can be generated within the Gotify
142 * `comment`: Comment for this target
144 NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
145 xref:datacenter_configuration_file[datacenter configuration]
147 Example configuration (`/etc/pve/notifications.cfg`):
150 server http://gotify.example.com:8888
151 comment Send to multiple users/addresses
154 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
158 token somesecrettoken
161 [[notification_matchers]]
162 Notification Matchers
163 ---------------------
164 Notification matchers route notifications to notification targets based
165 on their matching rules. These rules can match of certain properties of
166 a notification, such as the timestamp (`match-calendar`), the severity of
167 the notificaiton (`match-severity`) or metadata fiels (`match-field`).
168 If a matcher matches a notification, all targets configured for the matcher
169 will receive the notification.
171 An arbitrary number of matchers can be created, each with with their own
172 matching rules and targets to notify.
173 Every target is notified at most once for every notification, even if
174 the target is used in multiple matchers.
176 A matcher without any matching rules is always true; the configured targets
177 will always be notified.
179 matcher: always-matches
181 comment This matcher always matches
187 * `target`: Determine which target should be notified if the matcher matches.
188 can be used multiple times to notify multiple targets.
189 * `invert-match`: Inverts the result of the whole matcher
190 * `mode`: Determines how the individual match rules are evaluated to compute
191 the result for the whole matcher. If set to `all`, all matching rules must
192 match. If set to `any`, at least one rule must match.
193 a matcher must be true. Defaults to `all`.
194 * `match-calendar`: Match the notification's timestamp against a schedule
195 * `match-field`: Match the notification's metadata fields
196 * `match-severity`: Match the notification's severity
197 * `comment`: Comment for this matcher
199 Calendar Matching Rules
200 ~~~~~~~~~~~~~~~~~~~~~~~
201 A calendar matcher matches the time when a notification is sent agaist a
202 configurable schedule.
204 * `match-calendar 8-12`
205 * `match-calendar 8:00-15:30`
206 * `match-calendar mon-fri 9:00-17:00`
207 * `match-calendar sun,tue-wed,fri 9-17`
211 Notifications have a selection of metadata fields that can be matched.
213 * `match-field exact:type=vzdump` Only match notifications about backups.
214 * `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
217 If a matched metadata field does not exist, the notification will not be
219 For instance, a `match-field regex:hostname=.*` directive will only match
220 notifications that have an arbitraty `hostname` metadata field, but will
221 not match if the field does not exist.
223 Severity Matching Rules
224 ~~~~~~~~~~~~~~~~~~~~~~~
225 A notification has a associated severity that can be matched.
227 * `match-severity error`: Only match errors
228 * `match-severity warning,error`: Match warnings and error
230 The following severities are in use:
231 `info`, `notice`, `warning`, `error`.
238 match-calendar mon-fri 9-17
240 comment Notify admins during working hours
242 matcher: night-and-weekend
243 match-calendar mon-fri 9-17
245 target on-call-admins
246 comment Separate target for non-working hours
250 matcher: backup-failures
251 match-field exact:type=vzdump
254 comment Send notifications about backup failures to one group of admins
256 matcher: cluster-failures
257 match-field exact:type=replication
258 match-field exact:type=fencing
260 target cluster-admins
261 comment Send cluster-related notifications to other group of admins
264 The last matcher could also be rewritten using a field matcher with a regular
267 matcher: cluster-failures
268 match-field regex:type=^(replication|fencing)$
269 target cluster-admins
270 comment Send cluster-related notifications to other group of admins
273 [[notification_events]]
277 [width="100%",options="header"]
278 |===========================================================================
279 | Event | `type` | Severity | Metadata fields (in addition to `type`)
280 | System updates available |`package-updates` | `info` | `hostname`
281 | Cluster node fenced |`fencing` | `error` | `hostname`
282 | Storage replication failed |`replication` | `error` | -
283 | Backup finished |`vzdump` | `info` (`error` on failure) | `hostname`
284 | Mail for root |`system-mail` | `unknown`| -
285 |===========================================================================
287 [width="100%",options="header"]
288 |=======================================================================
289 | Field name | Description
290 | `type` | Type of the notifcation
291 | `hostname` | Hostname, including domain (e.g. `pve1.example.com`)
292 |=======================================================================
294 System Mail Forwarding
295 ---------------------
297 Certain local system daemons, such as `smartd`, generate notification emails
298 that are initially directed to the local `root` user. {pve} will
299 feed these mails into the notification system as a notification of
300 type `system-mail` and with severity `unknown`.
302 When the forwarding process involves an email-based target
303 (like `sendmail` or `smtp`), the email is forwarded exactly as received, with all
304 original mail headers remaining intact. For all other targets,
305 the system tries to extract both a subject line and the main text body
306 from the email content. In instances where emails solely consist of HTML
307 content, they will be transformed into plain text format during this process.
312 In order to modify/view the configuration for notification targets,
313 the `Mapping.Modify/Mapping.Audit` permissions are required for the
314 `/mapping/notifications` ACL node.
316 Testing a target requires `Mapping.Use`, `Mapping.Audit` or `Mapping.Modify`
317 permissions on `/mapping/notifications`