notifications.adoc

   1 [[chapter_notifications]]
   2 Notifications
   3 =============
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 Overview
   9 --------
  10
  11 {pve} will send notifications if case of noteworthy events in the system.
  12
  13 There are a number of different xref:notification_events[notification events],
  14 each with their own set of metadata fields that can be used in
  15 notification matchers.
  16
  17 A xref:notification_matchers[notification matcher] determines
  18 _which_ notifications shall be sent _where_.
  19 A matcher has _match rules_, that can be used to
  20 match on certain notification properties (e.g. timestamp, severity,
  21 metadata fields).
  22 If a matcher matches a notification, the notification will be routed
  23 to a configurable set of notification targets.
  24
  25 A xref:notification_targets[notification target] is an abstraction for a
  26 destination where a notification should be sent to - for instance,
  27 a Gotify server instance, or a set of email addresses.
  28 There are multiple types of notification targets, including
  29 `sendmail`, which uses the system's sendmail command to send emails,
  30 or `gotify`, which sends a notification to a Gotify instance.
  31
  32 The notification system can be configured in the GUI under
  33 Datacenter -> Notifications. The configuration is stored in
  34 `/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
  35 the latter contains sensitive configuration options such as
  36 passwords or authentication tokens for notification targets.
  37
  38 [[notification_targets]]
  39 Notification Targets
  40 --------------------
  41
  42 [[notification_targets_sendmail]]
  43 Sendmail
  44 ~~~~~~~~
  45 The sendmail binary is a program commonly found on Unix-like operating systems
  46 that handles the sending of email messages.
  47 It is a command-line utility that allows users and applications to send emails
  48 directly from the command line or from within scripts.
  49
  50 The sendmail notification target uses the `sendmail` binary to send emails.
  51
  52
  53 NOTE: In standard {pve} installations, the `sendmail` binary is provided by
  54 Postfix. For this type of target to work correctly, it might be necessary to
  55 change Postfix's configuration so that it can correctly deliver emails.
  56 For cluster setups it is necessary to have a working Postfix configuration on
  57 every single cluster node.
  58
  59 The configuration for Sendmail target plugins has the following options:
  60
  61 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
  62 set multiple times to accomodate multiple recipients.
  63 * `mailto-user`: Users to which emails shall be sent to. The user's email
  64 address will be looked up in `users.cfg`. Can be set multiple times to
  65 accomodate multiple recipients.
  66 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
  67 * `from-address`: Sets the from address of the E-Mail. If the parameter is not
  68 set, the plugin will fall back to the `email_from` setting from
  69 `datacenter.cfg`. If that is also not set, the plugin will default to
  70 `root@$hostname`, where `$hostname` is the hostname of the node.
  71 * `comment`: Comment for this target
  72 The `From` header in the email will be set to `$author <$from-address>`.
  73
  74 Example configuration (`/etc/pve/notifications.cfg`):
  75 ----
  76 sendmail: example
  77         mailto-user root@pam
  78         mailto-user admin@pve
  79         mailto max@example.com
  80         from-address pve1@example.com
  81         comment Send to multiple users/addresses
  82 ----
  83
  84 [[notification_targets_smtp]]
  85 SMTP
  86 ~~~~
  87
  88 SMTP notification targets can send emails directly to an SMTP mail relay.
  89
  90 The configuration for SMTP target plugins has the following options:
  91
  92 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
  93 set multiple times to accomodate multiple recipients.
  94 * `mailto-user`: Users to which emails shall be sent to. The user's email
  95 address will be looked up in `users.cfg`. Can be set multiple times to
  96 accomodate multiple recipients.
  97 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
  98 * `from-address`: Sets the From-addresss of the email. SMTP relays might require
  99 that this address is owned by the user in order to avoid spoofing.
 100 The `From` header in the email will be set to `$author <$from-address>`.
 101 * `username`: Username to use during authentication. If no username is set,
 102 no authentication will be performed. The PLAIN and LOGIN authentication methods
 103 are supported.
 104 * `password`: Password to use when authenticating.
 105 * `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
 106 to `tls`.
 107 * `server`: Address/IP of the SMTP relay
 108 * `port`: The port to connect to. If not set, the used port
 109 defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
 110 value of `mode`.
 111 * `comment`: Comment for this target
 112
 113 Example configuration (`/etc/pve/notifications.cfg`):
 114 ----
 115 smtp: example
 116         mailto-user root@pam
 117         mailto-user admin@pve
 118         mailto max@example.com
 119         from-address pve1@example.com
 120         username pve1
 121         server mail.example.com
 122         mode starttls
 123 ----
 124 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
 125 secret token:
 126 ----
 127 smtp: example
 128         password somepassword
 129 ----
 130
 131 [[notification_targets_gotify]]
 132 Gotify
 133 ~~~~~~
 134
 135 http://gotify.net[Gotify] is an open-source self-hosted notification server that
 136 allows you to send and receive push notifications to various devices and
 137 applications. It provides a simple API and web interface, making it easy to
 138 integrate with different platforms and services.
 139
 140 The configuration for Gotify target plugins has the following options:
 141
 142 * `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
 143 * `token`: The authentication token. Tokens can be generated within the Gotify
 144 web interface.
 145 * `comment`: Comment for this target
 146
 147 NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
 148  xref:datacenter_configuration_file[datacenter configuration]
 149
 150 Example configuration (`/etc/pve/notifications.cfg`):
 151 ----
 152 gotify: example
 153         server http://gotify.example.com:8888
 154         comment Send to multiple users/addresses
 155 ----
 156
 157 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
 158 secret token:
 159 ----
 160 gotify: example
 161         token somesecrettoken
 162 ----
 163
 164 [[notification_matchers]]
 165 Notification Matchers
 166 ---------------------
 167 Notification matchers route notifications to notification targets based
 168 on their matching rules. These rules can match of certain properties of
 169 a notification, such as the timestamp (`match-calendar`), the severity of
 170 the notificaiton (`match-severity`) or metadata fiels (`match-field`).
 171 If a matcher matches a notification, all targets configured for the matcher
 172 will receive the notification.
 173
 174 An arbitrary number of matchers can be created, each with with their own
 175 matching rules and targets to notify.
 176 Every target is notified at most once for every notification, even if
 177 the target is used in multiple matchers.
 178
 179 A matcher without any matching rules is always true; the configured targets
 180 will always be notified.
 181 ----
 182 matcher: always-matches
 183         target admin
 184         comment This matcher always matches
 185 ----
 186
 187 Matcher Options
 188 ~~~~~~~~~~~~~~~
 189
 190 * `target`: Determine which target should be notified if the matcher matches.
 191 can be used multiple times to notify multiple targets.
 192 * `invert-match`: Inverts the result of the whole matcher
 193 * `mode`: Determines how the individual match rules are evaluated to compute
 194 the result for the whole matcher. If set to `all`, all matching rules must
 195 match. If set to `any`, at least one rule must match.
 196 a matcher must be true. Defaults to `all`.
 197 * `match-calendar`: Match the notification's timestamp against a schedule
 198 * `match-field`: Match the notification's metadata fields
 199 * `match-severity`: Match the notification's severity
 200 * `comment`: Comment for this matcher
 201
 202 [[notification_matchers_calendar]]
 203 Calendar Matching Rules
 204 ~~~~~~~~~~~~~~~~~~~~~~~
 205 A calendar matcher matches the time when a notification is sent agaist a
 206 configurable schedule.
 207
 208 * `match-calendar 8-12`
 209 * `match-calendar 8:00-15:30`
 210 * `match-calendar mon-fri 9:00-17:00`
 211 * `match-calendar sun,tue-wed,fri 9-17`
 212
 213 [[notification_matchers_field]]
 214 Field Matching Rules
 215 ~~~~~~~~~~~~~~~~~~~~
 216 Notifications have a selection of metadata fields that can be matched.
 217
 218 * `match-field exact:type=vzdump` Only match notifications about backups.
 219 * `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
 220 the node.
 221
 222 If a matched metadata field does not exist, the notification will not be
 223 matched.
 224 For instance, a `match-field regex:hostname=.*` directive will only match
 225 notifications that have an arbitraty `hostname` metadata field, but will
 226 not match if the field does not exist.
 227
 228 [[notification_matchers_severity]]
 229 Severity Matching Rules
 230 ~~~~~~~~~~~~~~~~~~~~~~~
 231 A notification has a associated severity that can be matched.
 232
 233 * `match-severity error`: Only match errors
 234 * `match-severity warning,error`: Match warnings and error
 235
 236 The following severities are in use:
 237 `info`, `notice`, `warning`, `error`, `unknown`.
 238
 239
 240 Examples
 241 ~~~~~~~~
 242 ----
 243 matcher: workday
 244         match-calendar mon-fri 9-17
 245         target admin
 246         comment Notify admins during working hours
 247
 248 matcher: night-and-weekend
 249         match-calendar mon-fri 9-17
 250         invert-match true
 251         target on-call-admins
 252         comment Separate target for non-working hours
 253 ----
 254
 255 ----
 256 matcher: backup-failures
 257         match-field exact:type=vzdump
 258         match-severity error
 259         target backup-admins
 260         comment Send notifications about backup failures to one group of admins
 261
 262 matcher: cluster-failures
 263         match-field exact:type=replication
 264         match-field exact:type=fencing
 265         mode any
 266         target cluster-admins
 267         comment Send cluster-related notifications to other group of admins
 268 ----
 269
 270 The last matcher could also be rewritten using a field matcher with a regular
 271 expression:
 272 ----
 273 matcher: cluster-failures
 274         match-field regex:type=^(replication|fencing)$
 275         target cluster-admins
 276         comment Send cluster-related notifications to other group of admins
 277 ----
 278
 279 [[notification_events]]
 280 Notification Events
 281 -------------------
 282
 283 [width="100%",options="header"]
 284 |===========================================================================
 285 | Event                        | `type`            | Severity | Metadata fields (in addition to `type`)
 286 | System updates available     |`package-updates`  | `info`   | `hostname`
 287 | Cluster node fenced          |`fencing`          | `error`  | `hostname`
 288 | Storage replication failed   |`replication`      | `error`  | -
 289 | Backup finished              |`vzdump`           | `info` (`error` on failure) | `hostname`
 290 | Mail for root                |`system-mail`      | `unknown`| -
 291 |===========================================================================
 292
 293 [width="100%",options="header"]
 294 |=======================================================================
 295 | Field name | Description
 296 | `type`     | Type of the notifcation
 297 | `hostname` | Hostname, including domain (e.g. `pve1.example.com`)
 298 |=======================================================================
 299
 300 System Mail Forwarding
 301 ---------------------
 302
 303 Certain local system daemons, such as `smartd`, generate notification emails
 304 that are initially directed to the local `root` user. {pve} will
 305 feed these mails into the notification system as a notification of
 306 type `system-mail` and with severity `unknown`.
 307
 308 When the forwarding process involves an email-based target
 309 (like `sendmail` or `smtp`), the email is forwarded exactly as received, with all
 310 original mail headers remaining intact. For all other targets,
 311 the system tries to extract both a subject line and the main text body
 312 from the email content. In instances where emails solely consist of HTML
 313 content, they will be transformed into plain text format during this process.
 314
 315 Permissions
 316 -----------
 317
 318 In order to modify/view the configuration for notification targets,
 319 the `Mapping.Modify/Mapping.Audit` permissions are required for the
 320 `/mapping/notifications` ACL node.
 321
 322 Testing a target requires `Mapping.Use`, `Mapping.Audit` or `Mapping.Modify`
 323 permissions on `/mapping/notifications`