notifications.adoc

   1 [[chapter_notifications]]
   2 Notifications
   3 =============
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 Overview
   9 --------
  10 [thumbnail="screenshot/gui-datacenter-notification-overview.png"]
  11
  12 {pve} will send notifications if case of noteworthy events in the system.
  13
  14 There are a number of different xref:notification_events[notification events],
  15 each with their own set of metadata fields that can be used in
  16 notification matchers.
  17
  18 A xref:notification_matchers[notification matcher] determines
  19 _which_ notifications shall be sent _where_.
  20 A matcher has _match rules_, that can be used to
  21 match on certain notification properties (e.g. timestamp, severity,
  22 metadata fields).
  23 If a matcher matches a notification, the notification will be routed
  24 to a configurable set of notification targets.
  25
  26 A xref:notification_targets[notification target] is an abstraction for a
  27 destination where a notification should be sent to - for instance,
  28 a Gotify server instance, or a set of email addresses.
  29 There are multiple types of notification targets, including
  30 `sendmail`, which uses the system's sendmail command to send emails,
  31 or `gotify`, which sends a notification to a Gotify instance.
  32
  33 The notification system can be configured in the GUI under
  34 Datacenter -> Notifications. The configuration is stored in
  35 `/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
  36 the latter contains sensitive configuration options such as
  37 passwords or authentication tokens for notification targets.
  38
  39 [[notification_targets]]
  40 Notification Targets
  41 --------------------
  42
  43 [[notification_targets_sendmail]]
  44 Sendmail
  45 ~~~~~~~~
  46 [thumbnail="screenshot/gui-datacenter-notification-sendmail.png"]
  47
  48 The sendmail binary is a program commonly found on Unix-like operating systems
  49 that handles the sending of email messages.
  50 It is a command-line utility that allows users and applications to send emails
  51 directly from the command line or from within scripts.
  52
  53 The sendmail notification target uses the `sendmail` binary to send emails.
  54
  55
  56 NOTE: In standard {pve} installations, the `sendmail` binary is provided by
  57 Postfix. For this type of target to work correctly, it might be necessary to
  58 change Postfix's configuration so that it can correctly deliver emails.
  59 For cluster setups it is necessary to have a working Postfix configuration on
  60 every single cluster node.
  61
  62 The configuration for Sendmail target plugins has the following options:
  63
  64 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
  65 set multiple times to accomodate multiple recipients.
  66 * `mailto-user`: Users to which emails shall be sent to. The user's email
  67 address will be looked up in `users.cfg`. Can be set multiple times to
  68 accomodate multiple recipients.
  69 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
  70 * `from-address`: Sets the from address of the E-Mail. If the parameter is not
  71 set, the plugin will fall back to the `email_from` setting from
  72 `datacenter.cfg`. If that is also not set, the plugin will default to
  73 `root@$hostname`, where `$hostname` is the hostname of the node.
  74 * `comment`: Comment for this target
  75 The `From` header in the email will be set to `$author <$from-address>`.
  76
  77 Example configuration (`/etc/pve/notifications.cfg`):
  78 ----
  79 sendmail: example
  80         mailto-user root@pam
  81         mailto-user admin@pve
  82         mailto max@example.com
  83         from-address pve1@example.com
  84         comment Send to multiple users/addresses
  85 ----
  86
  87 [[notification_targets_smtp]]
  88 SMTP
  89 ~~~~
  90 [thumbnail="screenshot/gui-datacenter-notification-smtp.png"]
  91
  92 SMTP notification targets can send emails directly to an SMTP mail relay.
  93
  94 The configuration for SMTP target plugins has the following options:
  95
  96 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
  97 set multiple times to accomodate multiple recipients.
  98 * `mailto-user`: Users to which emails shall be sent to. The user's email
  99 address will be looked up in `users.cfg`. Can be set multiple times to
 100 accomodate multiple recipients.
 101 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
 102 * `from-address`: Sets the From-addresss of the email. SMTP relays might require
 103 that this address is owned by the user in order to avoid spoofing.
 104 The `From` header in the email will be set to `$author <$from-address>`.
 105 * `username`: Username to use during authentication. If no username is set,
 106 no authentication will be performed. The PLAIN and LOGIN authentication methods
 107 are supported.
 108 * `password`: Password to use when authenticating.
 109 * `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
 110 to `tls`.
 111 * `server`: Address/IP of the SMTP relay
 112 * `port`: The port to connect to. If not set, the used port
 113 defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
 114 value of `mode`.
 115 * `comment`: Comment for this target
 116
 117 Example configuration (`/etc/pve/notifications.cfg`):
 118 ----
 119 smtp: example
 120         mailto-user root@pam
 121         mailto-user admin@pve
 122         mailto max@example.com
 123         from-address pve1@example.com
 124         username pve1
 125         server mail.example.com
 126         mode starttls
 127 ----
 128 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
 129 secret token:
 130 ----
 131 smtp: example
 132         password somepassword
 133 ----
 134
 135 [[notification_targets_gotify]]
 136 Gotify
 137 ~~~~~~
 138 [thumbnail="screenshot/gui-datacenter-notification-gotify.png"]
 139
 140 http://gotify.net[Gotify] is an open-source self-hosted notification server that
 141 allows you to send and receive push notifications to various devices and
 142 applications. It provides a simple API and web interface, making it easy to
 143 integrate with different platforms and services.
 144
 145 The configuration for Gotify target plugins has the following options:
 146
 147 * `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
 148 * `token`: The authentication token. Tokens can be generated within the Gotify
 149 web interface.
 150 * `comment`: Comment for this target
 151
 152 NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
 153  xref:datacenter_configuration_file[datacenter configuration]
 154
 155 Example configuration (`/etc/pve/notifications.cfg`):
 156 ----
 157 gotify: example
 158         server http://gotify.example.com:8888
 159         comment Send to multiple users/addresses
 160 ----
 161
 162 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
 163 secret token:
 164 ----
 165 gotify: example
 166         token somesecrettoken
 167 ----
 168
 169 [[notification_matchers]]
 170 Notification Matchers
 171 ---------------------
 172 [thumbnail="screenshot/gui-datacenter-notification-matcher.png"]
 173
 174 Notification matchers route notifications to notification targets based
 175 on their matching rules. These rules can match certain properties of a
 176 notification, such as the timestamp (`match-calendar`), the severity of
 177 the notification (`match-severity`) or metadata fields (`match-field`).
 178 If a notification is matched by a matcher, all targets configured for the
 179 matcher will receive the notification.
 180
 181 An arbitrary number of matchers can be created, each with with their own
 182 matching rules and targets to notify.
 183 Every target is notified at most once for every notification, even if
 184 the target is used in multiple matchers.
 185
 186 A matcher without any matching rules is always true; the configured targets
 187 will always be notified.
 188 ----
 189 matcher: always-matches
 190         target admin
 191         comment This matcher always matches
 192 ----
 193
 194 Matcher Options
 195 ~~~~~~~~~~~~~~~
 196
 197 * `target`: Determine which target should be notified if the matcher matches.
 198 can be used multiple times to notify multiple targets.
 199 * `invert-match`: Inverts the result of the whole matcher
 200 * `mode`: Determines how the individual match rules are evaluated to compute
 201 the result for the whole matcher. If set to `all`, all matching rules must
 202 match. If set to `any`, at least one rule must match.
 203 a matcher must be true. Defaults to `all`.
 204 * `match-calendar`: Match the notification's timestamp against a schedule
 205 * `match-field`: Match the notification's metadata fields
 206 * `match-severity`: Match the notification's severity
 207 * `comment`: Comment for this matcher
 208
 209 [[notification_matchers_calendar]]
 210 Calendar Matching Rules
 211 ~~~~~~~~~~~~~~~~~~~~~~~
 212 A calendar matcher matches the time when a notification is sent agaist a
 213 configurable schedule.
 214
 215 * `match-calendar 8-12`
 216 * `match-calendar 8:00-15:30`
 217 * `match-calendar mon-fri 9:00-17:00`
 218 * `match-calendar sun,tue-wed,fri 9-17`
 219
 220 [[notification_matchers_field]]
 221 Field Matching Rules
 222 ~~~~~~~~~~~~~~~~~~~~
 223 Notifications have a selection of metadata fields that can be matched.
 224
 225 * `match-field exact:type=vzdump` Only match notifications about backups.
 226 * `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
 227 the node.
 228
 229 If a matched metadata field does not exist, the notification will not be
 230 matched.
 231 For instance, a `match-field regex:hostname=.*` directive will only match
 232 notifications that have an arbitraty `hostname` metadata field, but will
 233 not match if the field does not exist.
 234
 235 [[notification_matchers_severity]]
 236 Severity Matching Rules
 237 ~~~~~~~~~~~~~~~~~~~~~~~
 238 A notification has a associated severity that can be matched.
 239
 240 * `match-severity error`: Only match errors
 241 * `match-severity warning,error`: Match warnings and error
 242
 243 The following severities are in use:
 244 `info`, `notice`, `warning`, `error`, `unknown`.
 245
 246
 247 Examples
 248 ~~~~~~~~
 249 ----
 250 matcher: workday
 251         match-calendar mon-fri 9-17
 252         target admin
 253         comment Notify admins during working hours
 254
 255 matcher: night-and-weekend
 256         match-calendar mon-fri 9-17
 257         invert-match true
 258         target on-call-admins
 259         comment Separate target for non-working hours
 260 ----
 261
 262 ----
 263 matcher: backup-failures
 264         match-field exact:type=vzdump
 265         match-severity error
 266         target backup-admins
 267         comment Send notifications about backup failures to one group of admins
 268
 269 matcher: cluster-failures
 270         match-field exact:type=replication
 271         match-field exact:type=fencing
 272         mode any
 273         target cluster-admins
 274         comment Send cluster-related notifications to other group of admins
 275 ----
 276
 277 The last matcher could also be rewritten using a field matcher with a regular
 278 expression:
 279 ----
 280 matcher: cluster-failures
 281         match-field regex:type=^(replication|fencing)$
 282         target cluster-admins
 283         comment Send cluster-related notifications to other group of admins
 284 ----
 285
 286 [[notification_events]]
 287 Notification Events
 288 -------------------
 289
 290 [width="100%",options="header"]
 291 |===========================================================================
 292 | Event                        | `type`            | Severity | Metadata fields (in addition to `type`)
 293 | System updates available     |`package-updates`  | `info`   | `hostname`
 294 | Cluster node fenced          |`fencing`          | `error`  | `hostname`
 295 | Storage replication failed   |`replication`      | `error`  | -
 296 | Backup finished              |`vzdump`           | `info` (`error` on failure) | `hostname`
 297 | Mail for root                |`system-mail`      | `unknown`| -
 298 |===========================================================================
 299
 300 [width="100%",options="header"]
 301 |=======================================================================
 302 | Field name | Description
 303 | `type`     | Type of the notifcation
 304 | `hostname` | Hostname, including domain (e.g. `pve1.example.com`)
 305 |=======================================================================
 306
 307 System Mail Forwarding
 308 ---------------------
 309
 310 Certain local system daemons, such as `smartd`, generate notification emails
 311 that are initially directed to the local `root` user. {pve} will
 312 feed these mails into the notification system as a notification of
 313 type `system-mail` and with severity `unknown`.
 314
 315 When the forwarding process involves an email-based target
 316 (like `sendmail` or `smtp`), the email is forwarded exactly as received, with all
 317 original mail headers remaining intact. For all other targets,
 318 the system tries to extract both a subject line and the main text body
 319 from the email content. In instances where emails solely consist of HTML
 320 content, they will be transformed into plain text format during this process.
 321
 322 Permissions
 323 -----------
 324
 325 In order to modify/view the configuration for notification targets,
 326 the `Mapping.Modify/Mapping.Audit` permissions are required for the
 327 `/mapping/notifications` ACL node.
 328
 329 Testing a target requires `Mapping.Use`, `Mapping.Audit` or `Mapping.Modify`
 330 permissions on `/mapping/notifications`