notifications.adoc

   1 [[chapter_notifications]]
   2 Notifications
   3 =============
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 Overview
   9 --------
  10
  11 {pve} will send notifications if case of noteworthy events in the system.
  12
  13 There are a number of different xref:notification_events[notification events],
  14 each with their own set of metadata fields that can be used in
  15 notification matchers.
  16
  17 A xref:notification_matchers[notification matcher] determines
  18 _which_ notifications shall be sent _where_.
  19 A matcher has _match rules_, that can be used to
  20 match on certain notification properties (e.g. timestamp, severity,
  21 metadata fields).
  22 If a matcher matches a notification, the notification will be routed
  23 to a configurable set of notification targets.
  24
  25 A xref:notification_targets[notification target] is an abstraction for a
  26 destination where a notification should be sent to - for instance,
  27 a Gotify server instance, or a set of email addresses.
  28 There are multiple types of notification targets, including
  29 `sendmail`, which uses the system's sendmail command to send emails,
  30 or `gotify`, which sends a notification to a Gotify instance.
  31
  32 The notification system can be configured in the GUI under
  33 Datacenter -> Notifications. The configuration is stored in
  34 `/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
  35 the latter contains sensitive configuration options such as
  36 passwords or authentication tokens for notification targets.
  37
  38 [[notification_targets]]
  39 Notification Targets
  40 --------------------
  41
  42 Sendmail
  43 ~~~~~~~~
  44 The sendmail binary is a program commonly found on Unix-like operating systems
  45 that handles the sending of email messages.
  46 It is a command-line utility that allows users and applications to send emails
  47 directly from the command line or from within scripts.
  48
  49 The sendmail notification target uses the `sendmail` binary to send emails.
  50
  51
  52 NOTE: In standard {pve} installations, the `sendmail` binary is provided by
  53 Postfix. For this type of target to work correctly, it might be necessary to
  54 change Postfix's configuration so that it can correctly deliver emails.
  55 For cluster setups it is necessary to have a working Postfix configuration on
  56 every single cluster node.
  57
  58 The configuration for Sendmail target plugins has the following options:
  59
  60 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
  61 set multiple times to accomodate multiple recipients.
  62 * `mailto-user`: Users to which emails shall be sent to. The user's email
  63 address will be looked up in `users.cfg`. Can be set multiple times to
  64 accomodate multiple recipients.
  65 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
  66 * `from-address`: Sets the from address of the E-Mail. If the parameter is not
  67 set, the plugin will fall back to the `email_from` setting from
  68 `datacenter.cfg`. If that is also not set, the plugin will default to
  69 `root@$hostname`, where `$hostname` is the hostname of the node.
  70 * `comment`: Comment for this target
  71 The `From` header in the email will be set to `$author <$from-address>`.
  72
  73 Example configuration (`/etc/pve/notifications.cfg`):
  74 ----
  75 sendmail: example
  76         mailto-user root@pam
  77         mailto-user admin@pve
  78         mailto max@example.com
  79         from-address pve1@example.com
  80         comment Send to multiple users/addresses
  81 ----
  82
  83 SMTP
  84 ~~~~
  85
  86 SMTP notification targets can send emails directly to an SMTP mail relay.
  87
  88 The configuration for SMTP target plugins has the following options:
  89
  90 * `mailto`: E-Mail address to which the notification shall be sent to. Can be
  91 set multiple times to accomodate multiple recipients.
  92 * `mailto-user`: Users to which emails shall be sent to. The user's email
  93 address will be looked up in `users.cfg`. Can be set multiple times to
  94 accomodate multiple recipients.
  95 * `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
  96 * `from-address`: Sets the From-addresss of the email. SMTP relays might require
  97 that this address is owned by the user in order to avoid spoofing.
  98 The `From` header in the email will be set to `$author <$from-address>`.
  99 * `username`: Username to use during authentication. If no username is set,
 100 no authentication will be performed. The PLAIN and LOGIN authentication methods
 101 are supported.
 102 * `password`: Password to use when authenticating.
 103 * `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
 104 to `tls`.
 105 * `server`: Address/IP of the SMTP relay
 106 * `port`: The port to connect to. If not set, the used port
 107 defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
 108 value of `mode`.
 109 * `comment`: Comment for this target
 110
 111 Example configuration (`/etc/pve/notifications.cfg`):
 112 ----
 113 smtp: example
 114         mailto-user root@pam
 115         mailto-user admin@pve
 116         mailto max@example.com
 117         from-address pve1@example.com
 118         username pve1
 119         server mail.example.com
 120         mode starttls
 121 ----
 122 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
 123 secret token:
 124 ----
 125 smtp: example
 126         password somepassword
 127 ----
 128
 129 Gotify
 130 ~~~~~~
 131
 132 http://gotify.net[Gotify] is an open-source self-hosted notification server that
 133 allows you to send and receive push notifications to various devices and
 134 applications. It provides a simple API and web interface, making it easy to
 135 integrate with different platforms and services.
 136
 137 The configuration for Gotify target plugins has the following options:
 138
 139 * `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
 140 * `token`: The authentication token. Tokens can be generated within the Gotify
 141 web interface.
 142 * `comment`: Comment for this target
 143
 144 NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
 145  xref:datacenter_configuration_file[datacenter configuration]
 146
 147 Example configuration (`/etc/pve/notifications.cfg`):
 148 ----
 149 gotify: example
 150         server http://gotify.example.com:8888
 151         comment Send to multiple users/addresses
 152 ----
 153
 154 The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
 155 secret token:
 156 ----
 157 gotify: example
 158         token somesecrettoken
 159 ----
 160
 161 [[notification_matchers]]
 162 Notification Matchers
 163 ---------------------
 164 Notification matchers route notifications to notification targets based
 165 on their matching rules. These rules can match of certain properties of
 166 a notification, such as the timestamp (`match-calendar`), the severity of
 167 the notificaiton (`match-severity`) or metadata fiels (`match-field`).
 168 If a matcher matches a notification, all targets configured for the matcher
 169 will receive the notification.
 170
 171 An arbitrary number of matchers can be created, each with with their own
 172 matching rules and targets to notify.
 173 Every target is notified at most once for every notification, even if
 174 the target is used in multiple matchers.
 175
 176 A matcher without any matching rules is always true; the configured targets
 177 will always be notified.
 178 ----
 179 matcher: always-matches
 180         target admin
 181         comment This matcher always matches
 182 ----
 183
 184 Matcher Options
 185 ~~~~~~~~~~~~~~~
 186
 187 * `target`: Determine which target should be notified if the matcher matches.
 188 can be used multiple times to notify multiple targets.
 189 * `invert-match`: Inverts the result of the whole matcher
 190 * `mode`: Determines how the individual match rules are evaluated to compute
 191 the result for the whole matcher. If set to `all`, all matching rules must
 192 match. If set to `any`, at least one rule must match.
 193 a matcher must be true. Defaults to `all`.
 194 * `match-calendar`: Match the notification's timestamp against a schedule
 195 * `match-field`: Match the notification's metadata fields
 196 * `match-severity`: Match the notification's severity
 197 * `comment`: Comment for this matcher
 198
 199 Calendar Matching Rules
 200 ~~~~~~~~~~~~~~~~~~~~~~~
 201 A calendar matcher matches the time when a notification is sent agaist a
 202 configurable schedule.
 203
 204 * `match-calendar 8-12`
 205 * `match-calendar 8:00-15:30`
 206 * `match-calendar mon-fri 9:00-17:00`
 207 * `match-calendar sun,tue-wed,fri 9-17`
 208
 209 Field Matching Rules
 210 ~~~~~~~~~~~~~~~~~~~~
 211 Notifications have a selection of metadata fields that can be matched.
 212
 213 * `match-field exact:type=vzdump` Only match notifications about backups.
 214 * `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
 215 the node.
 216
 217 If a matched metadata field does not exist, the notification will not be
 218 matched.
 219 For instance, a `match-field regex:hostname=.*` directive will only match
 220 notifications that have an arbitraty `hostname` metadata field, but will
 221 not match if the field does not exist.
 222
 223 Severity Matching Rules
 224 ~~~~~~~~~~~~~~~~~~~~~~~
 225 A notification has a associated severity that can be matched.
 226
 227 * `match-severity error`: Only match errors
 228 * `match-severity warning,error`: Match warnings and error
 229
 230 The following severities are in use:
 231 `info`, `notice`, `warning`, `error`.
 232
 233
 234 Examples
 235 ~~~~~~~~
 236 ----
 237 matcher: workday
 238         match-calendar mon-fri 9-17
 239         target admin
 240         comment Notify admins during working hours
 241
 242 matcher: night-and-weekend
 243         match-calendar mon-fri 9-17
 244         invert-match true
 245         target on-call-admins
 246         comment Separate target for non-working hours
 247 ----
 248
 249 ----
 250 matcher: backup-failures
 251         match-field exact:type=vzdump
 252         match-severity error
 253         target backup-admins
 254         comment Send notifications about backup failures to one group of admins
 255
 256 matcher: cluster-failures
 257         match-field exact:type=replication
 258         match-field exact:type=fencing
 259         mode any
 260         target cluster-admins
 261         comment Send cluster-related notifications to other group of admins
 262 ----
 263
 264 The last matcher could also be rewritten using a field matcher with a regular
 265 expression:
 266 ----
 267 matcher: cluster-failures
 268         match-field regex:type=^(replication|fencing)$
 269         target cluster-admins
 270         comment Send cluster-related notifications to other group of admins
 271 ----
 272
 273 [[notification_events]]
 274 Notification Events
 275 -------------------
 276
 277 [width="100%",options="header"]
 278 |===========================================================================
 279 | Event                        | `type`            | Severity | Metadata fields (in addition to `type`)
 280 | System updates available     |`package-updates`  | `info`   | `hostname`
 281 | Cluster node fenced          |`fencing`          | `error`  | `hostname`
 282 | Storage replication failed   |`replication`      | `error`  | -
 283 | Backup finished              |`vzdump`           | `info` (`error` on failure) | `hostname`
 284 | Mail for root                |`system-mail`      | `unknown`| -
 285 |===========================================================================
 286
 287 [width="100%",options="header"]
 288 |=======================================================================
 289 | Field name | Description
 290 | `type`     | Type of the notifcation
 291 | `hostname` | Hostname, including domain (e.g. `pve1.example.com`)
 292 |=======================================================================
 293
 294 System Mail Forwarding
 295 ---------------------
 296
 297 Certain local system daemons, such as `smartd`, generate notification emails
 298 that are initially directed to the local `root` user. {pve} will
 299 feed these mails into the notification system as a notification of
 300 type `system-mail` and with severity `unknown`.
 301
 302 When the forwarding process involves an email-based target
 303 (like `sendmail` or `smtp`), the email is forwarded exactly as received, with all
 304 original mail headers remaining intact. For all other targets,
 305 the system tries to extract both a subject line and the main text body
 306 from the email content. In instances where emails solely consist of HTML
 307 content, they will be transformed into plain text format during this process.
 308
 309 Permissions
 310 -----------
 311
 312 For every target, there exists a corresponding ACL path
 313 `/mapping/notification/targets/<name>`. Matchers use
 314 a seperate namespace in the ACL tree: `/mapping/notification/matchers/<name>`.
 315
 316 To test a target, a user must have the `Mapping.Use` permission on the corresponding
 317 node in the ACL tree.
 318 `Mapping.Modify` and `Mapping.Audit` are needed to read/modify the
 319 configuration of a target or matcher.