[pve-docs.git] / notifications.adoc

[[chapter_notifications]]
Notifications
=============
ifndef::manvolnum[]
:pve-toplevel:
endif::manvolnum[]

Overview
--------

{pve} will send notifications if case of noteworthy events in the system.

There are a number of different xref:notification_events[notification events],
each with their own set of metadata fields that can be used in
notification matchers.

A xref:notification_matchers[notification matcher] determines
_which_ notifications shall be sent _where_.
A matcher has _match rules_, that can be used to
match on certain notification properties (e.g. timestamp, severity,
metadata fields).
If a matcher matches a notification, the notification will be routed
to a configurable set of notification targets.

A xref:notification_targets[notification target] is an abstraction for a
destination where a notification should be sent to - for instance,
a Gotify server instance, or a set of email addresses.
There are multiple types of notification targets, including
`sendmail`, which uses the system's sendmail command to send emails,
or `gotify`, which sends a notification to a Gotify instance.

The notification system can be configured in the GUI under
Datacenter -> Notifications. The configuration is stored in
`/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
the latter contains sensitive configuration options such as
passwords or authentication tokens for notification targets.

[[notification_targets]]
Notification Targets
--------------------

[[notification_targets_sendmail]]
Sendmail
~~~~~~~~
The sendmail binary is a program commonly found on Unix-like operating systems
that handles the sending of email messages.
It is a command-line utility that allows users and applications to send emails
directly from the command line or from within scripts.

The sendmail notification target uses the `sendmail` binary to send emails.


NOTE: In standard {pve} installations, the `sendmail` binary is provided by
Postfix. For this type of target to work correctly, it might be necessary to
change Postfix's configuration so that it can correctly deliver emails.
For cluster setups it is necessary to have a working Postfix configuration on
every single cluster node.

The configuration for Sendmail target plugins has the following options:

* `mailto`: E-Mail address to which the notification shall be sent to. Can be
set multiple times to accomodate multiple recipients.
* `mailto-user`: Users to which emails shall be sent to. The user's email
address will be looked up in `users.cfg`. Can be set multiple times to
accomodate multiple recipients.
* `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
* `from-address`: Sets the from address of the E-Mail. If the parameter is not
set, the plugin will fall back to the `email_from` setting from
`datacenter.cfg`. If that is also not set, the plugin will default to
`root@$hostname`, where `$hostname` is the hostname of the node.
* `comment`: Comment for this target
The `From` header in the email will be set to `$author <$from-address>`.

Example configuration (`/etc/pve/notifications.cfg`):
----
sendmail: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto max@example.com
        from-address pve1@example.com
        comment Send to multiple users/addresses
----

[[notification_targets_smtp]]
SMTP
~~~~

SMTP notification targets can send emails directly to an SMTP mail relay.

The configuration for SMTP target plugins has the following options:

* `mailto`: E-Mail address to which the notification shall be sent to. Can be
set multiple times to accomodate multiple recipients.
* `mailto-user`: Users to which emails shall be sent to. The user's email
address will be looked up in `users.cfg`. Can be set multiple times to
accomodate multiple recipients.
* `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
* `from-address`: Sets the From-addresss of the email. SMTP relays might require
that this address is owned by the user in order to avoid spoofing.
The `From` header in the email will be set to `$author <$from-address>`.
* `username`: Username to use during authentication. If no username is set,
no authentication will be performed. The PLAIN and LOGIN authentication methods
are supported.
* `password`: Password to use when authenticating.
* `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
to `tls`.
* `server`: Address/IP of the SMTP relay
* `port`: The port to connect to. If not set, the used port
defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
value of `mode`.
* `comment`: Comment for this target

Example configuration (`/etc/pve/notifications.cfg`):
----
smtp: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto max@example.com
        from-address pve1@example.com
        username pve1
        server mail.example.com
        mode starttls
----
The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
secret token:
----
smtp: example
        password somepassword
----

[[notification_targets_gotify]]
Gotify
~~~~~~

http://gotify.net[Gotify] is an open-source self-hosted notification server that
allows you to send and receive push notifications to various devices and
applications. It provides a simple API and web interface, making it easy to
integrate with different platforms and services.

The configuration for Gotify target plugins has the following options:

* `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
* `token`: The authentication token. Tokens can be generated within the Gotify
web interface.
* `comment`: Comment for this target

NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
 xref:datacenter_configuration_file[datacenter configuration]

Example configuration (`/etc/pve/notifications.cfg`):
----
gotify: example
        server http://gotify.example.com:8888
        comment Send to multiple users/addresses
----

The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
secret token:
----
gotify: example
        token somesecrettoken
----

[[notification_matchers]]
Notification Matchers
---------------------
Notification matchers route notifications to notification targets based
on their matching rules. These rules can match of certain properties of
a notification, such as the timestamp (`match-calendar`), the severity of
the notificaiton (`match-severity`) or metadata fiels (`match-field`).
If a matcher matches a notification, all targets configured for the matcher
will receive the notification.

An arbitrary number of matchers can be created, each with with their own
matching rules and targets to notify.
Every target is notified at most once for every notification, even if
the target is used in multiple matchers.

A matcher without any matching rules is always true; the configured targets
will always be notified.
----
matcher: always-matches
        target admin
        comment This matcher always matches
----

Matcher Options
~~~~~~~~~~~~~~~

* `target`: Determine which target should be notified if the matcher matches.
can be used multiple times to notify multiple targets.
* `invert-match`: Inverts the result of the whole matcher
* `mode`: Determines how the individual match rules are evaluated to compute
the result for the whole matcher. If set to `all`, all matching rules must
match. If set to `any`, at least one rule must match.
a matcher must be true. Defaults to `all`.
* `match-calendar`: Match the notification's timestamp against a schedule
* `match-field`: Match the notification's metadata fields
* `match-severity`: Match the notification's severity
* `comment`: Comment for this matcher

[[notification_matchers_calendar]]
Calendar Matching Rules
~~~~~~~~~~~~~~~~~~~~~~~
A calendar matcher matches the time when a notification is sent agaist a
configurable schedule.

* `match-calendar 8-12`
* `match-calendar 8:00-15:30`
* `match-calendar mon-fri 9:00-17:00`
* `match-calendar sun,tue-wed,fri 9-17`

[[notification_matchers_field]]
Field Matching Rules
~~~~~~~~~~~~~~~~~~~~
Notifications have a selection of metadata fields that can be matched.

* `match-field exact:type=vzdump` Only match notifications about backups.
* `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
the node.

If a matched metadata field does not exist, the notification will not be
matched.
For instance, a `match-field regex:hostname=.*` directive will only match
notifications that have an arbitraty `hostname` metadata field, but will
not match if the field does not exist.

[[notification_matchers_severity]]
Severity Matching Rules
~~~~~~~~~~~~~~~~~~~~~~~
A notification has a associated severity that can be matched.

* `match-severity error`: Only match errors
* `match-severity warning,error`: Match warnings and error

The following severities are in use:
`info`, `notice`, `warning`, `error`, `unknown`.


Examples
~~~~~~~~
----
matcher: workday
        match-calendar mon-fri 9-17
        target admin
        comment Notify admins during working hours

matcher: night-and-weekend
        match-calendar mon-fri 9-17
        invert-match true
        target on-call-admins
        comment Separate target for non-working hours
----

----
matcher: backup-failures
        match-field exact:type=vzdump
        match-severity error
        target backup-admins
        comment Send notifications about backup failures to one group of admins

matcher: cluster-failures
        match-field exact:type=replication
        match-field exact:type=fencing
        mode any
        target cluster-admins
        comment Send cluster-related notifications to other group of admins
----

The last matcher could also be rewritten using a field matcher with a regular
expression:
----
matcher: cluster-failures
        match-field regex:type=^(replication|fencing)$
        target cluster-admins
        comment Send cluster-related notifications to other group of admins
----

[[notification_events]]
Notification Events
-------------------

[width="100%",options="header"]
|===========================================================================
| Event                        | `type`            | Severity | Metadata fields (in addition to `type`)
| System updates available     |`package-updates`  | `info`   | `hostname`
| Cluster node fenced          |`fencing`          | `error`  | `hostname`
| Storage replication failed   |`replication`      | `error`  | -
| Backup finished              |`vzdump`           | `info` (`error` on failure) | `hostname`
| Mail for root                |`system-mail`      | `unknown`| -
|===========================================================================

[width="100%",options="header"]
|=======================================================================
| Field name | Description
| `type`     | Type of the notifcation
| `hostname` | Hostname, including domain (e.g. `pve1.example.com`)
|=======================================================================

System Mail Forwarding
---------------------

Certain local system daemons, such as `smartd`, generate notification emails
that are initially directed to the local `root` user. {pve} will
feed these mails into the notification system as a notification of
type `system-mail` and with severity `unknown`.

When the forwarding process involves an email-based target
(like `sendmail` or `smtp`), the email is forwarded exactly as received, with all
original mail headers remaining intact. For all other targets,
the system tries to extract both a subject line and the main text body
from the email content. In instances where emails solely consist of HTML
content, they will be transformed into plain text format during this process.

Permissions
-----------

In order to modify/view the configuration for notification targets,
the `Mapping.Modify/Mapping.Audit` permissions are required for the
`/mapping/notifications` ACL node.

Testing a target requires `Mapping.Use`, `Mapping.Audit` or `Mapping.Modify`
permissions on `/mapping/notifications`
Commit	Line	Data
1ba1e952 LW	1	[[chapter_notifications]]
	2	Notifications
	3	=============
	4	ifndef::manvolnum[]
	5	:pve-toplevel:
	6	endif::manvolnum[]
	7
c877f0aa LW	8	Overview
	9	--------
	10
	11	{pve} will send notifications if case of noteworthy events in the system.
	12
	13	There are a number of different xref:notification_events[notification events],
	14	each with their own set of metadata fields that can be used in
	15	notification matchers.
	16
	17	A xref:notification_matchers[notification matcher] determines
	18	_which_ notifications shall be sent _where_.
	19	A matcher has _match rules_, that can be used to
	20	match on certain notification properties (e.g. timestamp, severity,
	21	metadata fields).
	22	If a matcher matches a notification, the notification will be routed
	23	to a configurable set of notification targets.
	24
	25	A xref:notification_targets[notification target] is an abstraction for a
	26	destination where a notification should be sent to - for instance,
	27	a Gotify server instance, or a set of email addresses.
	28	There are multiple types of notification targets, including
	29	`sendmail`, which uses the system's sendmail command to send emails,
	30	or `gotify`, which sends a notification to a Gotify instance.
	31
	32	The notification system can be configured in the GUI under
	33	Datacenter -> Notifications. The configuration is stored in
	34	`/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
	35	the latter contains sensitive configuration options such as
	36	passwords or authentication tokens for notification targets.
1ba1e952 LW	37
	38	[[notification_targets]]
	39	Notification Targets
	40	--------------------
	41
f5f316bd	42	[[notification_targets_sendmail]]
1ba1e952 LW	43	Sendmail
	44	~~~~~~~~
	45	The sendmail binary is a program commonly found on Unix-like operating systems
	46	that handles the sending of email messages.
	47	It is a command-line utility that allows users and applications to send emails
	48	directly from the command line or from within scripts.
	49
	50	The sendmail notification target uses the `sendmail` binary to send emails.
	51
	52
	53	NOTE: In standard {pve} installations, the `sendmail` binary is provided by
	54	Postfix. For this type of target to work correctly, it might be necessary to
	55	change Postfix's configuration so that it can correctly deliver emails.
	56	For cluster setups it is necessary to have a working Postfix configuration on
	57	every single cluster node.
	58
	59	The configuration for Sendmail target plugins has the following options:
	60
	61	* `mailto`: E-Mail address to which the notification shall be sent to. Can be
	62	set multiple times to accomodate multiple recipients.
	63	* `mailto-user`: Users to which emails shall be sent to. The user's email
	64	address will be looked up in `users.cfg`. Can be set multiple times to
	65	accomodate multiple recipients.
	66	* `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
	67	* `from-address`: Sets the from address of the E-Mail. If the parameter is not
	68	set, the plugin will fall back to the `email_from` setting from
	69	`datacenter.cfg`. If that is also not set, the plugin will default to
	70	`root@$hostname`, where `$hostname` is the hostname of the node.
e39dfad9	71	* `comment`: Comment for this target
7a20522e	72	The `From` header in the email will be set to `$author <$from-address>`.
1ba1e952	73
c877f0aa LW	74	Example configuration (`/etc/pve/notifications.cfg`):
	75	----
	76	sendmail: example
	77	mailto-user root@pam
	78	mailto-user admin@pve
	79	mailto max@example.com
	80	from-address pve1@example.com
	81	comment Send to multiple users/addresses
	82	----
1ba1e952	83
f5f316bd	84	[[notification_targets_smtp]]
7a20522e LW	85	SMTP
	86	~~~~
	87
	88	SMTP notification targets can send emails directly to an SMTP mail relay.
	89
	90	The configuration for SMTP target plugins has the following options:
	91
	92	* `mailto`: E-Mail address to which the notification shall be sent to. Can be
	93	set multiple times to accomodate multiple recipients.
	94	* `mailto-user`: Users to which emails shall be sent to. The user's email
	95	address will be looked up in `users.cfg`. Can be set multiple times to
	96	accomodate multiple recipients.
	97	* `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
	98	* `from-address`: Sets the From-addresss of the email. SMTP relays might require
	99	that this address is owned by the user in order to avoid spoofing.
	100	The `From` header in the email will be set to `$author <$from-address>`.
	101	* `username`: Username to use during authentication. If no username is set,
	102	no authentication will be performed. The PLAIN and LOGIN authentication methods
	103	are supported.
	104	* `password`: Password to use when authenticating.
	105	* `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
	106	to `tls`.
	107	* `server`: Address/IP of the SMTP relay
	108	* `port`: The port to connect to. If not set, the used port
	109	defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
	110	value of `mode`.
	111	* `comment`: Comment for this target
	112
	113	Example configuration (`/etc/pve/notifications.cfg`):
	114	----
	115	smtp: example
	116	mailto-user root@pam
	117	mailto-user admin@pve
	118	mailto max@example.com
	119	from-address pve1@example.com
	120	username pve1
	121	server mail.example.com
	122	mode starttls
	123	----
	124	The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
	125	secret token:
	126	----
	127	smtp: example
	128	password somepassword
	129	----
	130
f5f316bd	131	[[notification_targets_gotify]]
1ba1e952 LW	132	Gotify
	133	~~~~~~
	134
	135	http://gotify.net[Gotify] is an open-source self-hosted notification server that
	136	allows you to send and receive push notifications to various devices and
	137	applications. It provides a simple API and web interface, making it easy to
	138	integrate with different platforms and services.
	139
	140	The configuration for Gotify target plugins has the following options:
	141
	142	* `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
	143	* `token`: The authentication token. Tokens can be generated within the Gotify
	144	web interface.
e39dfad9	145	* `comment`: Comment for this target
1ba1e952 LW	146
	147	NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
	148	xref:datacenter_configuration_file[datacenter configuration]
	149
c877f0aa LW	150	Example configuration (`/etc/pve/notifications.cfg`):
	151	----
	152	gotify: example
	153	server http://gotify.example.com:8888
	154	comment Send to multiple users/addresses
	155	----
1ba1e952	156
c877f0aa LW	157	The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
	158	secret token:
	159	----
	160	gotify: example
	161	token somesecrettoken
	162	----
1ba1e952	163
c877f0aa LW	164	[[notification_matchers]]
	165	Notification Matchers
	166	---------------------
	167	Notification matchers route notifications to notification targets based
	168	on their matching rules. These rules can match of certain properties of
	169	a notification, such as the timestamp (`match-calendar`), the severity of
	170	the notificaiton (`match-severity`) or metadata fiels (`match-field`).
	171	If a matcher matches a notification, all targets configured for the matcher
	172	will receive the notification.
	173
	174	An arbitrary number of matchers can be created, each with with their own
	175	matching rules and targets to notify.
	176	Every target is notified at most once for every notification, even if
	177	the target is used in multiple matchers.
	178
	179	A matcher without any matching rules is always true; the configured targets
	180	will always be notified.
	181	----
	182	matcher: always-matches
	183	target admin
	184	comment This matcher always matches
	185	----
1ba1e952	186
c877f0aa LW	187	Matcher Options
c877f0aa LW	188	~~~~~~~~~~~~~~~
1ba1e952	189
c877f0aa LW	190	* `target`: Determine which target should be notified if the matcher matches.
	191	can be used multiple times to notify multiple targets.
	192	* `invert-match`: Inverts the result of the whole matcher
	193	* `mode`: Determines how the individual match rules are evaluated to compute
	194	the result for the whole matcher. If set to `all`, all matching rules must
	195	match. If set to `any`, at least one rule must match.
	196	a matcher must be true. Defaults to `all`.
	197	* `match-calendar`: Match the notification's timestamp against a schedule
	198	* `match-field`: Match the notification's metadata fields
	199	* `match-severity`: Match the notification's severity
e39dfad9	200	* `comment`: Comment for this matcher
1ba1e952	201
f5f316bd	202	[[notification_matchers_calendar]]
c877f0aa LW	203	Calendar Matching Rules
	204	~~~~~~~~~~~~~~~~~~~~~~~
	205	A calendar matcher matches the time when a notification is sent agaist a
	206	configurable schedule.
1ba1e952	207
c877f0aa LW	208	* `match-calendar 8-12`
	209	* `match-calendar 8:00-15:30`
	210	* `match-calendar mon-fri 9:00-17:00`
	211	* `match-calendar sun,tue-wed,fri 9-17`
1ba1e952	212
f5f316bd	213	[[notification_matchers_field]]
c877f0aa LW	214	Field Matching Rules
	215	~~~~~~~~~~~~~~~~~~~~
	216	Notifications have a selection of metadata fields that can be matched.
1ba1e952	217
c877f0aa LW	218	* `match-field exact:type=vzdump` Only match notifications about backups.
	219	* `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
	220	the node.
1ba1e952	221
c877f0aa LW	222	If a matched metadata field does not exist, the notification will not be
	223	matched.
	224	For instance, a `match-field regex:hostname=.*` directive will only match
	225	notifications that have an arbitraty `hostname` metadata field, but will
	226	not match if the field does not exist.
1ba1e952	227
f5f316bd	228	[[notification_matchers_severity]]
c877f0aa LW	229	Severity Matching Rules
	230	~~~~~~~~~~~~~~~~~~~~~~~
	231	A notification has a associated severity that can be matched.
1ba1e952	232
c877f0aa LW	233	* `match-severity error`: Only match errors
c877f0aa LW	234	* `match-severity warning,error`: Match warnings and error
1ba1e952	235
c877f0aa	236	The following severities are in use:
24451e5d	237	`info`, `notice`, `warning`, `error`, `unknown`.
1ba1e952	238
1ba1e952	239
c877f0aa LW	240	Examples
	241	~~~~~~~~
	242	----
	243	matcher: workday
	244	match-calendar mon-fri 9-17
	245	target admin
	246	comment Notify admins during working hours
	247
	248	matcher: night-and-weekend
	249	match-calendar mon-fri 9-17
	250	invert-match true
	251	target on-call-admins
	252	comment Separate target for non-working hours
	253	----
1ba1e952	254
c877f0aa LW	255	----
	256	matcher: backup-failures
	257	match-field exact:type=vzdump
	258	match-severity error
	259	target backup-admins
	260	comment Send notifications about backup failures to one group of admins
	261
	262	matcher: cluster-failures
	263	match-field exact:type=replication
	264	match-field exact:type=fencing
	265	mode any
	266	target cluster-admins
	267	comment Send cluster-related notifications to other group of admins
	268	----
1ba1e952	269
c877f0aa LW	270	The last matcher could also be rewritten using a field matcher with a regular
	271	expression:
	272	----
	273	matcher: cluster-failures
	274	match-field regex:type=^(replication\|fencing)$
	275	target cluster-admins
	276	comment Send cluster-related notifications to other group of admins
	277	----
1ba1e952	278
c877f0aa LW	279	[[notification_events]]
	280	Notification Events
	281	-------------------
1ba1e952	282
c877f0aa LW	283	[width="100%",options="header"]
	284	\|===========================================================================
	285	\| Event \| `type` \| Severity \| Metadata fields (in addition to `type`)
	286	\| System updates available \|`package-updates` \| `info` \| `hostname`
	287	\| Cluster node fenced \|`fencing` \| `error` \| `hostname`
	288	\| Storage replication failed \|`replication` \| `error` \| -
	289	\| Backup finished \|`vzdump` \| `info` (`error` on failure) \| `hostname`
7b293b5d	290	\| Mail for root \|`system-mail` \| `unknown`\| -
c877f0aa	291	\|===========================================================================
1ba1e952	292
c877f0aa LW	293	[width="100%",options="header"]
	294	\|=======================================================================
	295	\| Field name \| Description
	296	\| `type` \| Type of the notifcation
	297	\| `hostname` \| Hostname, including domain (e.g. `pve1.example.com`)
	298	\|=======================================================================
1ba1e952	299
7b293b5d LW	300	System Mail Forwarding
	301	---------------------
	302
	303	Certain local system daemons, such as `smartd`, generate notification emails
	304	that are initially directed to the local `root` user. {pve} will
	305	feed these mails into the notification system as a notification of
	306	type `system-mail` and with severity `unknown`.
	307
	308	When the forwarding process involves an email-based target
	309	(like `sendmail` or `smtp`), the email is forwarded exactly as received, with all
	310	original mail headers remaining intact. For all other targets,
	311	the system tries to extract both a subject line and the main text body
	312	from the email content. In instances where emails solely consist of HTML
	313	content, they will be transformed into plain text format during this process.
	314
c877f0aa LW	315	Permissions
	316	-----------
	317
b4c816f4 LW	318	In order to modify/view the configuration for notification targets,
	319	the `Mapping.Modify/Mapping.Audit` permissions are required for the
	320	`/mapping/notifications` ACL node.
	321
	322	Testing a target requires `Mapping.Use`, `Mapping.Audit` or `Mapping.Modify`
	323	permissions on `/mapping/notifications`