[ceph.git] / ceph / doc / radosgw / s3-notification-compatibility.rst

=====================================
S3 Bucket Notifications Compatibility
=====================================

Ceph's `Bucket Notifications`_ API follows `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below.


.. note:: 

    Compatibility is different depending on which of the above mechanism is used

Supported Destination
---------------------

AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations). 
Currently, we support: **HTTP/S**, **Kafka** and **AMQP**.

We are using the **SNS** ARNs to represent the **HTTP/S**, **Kafka** and **AMQP** destinations.

Notification Configuration XML
------------------------------

Following tags (and the tags inside them) are not supported:

+-----------------------------------+----------------------------------------------+
| Tag                               | Remaks                                       |
+===================================+==============================================+
| ``<QueueConfiguration>``          | not needed, we treat all destinations as SNS |
+-----------------------------------+----------------------------------------------+
| ``<CloudFunctionConfiguration>``  | not needed, we treat all destinations as SNS |
+-----------------------------------+----------------------------------------------+

REST API Extension 
------------------

Ceph's bucket notification API has the following extensions:

- Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb

 - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket

- Getting the information on a specific notification (when more than one exists on a bucket)

  - In S3, it is only possible to fetch all notifications on a bucket

- In addition to filtering based on prefix/suffix of object keys we support:

  - Filtering based on regular expression matching

  - Filtering based on metadata attributes attached to the object

  - Filtering based on object tags

- Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP). 

- Filtering overlapping is allowed, so that same event could be sent as different notification


Unsupported Fields in the Event Record
--------------------------------------

The records sent for bucket notification follows the format described in: `Event Message Structure`_.
However, the ``requestParameters.sourceIPAddress`` field will be sent empty.


Event Types
-----------

+--------------------------------------------------------+-----------------------------------------+
| Event                                                  | Note                                    |
+========================================================+=========================================+
| ``s3:ObjectCreated:*``                                 | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectCreated:Put``                               | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectCreated:Post``                              | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectCreated:Copy``                              | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectCreated:CompleteMultipartUpload``           | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectRemoved:*``                                 | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectRemoved:Delete``                            | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectRemoved:DeleteMarkerCreated``               | Supported                               |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectLifecycle:Expiration:Current``              | Ceph extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectLifecycle:Expiration:NonCurrent``           | Ceph extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectLifecycle:Expiration:DeleteMarker``         | Ceph extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectLifecycle:Expiration:AbortMultipartUpload`` | Defined, Ceph extension (not generated) |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectLifecycle:Transition:Current``              | Ceph extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectLifecycle:Transition:NonCurrent``           | Ceph extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectSynced:*``                                  | Ceph extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectSynced:Create``                             | Ceph Extension                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectSynced:Delete``                             | Defined, Ceph extension (not generated) |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectSynced:DeletionMarkerCreated``              | Defined, Ceph extension (not generated) |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectRestore:Post``                              | Not applicable                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ObjectRestore:Complete``                          | Not applicable                          |
+--------------------------------------------------------+-----------------------------------------+
| ``s3:ReducedRedundancyLostObject``                     | Not applicable                          |
+--------------------------------------------------------+-----------------------------------------+

.. note:: 

   The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object

.. note::

   In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process.
   
.. note::

   The ``s3:ObjectSynced:Create`` event is sent when an object successfully syncs to a zone. It must be explicitly set for each zone. 

Topic Configuration
-------------------
In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_. 
Note that most of the API is not applicable to Ceph, and only the following actions are implemented:

 - ``CreateTopic``
 - ``DeleteTopic``
 - ``ListTopics``

We also have the following extensions to topic configuration: 

 - In ``GetTopic`` we allow fetching a specific topic, instead of all user topics
 - In ``CreateTopic``

  - we allow setting endpoint attributes
  - we allow setting opaque data that will be sent to the endpoint in the notification


.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html
.. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
.. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
.. _`Bucket Notifications`: ../notifications
.. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/main/examples/rgw/boto3
Commit	Line	Data
eafe8130 TL	1	=====================================
	2	S3 Bucket Notifications Compatibility
	3	=====================================
	4
1e59de90	5	Ceph's `Bucket Notifications`_ API follows `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below.
eafe8130 TL	6
	7
	8	.. note::
	9
	10	Compatibility is different depending on which of the above mechanism is used
	11
	12	Supported Destination
	13	---------------------
	14
	15	AWS supports: SNS, SQS and Lambda as possible destinations (AWS internal destinations).
aee94f69	16	Currently, we support: HTTP/S, Kafka and AMQP.
eafe8130	17
f67539c2	18	We are using the SNS ARNs to represent the HTTP/S, Kafka and AMQP destinations.
eafe8130 TL	19
	20	Notification Configuration XML
	21	------------------------------
	22
	23	Following tags (and the tags inside them) are not supported:
	24
	25	+-----------------------------------+----------------------------------------------+
	26	\| Tag \| Remaks \|
	27	+===================================+==============================================+
	28	\| ``<QueueConfiguration>`` \| not needed, we treat all destinations as SNS \|
	29	+-----------------------------------+----------------------------------------------+
	30	\| ``<CloudFunctionConfiguration>`` \| not needed, we treat all destinations as SNS \|
	31	+-----------------------------------+----------------------------------------------+
	32
	33	REST API Extension
	34	------------------
	35
	36	Ceph's bucket notification API has the following extensions:
	37
	38	- Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb
	39
	40	- In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket
	41
	42	- Getting the information on a specific notification (when more than one exists on a bucket)
	43
	44	- In S3, it is only possible to fetch all notifications on a bucket
	45
	46	- In addition to filtering based on prefix/suffix of object keys we support:
	47
	48	- Filtering based on regular expression matching
	49
	50	- Filtering based on metadata attributes attached to the object
	51
9f95a23c TL	52	- Filtering based on object tags
9f95a23c TL	53
f67539c2 TL	54	- Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP).
f67539c2 TL	55
eafe8130 TL	56	- Filtering overlapping is allowed, so that same event could be sent as different notification
	57
	58
	59	Unsupported Fields in the Event Record
	60	--------------------------------------
	61
1e59de90 TL	62	The records sent for bucket notification follows the format described in: `Event Message Structure`_.
	63	However, the ``requestParameters.sourceIPAddress`` field will be sent empty.
	64
eafe8130 TL	65
	66	Event Types
	67	-----------
	68
1e59de90 TL	69	+--------------------------------------------------------+-----------------------------------------+
	70	\| Event \| Note \|
	71	+========================================================+=========================================+
	72	\| ``s3:ObjectCreated:*`` \| Supported \|
	73	+--------------------------------------------------------+-----------------------------------------+
	74	\| ``s3:ObjectCreated:Put`` \| Supported \|
	75	+--------------------------------------------------------+-----------------------------------------+
	76	\| ``s3:ObjectCreated:Post`` \| Supported \|
	77	+--------------------------------------------------------+-----------------------------------------+
	78	\| ``s3:ObjectCreated:Copy`` \| Supported \|
	79	+--------------------------------------------------------+-----------------------------------------+
	80	\| ``s3:ObjectCreated:CompleteMultipartUpload`` \| Supported \|
	81	+--------------------------------------------------------+-----------------------------------------+
	82	\| ``s3:ObjectRemoved:*`` \| Supported \|
	83	+--------------------------------------------------------+-----------------------------------------+
	84	\| ``s3:ObjectRemoved:Delete`` \| Supported \|
	85	+--------------------------------------------------------+-----------------------------------------+
	86	\| ``s3:ObjectRemoved:DeleteMarkerCreated`` \| Supported \|
	87	+--------------------------------------------------------+-----------------------------------------+
	88	\| ``s3:ObjectLifecycle:Expiration:Current`` \| Ceph extension \|
	89	+--------------------------------------------------------+-----------------------------------------+
	90	\| ``s3:ObjectLifecycle:Expiration:NonCurrent`` \| Ceph extension \|
	91	+--------------------------------------------------------+-----------------------------------------+
	92	\| ``s3:ObjectLifecycle:Expiration:DeleteMarker`` \| Ceph extension \|
	93	+--------------------------------------------------------+-----------------------------------------+
	94	\| ``s3:ObjectLifecycle:Expiration:AbortMultipartUpload`` \| Defined, Ceph extension (not generated) \|
	95	+--------------------------------------------------------+-----------------------------------------+
	96	\| ``s3:ObjectLifecycle:Transition:Current`` \| Ceph extension \|
	97	+--------------------------------------------------------+-----------------------------------------+
	98	\| ``s3:ObjectLifecycle:Transition:NonCurrent`` \| Ceph extension \|
	99	+--------------------------------------------------------+-----------------------------------------+
	100	\| ``s3:ObjectSynced:*`` \| Ceph extension \|
	101	+--------------------------------------------------------+-----------------------------------------+
	102	\| ``s3:ObjectSynced:Create`` \| Ceph Extension \|
	103	+--------------------------------------------------------+-----------------------------------------+
	104	\| ``s3:ObjectSynced:Delete`` \| Defined, Ceph extension (not generated) \|
	105	+--------------------------------------------------------+-----------------------------------------+
	106	\| ``s3:ObjectSynced:DeletionMarkerCreated`` \| Defined, Ceph extension (not generated) \|
	107	+--------------------------------------------------------+-----------------------------------------+
	108	\| ``s3:ObjectRestore:Post`` \| Not applicable \|
	109	+--------------------------------------------------------+-----------------------------------------+
	110	\| ``s3:ObjectRestore:Complete`` \| Not applicable \|
	111	+--------------------------------------------------------+-----------------------------------------+
	112	\| ``s3:ReducedRedundancyLostObject`` \| Not applicable \|
	113	+--------------------------------------------------------+-----------------------------------------+
eafe8130	114
a4b75251 TL	115	.. note::
	116
	117	The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object
	118
20effc67 TL	119	.. note::
	120
	121	In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process.
1e59de90 TL	122
	123	.. note::
	124
	125	The ``s3:ObjectSynced:Create`` event is sent when an object successfully syncs to a zone. It must be explicitly set for each zone.
20effc67	126
eafe8130 TL	127	Topic Configuration
	128	-------------------
	129	In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_.
	130	Note that most of the API is not applicable to Ceph, and only the following actions are implemented:
	131
	132	- ``CreateTopic``
	133	- ``DeleteTopic``
	134	- ``ListTopics``
	135
9f95a23c TL	136	We also have the following extensions to topic configuration:
	137
	138	- In ``GetTopic`` we allow fetching a specific topic, instead of all user topics
	139	- In ``CreateTopic``
	140
	141	- we allow setting endpoint attributes
20effc67	142	- we allow setting opaque data that will be sent to the endpoint in the notification
eafe8130	143
eafe8130 TL	144
	145	.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html
	146	.. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
	147	.. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
eafe8130	148	.. _`Bucket Notifications`: ../notifications
1e59de90	149	.. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/main/examples/rgw/boto3