ceph/doc/radosgw/s3-notification-compatibility.rst

   1 =====================================
   2 S3 Bucket Notifications Compatibility
   3 =====================================
   4
   5 Ceph's `Bucket Notifications`_ API follows `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below.
   6
   7
   8 .. note::
   9
  10     Compatibility is different depending on which of the above mechanism is used
  11
  12 Supported Destination
  13 ---------------------
  14
  15 AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations).
  16 Currently, we support: **HTTP/S**, **Kafka** and **AMQP**.
  17
  18 We are using the **SNS** ARNs to represent the **HTTP/S**, **Kafka** and **AMQP** destinations.
  19
  20 Notification Configuration XML
  21 ------------------------------
  22
  23 Following tags (and the tags inside them) are not supported:
  24
  25 +-----------------------------------+----------------------------------------------+
  26 | Tag                               | Remaks                                       |
  27 +===================================+==============================================+
  28 | ``<QueueConfiguration>``          | not needed, we treat all destinations as SNS |
  29 +-----------------------------------+----------------------------------------------+
  30 | ``<CloudFunctionConfiguration>``  | not needed, we treat all destinations as SNS |
  31 +-----------------------------------+----------------------------------------------+
  32
  33 REST API Extension
  34 ------------------
  35
  36 Ceph's bucket notification API has the following extensions:
  37
  38 - Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb
  39
  40  - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket
  41
  42 - Getting the information on a specific notification (when more than one exists on a bucket)
  43
  44   - In S3, it is only possible to fetch all notifications on a bucket
  45
  46 - In addition to filtering based on prefix/suffix of object keys we support:
  47
  48   - Filtering based on regular expression matching
  49
  50   - Filtering based on metadata attributes attached to the object
  51
  52   - Filtering based on object tags
  53
  54 - Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP).
  55
  56 - Filtering overlapping is allowed, so that same event could be sent as different notification
  57
  58
  59 Unsupported Fields in the Event Record
  60 --------------------------------------
  61
  62 The records sent for bucket notification follows the format described in: `Event Message Structure`_.
  63 However, the ``requestParameters.sourceIPAddress`` field will be sent empty.
  64
  65
  66 Event Types
  67 -----------
  68
  69 +--------------------------------------------------------+-----------------------------------------+
  70 | Event                                                  | Note                                    |
  71 +========================================================+=========================================+
  72 | ``s3:ObjectCreated:*``                                 | Supported                               |
  73 +--------------------------------------------------------+-----------------------------------------+
  74 | ``s3:ObjectCreated:Put``                               | Supported                               |
  75 +--------------------------------------------------------+-----------------------------------------+
  76 | ``s3:ObjectCreated:Post``                              | Supported                               |
  77 +--------------------------------------------------------+-----------------------------------------+
  78 | ``s3:ObjectCreated:Copy``                              | Supported                               |
  79 +--------------------------------------------------------+-----------------------------------------+
  80 | ``s3:ObjectCreated:CompleteMultipartUpload``           | Supported                               |
  81 +--------------------------------------------------------+-----------------------------------------+
  82 | ``s3:ObjectRemoved:*``                                 | Supported                               |
  83 +--------------------------------------------------------+-----------------------------------------+
  84 | ``s3:ObjectRemoved:Delete``                            | Supported                               |
  85 +--------------------------------------------------------+-----------------------------------------+
  86 | ``s3:ObjectRemoved:DeleteMarkerCreated``               | Supported                               |
  87 +--------------------------------------------------------+-----------------------------------------+
  88 | ``s3:ObjectLifecycle:Expiration:Current``              | Ceph extension                          |
  89 +--------------------------------------------------------+-----------------------------------------+
  90 | ``s3:ObjectLifecycle:Expiration:NonCurrent``           | Ceph extension                          |
  91 +--------------------------------------------------------+-----------------------------------------+
  92 | ``s3:ObjectLifecycle:Expiration:DeleteMarker``         | Ceph extension                          |
  93 +--------------------------------------------------------+-----------------------------------------+
  94 | ``s3:ObjectLifecycle:Expiration:AbortMultipartUpload`` | Defined, Ceph extension (not generated) |
  95 +--------------------------------------------------------+-----------------------------------------+
  96 | ``s3:ObjectLifecycle:Transition:Current``              | Ceph extension                          |
  97 +--------------------------------------------------------+-----------------------------------------+
  98 | ``s3:ObjectLifecycle:Transition:NonCurrent``           | Ceph extension                          |
  99 +--------------------------------------------------------+-----------------------------------------+
 100 | ``s3:ObjectSynced:*``                                  | Ceph extension                          |
 101 +--------------------------------------------------------+-----------------------------------------+
 102 | ``s3:ObjectSynced:Create``                             | Ceph Extension                          |
 103 +--------------------------------------------------------+-----------------------------------------+
 104 | ``s3:ObjectSynced:Delete``                             | Defined, Ceph extension (not generated) |
 105 +--------------------------------------------------------+-----------------------------------------+
 106 | ``s3:ObjectSynced:DeletionMarkerCreated``              | Defined, Ceph extension (not generated) |
 107 +--------------------------------------------------------+-----------------------------------------+
 108 | ``s3:ObjectRestore:Post``                              | Not applicable                          |
 109 +--------------------------------------------------------+-----------------------------------------+
 110 | ``s3:ObjectRestore:Complete``                          | Not applicable                          |
 111 +--------------------------------------------------------+-----------------------------------------+
 112 | ``s3:ReducedRedundancyLostObject``                     | Not applicable                          |
 113 +--------------------------------------------------------+-----------------------------------------+
 114
 115 .. note::
 116
 117    The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object
 118
 119 .. note::
 120
 121    In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process.
 122
 123 .. note::
 124
 125    The ``s3:ObjectSynced:Create`` event is sent when an object successfully syncs to a zone. It must be explicitly set for each zone.
 126
 127 Topic Configuration
 128 -------------------
 129 In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_.
 130 Note that most of the API is not applicable to Ceph, and only the following actions are implemented:
 131
 132  - ``CreateTopic``
 133  - ``DeleteTopic``
 134  - ``ListTopics``
 135
 136 We also have the following extensions to topic configuration:
 137
 138  - In ``GetTopic`` we allow fetching a specific topic, instead of all user topics
 139  - In ``CreateTopic``
 140
 141   - we allow setting endpoint attributes
 142   - we allow setting opaque data that will be sent to the endpoint in the notification
 143
 144
 145 .. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html
 146 .. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
 147 .. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
 148 .. _`Bucket Notifications`: ../notifications
 149 .. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/main/examples/rgw/boto3