ceph/doc/radosgw/s3-notification-compatibility.rst

   1 =====================================
   2 S3 Bucket Notifications Compatibility
   3 =====================================
   4
   5 Ceph's `Bucket Notifications`_ and `PubSub Module`_ APIs follow `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below.
   6
   7
   8 .. note::
   9
  10     Compatibility is different depending on which of the above mechanism is used
  11
  12 Supported Destination
  13 ---------------------
  14
  15 AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations).
  16 Currently, we support: **HTTP/S**, **Kafka** and **AMQP**. And also support pulling and acking of events stored in Ceph (as an internal destination).
  17
  18 We are using the **SNS** ARNs to represent the **HTTP/S**, **Kafka** and **AMQP** destinations.
  19
  20 Notification Configuration XML
  21 ------------------------------
  22
  23 Following tags (and the tags inside them) are not supported:
  24
  25 +-----------------------------------+----------------------------------------------+
  26 | Tag                               | Remaks                                       |
  27 +===================================+==============================================+
  28 | ``<QueueConfiguration>``          | not needed, we treat all destinations as SNS |
  29 +-----------------------------------+----------------------------------------------+
  30 | ``<CloudFunctionConfiguration>``  | not needed, we treat all destinations as SNS |
  31 +-----------------------------------+----------------------------------------------+
  32
  33 REST API Extension
  34 ------------------
  35
  36 Ceph's bucket notification API has the following extensions:
  37
  38 - Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb
  39
  40  - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket
  41
  42 - Getting the information on a specific notification (when more than one exists on a bucket)
  43
  44   - In S3, it is only possible to fetch all notifications on a bucket
  45
  46 - In addition to filtering based on prefix/suffix of object keys we support:
  47
  48   - Filtering based on regular expression matching
  49
  50   - Filtering based on metadata attributes attached to the object
  51
  52   - Filtering based on object tags
  53
  54 - Each one of the additional filters extends the S3 API and using it will require extension of the client SDK (unless you are using plain HTTP).
  55
  56 - Filtering overlapping is allowed, so that same event could be sent as different notification
  57
  58
  59 Unsupported Fields in the Event Record
  60 --------------------------------------
  61
  62 The records sent for bucket notification follow format described in: `Event Message Structure`_.
  63 However, the following fields may be sent empty, under the different deployment options (Notification/PubSub):
  64
  65 +----------------------------------------+--------------+---------------+------------------------------------------------------------+
  66 | Field                                  | Notification | PubSub        | Description                                                |
  67 +========================================+==============+===============+============================================================+
  68 | ``userIdentity.principalId``           | Supported    | Not Supported | The identity of the user that triggered the event          |
  69 +----------------------------------------+--------------+---------------+------------------------------------------------------------+
  70 | ``requestParameters.sourceIPAddress``  |         Not Supported        | The IP address of the client that triggered the event      |
  71 +----------------------------------------+--------------+---------------+------------------------------------------------------------+
  72 | ``requestParameters.x-amz-request-id`` | Supported    | Not Supported | The request id that triggered the event                    |
  73 +----------------------------------------+--------------+---------------+------------------------------------------------------------+
  74 | ``requestParameters.x-amz-id-2``       | Supported    | Not Supported | The IP address of the RGW on which the event was triggered |
  75 +----------------------------------------+--------------+---------------+------------------------------------------------------------+
  76 | ``s3.object.size``                     | Supported    | Not Supported | The size of the object                                     |
  77 +----------------------------------------+--------------+---------------+------------------------------------------------------------+
  78
  79 Event Types
  80 -----------
  81
  82 +------------------------------------------------+-----------------+-------------------------------------------+
  83 | Event                                          | Notification    | PubSub                                    |
  84 +================================================+=================+===========================================+
  85 | ``s3:ObjectCreated:*``                         | Supported                                                   |
  86 +------------------------------------------------+-----------------+-------------------------------------------+
  87 | ``s3:ObjectCreated:Put``                       | Supported       | Supported at ``s3:ObjectCreated:*`` level |
  88 +------------------------------------------------+-----------------+-------------------------------------------+
  89 | ``s3:ObjectCreated:Post``                      | Supported       | Not Supported                             |
  90 +------------------------------------------------+-----------------+-------------------------------------------+
  91 | ``s3:ObjectCreated:Copy``                      | Supported       | Supported at ``s3:ObjectCreated:*`` level |
  92 +------------------------------------------------+-----------------+-------------------------------------------+
  93 | ``s3:ObjectCreated:CompleteMultipartUpload``   | Supported       | Supported at ``s3:ObjectCreated:*`` level |
  94 +------------------------------------------------+-----------------+-------------------------------------------+
  95 | ``s3:ObjectRemoved:*``                         | Supported       | Supported only the specific events below  |
  96 +------------------------------------------------+-----------------+-------------------------------------------+
  97 | ``s3:ObjectRemoved:Delete``                    | Supported                                                   |
  98 +------------------------------------------------+-----------------+-------------------------------------------+
  99 | ``s3:ObjectRemoved:DeleteMarkerCreated``       | Supported                                                   |
 100 +------------------------------------------------+-----------------+-------------------------------------------+
 101 | ``s3:ObjectLifecycle:Expiration:Current``      | Supported, Ceph extension                                   |
 102 +------------------------------------------------+-----------------+-------------------------------------------+
 103 | ``s3:ObjectLifecycle:Expiration:NonCurrent``   | Supported, Ceph extension                                   |
 104 +------------------------------------------------+-----------------+-------------------------------------------+
 105 | ``s3:ObjectLifecycle:Expiration:DeleteMarker`` | Supported, Ceph extension                                   |
 106 +------------------------------------------------+-----------------+-------------------------------------------+
 107 | ``s3:ObjectLifecycle:Expiration:AbortMultipartUpload`` | Defined, Ceph extension (not generated)             |
 108 +------------------------------------------------+-----------------+-------------------------------------------+
 109 | ``s3:ObjectLifecycle:Transition:Current``      | Supported, Ceph extension                                   |
 110 +------------------------------------------------+-----------------+-------------------------------------------+
 111 | ``s3:ObjectLifecycle:Transition:NonCurrent``   | Supported, Ceph extension                                   |
 112 +------------------------------------------------+-----------------+-------------------------------------------+
 113 | ``s3:ObjectRestore:Post``                      | Not applicable to Ceph                                      |
 114 +------------------------------------------------+-----------------+-------------------------------------------+
 115 | ``s3:ObjectRestore:Complete``                  | Not applicable to Ceph                                      |
 116 +------------------------------------------------+-----------------+-------------------------------------------+
 117 | ``s3:ReducedRedundancyLostObject``             | Not applicable to Ceph                                      |
 118 +----------------------------------------------+-----------------+---------------------------------------------+
 119
 120 .. note::
 121
 122    The ``s3:ObjectRemoved:DeleteMarkerCreated`` event presents information on the latest version of the object
 123
 124 .. note::
 125
 126    In case of multipart upload, an ``ObjectCreated:CompleteMultipartUpload`` notification will be sent at the end of the process.
 127
 128 Topic Configuration
 129 -------------------
 130 In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_.
 131 Note that most of the API is not applicable to Ceph, and only the following actions are implemented:
 132
 133  - ``CreateTopic``
 134  - ``DeleteTopic``
 135  - ``ListTopics``
 136
 137 We also have the following extensions to topic configuration:
 138
 139  - In ``GetTopic`` we allow fetching a specific topic, instead of all user topics
 140  - In ``CreateTopic``
 141
 142   - we allow setting endpoint attributes
 143   - we allow setting opaque data that will be sent to the endpoint in the notification
 144
 145
 146 .. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html
 147 .. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
 148 .. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
 149 .. _`PubSub Module`: ../pubsub-module
 150 .. _`Bucket Notifications`: ../notifications
 151 .. _`boto3 SDK filter extensions`: https://github.com/ceph/ceph/tree/master/examples/boto3