ceph/doc/radosgw/notifications.rst

   1 ====================
   2 Bucket Notifications
   3 ====================
   4
   5 .. versionadded:: Nautilus
   6
   7 .. contents::
   8
   9 Bucket notifications provide a mechanism for sending information out of the radosgw when certain events are happening on the bucket.
  10 Currently, notifications could be sent to HTTP and AMQP0.9.1 endpoints.
  11
  12 Note, that if the events should be stored in Ceph, in addition, or instead of being pushed to an endpoint,
  13 the `PubSub Module`_ should be used instead of the bucket notification mechanism.
  14
  15 A user can create different topics. A topic entity is defined by its user and its name. A
  16 user can only manage its own topics, and can only associate them with buckets it owns.
  17
  18 In order to send notifications for events for a specific bucket, a notification entity needs to be created. A
  19 notification can be created on a subset of event types, or for all event types (default).
  20 The notification may also filter out events based on preffix/suffix and/or regular expression matching of the keys. As well as,
  21 on the metadata attributes attached to the object.
  22 There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications.
  23
  24 REST API has been defined to provide configuration and control interfaces for the bucket notification
  25 mechanism. This API is similar to the one defined as S3-compatible API of the pubsub sync module.
  26
  27 .. toctree::
  28    :maxdepth: 1
  29
  30    S3 Bucket Notification Compatibility <s3-notification-compatibility>
  31
  32 Notificatios Performance Stats
  33 ------------------------------
  34 Same counters are shared between the pubsub sync module and the bucket notification mechanism.
  35
  36 - ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them
  37 - ``pubsub_event_lost``: running counter of events that had topics associated with them but that were not pushed to any of the endpoints
  38 - ``pubsub_push_ok``: running counter, for all notifications, of events successfully pushed to their endpoint
  39 - ``pubsub_push_fail``: running counter, for all notifications, of events failed to be pushed to their endpoint
  40 - ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet
  41
  42 .. note::
  43
  44     ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while:
  45     ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per push action on each notification.
  46
  47 Bucket Notification REST API
  48 ----------------------------
  49
  50 Topics
  51 ~~~~~~
  52
  53 Create a Topic
  54 ``````````````
  55
  56 This will create a new topic. The topic should be provided with push endpoint parameters that would be used later
  57 when a notification is created.
  58 Upon successful request, the response will include the topic ARN that could be later used to reference this topic in the notification request.
  59 To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values.
  60
  61 .. tip:: Any notification already associated with the topic needs to be re-created for the topic update to take effect
  62
  63 ::
  64
  65    POST
  66    Action=CreateTopic
  67    &Name=<topic-name>
  68    &push-endpoint=<endpoint>
  69    [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>]
  70    [&Attributes.entry.2.key=amqp-sck-level&Attributes.entry.2.value=ack-level]
  71    &Attributes.entry.3.key=verify-sll&Attributes.entry.3.value=true|false]
  72
  73 Request parameters:
  74
  75 - push-endpoint: URI of endpoint to send push notification to
  76
  77  - URI schema is: ``http[s]|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]``
  78  - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP)
  79  - Default values for HTTP/S: no user/password, port 80/443
  80  - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/"
  81
  82 - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default)
  83 - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics
  84 - amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist:
  85
  86  - "none" - message is considered "delivered" if sent to broker
  87  - "broker" message is considered "delivered" if acked by broker
  88
  89 .. note::
  90
  91     - The key/value of a specific parameter does not have to reside in the same line, or in any specific order, but must use the same index
  92     - Attribute indexing does not need to be sequntial or start from any specific value
  93     - `AWS Create Topic`_ has detailed explanation on endpoint attributes format. However, in our case different keys and values are used
  94
  95 The response will have the following format:
  96
  97 ::
  98
  99     <CreateTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
 100         <CreateTopicResult>
 101             <TopicArn></TopicArn>
 102         </CreateTopicResult>
 103         <ResponseMetadata>
 104             <RequestId></RequestId>
 105         </ResponseMetadata>
 106     </CreateTopicResponse>
 107
 108 The topic ARN in the response will have the following format:
 109
 110 ::
 111
 112    arn:aws:sns:<zone-group>:<tenant>:<topic>
 113
 114 Get Topic Information
 115 `````````````````````
 116
 117 Returns information about specific topic. This includes push-endpoint information, if provided.
 118
 119 ::
 120
 121    POST
 122    Action=GetTopic&TopicArn=<topic-arn>
 123
 124 Response will have the following format:
 125
 126 ::
 127
 128     <GetTopicResponse>
 129         <GetTopicRersult>
 130             <Topic>
 131                 <User></User>
 132                 <Name></Name>
 133                 <EndPoint>
 134                     <EndpointAddress></EndpointAddress>
 135                     <EndpointArgs></EndpointArgs>
 136                     <EndpointTopic></EndpointTopic>
 137                 </EndPoint>
 138                 <TopicArn></TopicArn>
 139             </Topic>
 140         </GetTopicResult>
 141         <ResponseMetadata>
 142             <RequestId></RequestId>
 143         </ResponseMetadata>
 144     </GetTopicResponse>
 145
 146 - User: name of the user that created the topic
 147 - Name: name of the topic
 148 - EndPoinjtAddress: the push-endpoint URL
 149 - EndPointArgs: the push-endpoint args
 150 - EndpointTopic: the topic name that should be sent to the endpoint (mat be different than the above topic name)
 151 - TopicArn: topic ARN
 152
 153 Delete Topic
 154 ````````````
 155
 156 ::
 157
 158    POST
 159    Action=DeleteTopic&TopicArn=<topic-arn>
 160
 161 Delete the specified topic. Note that deleting a deleted topic should result with no-op and not a failure.
 162
 163 The response will have the following format:
 164
 165 ::
 166
 167     <DeleteTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
 168         <ResponseMetadata>
 169             <RequestId></RequestId>
 170         </ResponseMetadata>
 171     </DeleteTopicResponse>
 172
 173 List Topics
 174 ```````````
 175
 176 List all topics that user defined.
 177
 178 ::
 179
 180    POST
 181    Action=ListTopics
 182
 183 Response will have the following format:
 184
 185 ::
 186
 187     <ListTopicdResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
 188         <ListTopicsRersult>
 189             <Topics>
 190                 <member>
 191                     <User></User>
 192                     <Name></Name>
 193                     <EndPoint>
 194                         <EndpointAddress></EndpointAddress>
 195                         <EndpointArgs></EndpointArgs>
 196                         <EndpointTopic></EndpointTopic>
 197                     </EndPoint>
 198                     <TopicArn></TopicArn>
 199                 </member>
 200             </Topics>
 201         </ListTopicsResult>
 202         <ResponseMetadata>
 203             <RequestId></RequestId>
 204         </ResponseMetadata>
 205     </ListTopicsResponse>
 206
 207 Notifications
 208 ~~~~~~~~~~~~~
 209
 210 Detailed under: `Bucket Operations`_.
 211
 212 .. note::
 213
 214     - "Abort Multipart Upload" request does not emit a notification
 215     - "Delete Multiple Objects" request does not emit a notification
 216     - Both "Initiate Multipart Upload" and "POST Object" requests will emit an ``s3:ObjectCreated:Post`` notification
 217
 218
 219 Events
 220 ~~~~~~
 221
 222 The events are in JSON format (regardless of the actual endpoint), and share the same structure as the S3-compatible events
 223 pushed or pulled using the pubsub sync module.
 224
 225 ::
 226
 227    {"Records":[
 228        {
 229            "eventVersion":"2.1"
 230            "eventSource":"aws:s3",
 231            "awsRegion":"",
 232            "eventTime":"",
 233            "eventName":"",
 234            "userIdentity":{
 235                "principalId":""
 236            },
 237            "requestParameters":{
 238                "sourceIPAddress":""
 239            },
 240            "responseElements":{
 241                "x-amz-request-id":"",
 242                "x-amz-id-2":""
 243            },
 244            "s3":{
 245                "s3SchemaVersion":"1.0",
 246                "configurationId":"",
 247                "bucket":{
 248                    "name":"",
 249                    "ownerIdentity":{
 250                        "principalId":""
 251                    },
 252                    "arn":"",
 253                    "id:""
 254                },
 255                "object":{
 256                    "key":"",
 257                    "size":"",
 258                    "eTag":"",
 259                    "versionId":"",
 260                    "sequencer": "",
 261                    "metadata":""
 262                }
 263            },
 264            "eventId":"",
 265        }
 266    ]}
 267
 268 - awsRegion: zonegroup
 269 - eventTime: timestamp indicating when the event was triggered
 270 - eventName: for list of supported events see: `S3 Notification Compatibility`_
 271 - userIdentity.principalId: user that triggered the change
 272 - requestParameters.sourceIPAddress: not supported
 273 - responseElements.x-amz-request-id: request ID of the original change
 274 - responseElements.x_amz_id_2: RGW on which the change was made
 275 - s3.configurationId: notification ID that created the event
 276 - s3.bucket.name: name of the bucket
 277 - s3.bucket.ownerIdentity.principalId: owner of the bucket
 278 - s3.bucket.arn: ARN of the bucket
 279 - s3.bucket.id: Id of the bucket (an extension to the S3 notification API)
 280 - s3.object.key: object key
 281 - s3.object.size: object size
 282 - s3.object.eTag: object etag
 283 - s3.object.version: object version in case of versioned bucket
 284 - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format)
 285 - s3.object.metadata: any metadata set on the object sent as: ``x-amz-meta-`` (an extension to the S3 notification API)
 286 - s3.eventId: not supported (an extension to the S3 notification API)
 287
 288 .. _PubSub Module : ../pubsub-module
 289 .. _S3 Notification Compatibility: ../s3-notification-compatibility
 290 .. _AWS Create Topic: https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html
 291 .. _Bucket Operations: ../s3/bucketops