ceph/doc/radosgw/notifications.rst

   1 ====================
   2 Bucket Notifications
   3 ====================
   4
   5 .. versionadded:: Nautilus
   6
   7 .. contents::
   8
   9 Bucket notifications provide a mechanism for sending information out of radosgw
  10 when certain events happen on the bucket. Notifications can be sent to HTTP
  11 endpoints, AMQP0.9.1 endpoints, and Kafka endpoints.
  12
  13 A user can create topics. A topic entity is defined by its name and is "per
  14 tenant". A user can associate its topics (via notification configuration) only
  15 with buckets it owns.
  16
  17 A notification entity must be created in order to send event notifications for
  18 a specific bucket. A notification entity can be created either for a subset
  19 of event types or for all event types (which is the default). The
  20 notification may also filter out events based on matches of the prefixes and
  21 suffixes of (1) the keys, (2) the metadata attributes attached to the object,
  22 or (3) the object tags. Regular-expression matching can also be used on these
  23 to create filters. There can be multiple notifications for any specific topic,
  24 and the same topic can used for multiple notifications.
  25
  26 REST API has been defined so as to provide configuration and control interfaces
  27 for the bucket notification mechanism.
  28
  29 .. toctree::
  30    :maxdepth: 1
  31
  32    S3 Bucket Notification Compatibility <s3-notification-compatibility>
  33
  34 .. note:: To enable bucket notifications API, the `rgw_enable_apis` configuration parameter should contain: "notifications".
  35
  36 Notification Reliability
  37 ------------------------
  38
  39 Notifications can be sent synchronously or asynchronously. This section
  40 describes the latency and reliability that you should expect for synchronous
  41 and asynchronous notifications.
  42
  43 Synchronous Notifications
  44 ~~~~~~~~~~~~~~~~~~~~~~~~~
  45
  46 Notifications can be sent synchronously, as part of the operation that
  47 triggered them. In this mode, the operation is acknowledged (acked) only after
  48 the notification is sent to the topic's configured endpoint. This means that
  49 the round trip time of the notification (the time it takes to send the
  50 notification to the topic's endpoint plus the time it takes to receive the
  51 acknowledgement) is added to the latency of the operation itself.
  52
  53 .. note:: The original triggering operation is considered successful even if
  54    the notification fails with an error, cannot be delivered, or times out.
  55
  56 Asynchronous Notifications
  57 ~~~~~~~~~~~~~~~~~~~~~~~~~~
  58
  59 Notifications can be sent asynchronously. They are committed into persistent
  60 storage and then asynchronously sent to the topic's configured endpoint. In
  61 this case, the only latency added to the original operation is the latency
  62 added when the notification is committed to persistent storage.
  63
  64 .. note:: If the notification fails with an error, cannot be delivered, or
  65    times out, it is retried until it is successfully acknowledged.
  66
  67 .. tip:: To minimize the latency added by asynchronous notification, we
  68    recommended placing the "log" pool on fast media.
  69
  70
  71 Topic Management via CLI
  72 ------------------------
  73
  74 Fetch the configuration of all topics associated with tenants by running the
  75 following command:
  76
  77 .. prompt:: bash #
  78
  79    radosgw-admin topic list [--tenant={tenant}]
  80
  81
  82 Fetch the configuration of a specific topic by running the following command:
  83
  84 .. prompt:: bash #
  85
  86    radosgw-admin topic get --topic={topic-name} [--tenant={tenant}]
  87
  88
  89 Remove a topic by running the following command:
  90
  91 .. prompt:: bash #
  92
  93    radosgw-admin topic rm --topic={topic-name} [--tenant={tenant}]
  94
  95
  96 Notification Performance Statistics
  97 -----------------------------------
  98
  99 - ``pubsub_event_triggered``: a running counter of events that have at least one topic associated with them
 100 - ``pubsub_event_lost``: a running counter of events that had topics associated with them, but that were not pushed to any of the endpoints
 101 - ``pubsub_push_ok``: a running counter, for all notifications, of events successfully pushed to their endpoints
 102 - ``pubsub_push_fail``: a running counter, for all notifications, of events that failed to be pushed to their endpoints
 103 - ``pubsub_push_pending``: the gauge value of events pushed to an endpoint but not acked or nacked yet
 104
 105 .. note::
 106
 107     ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per
 108     event on each notification, but ``pubsub_push_ok`` and ``pubsub_push_fail``
 109     are incremented per push action on each notification.
 110
 111 Bucket Notification REST API
 112 ----------------------------
 113
 114 Topics
 115 ~~~~~~
 116
 117 .. note::
 118
 119     In all topic actions, the parameters are URL-encoded and sent in the
 120     message body using this content type:
 121     ``application/x-www-form-urlencoded``.
 122
 123
 124 .. _Create a Topic:
 125
 126 Create a Topic
 127 ``````````````
 128
 129 This creates a new topic. Provide the topic with push endpoint parameters,
 130 which will be used later when a notification is created. A response is
 131 generated. A successful response includes the topic's `ARN
 132 <https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html>`_
 133 (the "Amazon Resource Name", a unique identifier used to reference the topic).
 134 To update a topic, use the same command that you used to create it (but when
 135 updating, use the name of an existing topic and different endpoint values).
 136
 137 .. tip:: Any notification already associated with the topic must be re-created
 138    in order for the topic to update.
 139
 140 ::
 141
 142    POST
 143
 144    Action=CreateTopic
 145    &Name=<topic-name>
 146    [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>]
 147    [&Attributes.entry.2.key=amqp-ack-level&Attributes.entry.2.value=none|broker|routable]
 148    [&Attributes.entry.3.key=verify-ssl&Attributes.entry.3.value=true|false]
 149    [&Attributes.entry.4.key=kafka-ack-level&Attributes.entry.4.value=none|broker]
 150    [&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true|false]
 151    [&Attributes.entry.6.key=ca-location&Attributes.entry.6.value=<file path>]
 152    [&Attributes.entry.7.key=OpaqueData&Attributes.entry.7.value=<opaque data>]
 153    [&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=<endpoint>]
 154    [&Attributes.entry.9.key=persistent&Attributes.entry.9.value=true|false]
 155    [&Attributes.entry.10.key=cloudevents&Attributes.entry.10.value=true|false]
 156    [&Attributes.entry.11.key=mechanism&Attributes.entry.11.value=<mechanism>]
 157
 158 Request parameters:
 159
 160 - push-endpoint: This is the URI of an endpoint to send push notifications to.
 161 - OpaqueData: Opaque data is set in the topic configuration and added to all
 162   notifications that are triggered by the topic.
 163 - persistent: This indicates whether notifications to this endpoint are
 164   persistent (=asynchronous) or not persistent. (This is "false" by default.)
 165
 166 - HTTP endpoint
 167
 168  - URI: ``http[s]://<fqdn>[:<port]``
 169  - port: This defaults to 80 for HTTP and 443 for HTTPS.
 170  - verify-ssl: This indicates whether the server certificate is validated by
 171    the client. (This is "true" by default.)
 172  - cloudevents: This indicates whether the HTTP header should contain
 173    attributes according to the `S3 CloudEvents Spec`_. (This is "false" by
 174    default.)
 175
 176 - AMQP0.9.1 endpoint
 177
 178  - URI: ``amqp[s]://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]``
 179  - user/password: This defaults to "guest/guest".
 180  - user/password: This must be provided only over HTTPS. Topic creation
 181    requests will otherwise be rejected.
 182  - port: This defaults to 5672 for unencrypted connections and 5671 for
 183    SSL-encrypted connections.
 184  - vhost: This defaults to "/".
 185  - verify-ssl: This indicates whether the server certificate is validated by
 186    the client. (This is "true" by default.)
 187  - If ``ca-location`` is provided and a secure connection is used, the
 188    specified CA will be used to authenticate the broker. The default CA will
 189    not be used.
 190  - amqp-exchange: The exchanges must exist and must be able to route messages
 191    based on topics. This parameter is mandatory.
 192  - amqp-ack-level: No end2end acking is required. Messages may persist in the
 193    broker before being delivered to their final destinations. Three ack methods
 194    exist:
 195
 196   - "none": The message is considered "delivered" if it is sent to the broker.
 197   - "broker": The message is considered "delivered" if it is acked by the broker (default).
 198   - "routable": The message is considered "delivered" if the broker can route to a consumer.
 199
 200 .. tip:: The topic-name (see :ref:`Create a Topic`) is used for the
 201    AMQP topic ("routing key" for a topic exchange).
 202
 203 - Kafka endpoint
 204
 205  - URI: ``kafka://[<user>:<password>@]<fqdn>[:<port]``
 206  - ``use-ssl``: If this is set to "true", a secure connection is used to
 207    connect to the broker. (This is "false" by default.)
 208  - ``ca-location``: If this is provided and a secure connection is used, the
 209    specified CA will be used instead of the default CA to authenticate the
 210    broker.
 211  - user/password may be provided over HTTPS. If not, the config parameter
 212    `rgw_allow_notification_secrets_in_cleartext` must be `true` in order to create topic
 213  - user/password may be provided along with ``use-ssl``.
 214    The broker credentials will otherwise be sent over insecure transport
 215  - ``mechanism`` may be provided together with user/password (default: ``PLAIN``).
 216    The supported SASL mechanisms are:
 217
 218   - PLAIN
 219   - SCRAM-SHA-256
 220   - SCRAM-SHA-512
 221   - GSSAPI
 222   - OAUTHBEARER
 223
 224  - port: This defaults to 9092.
 225  - kafka-ack-level: No end2end acking is required. Messages may persist in the
 226    broker before being delivered to their final destinations. Two ack methods
 227    exist:
 228
 229   - "none": Messages are considered "delivered" if sent to the broker.
 230   - "broker": Messages are considered "delivered" if acked by the broker. (This
 231     is the default.)
 232
 233 .. note::
 234
 235     - The key-value pair of a specific parameter need not reside in the same
 236       line as the parameter, and need not appear in any specific order, but it
 237       must use the same index.
 238     - Attribute indexing need not be sequential and need not start from any
 239       specific value.
 240     - `AWS Create Topic`_ provides a detailed explanation of the endpoint
 241       attributes format. In our case, however, different keys and values are
 242       used.
 243
 244 The response has the following format:
 245
 246 ::
 247
 248     <CreateTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
 249         <CreateTopicResult>
 250             <TopicArn></TopicArn>
 251         </CreateTopicResult>
 252         <ResponseMetadata>
 253             <RequestId></RequestId>
 254         </ResponseMetadata>
 255     </CreateTopicResponse>
 256
 257 The topic `ARN
 258 <https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html>`_
 259 in the response has the following format:
 260
 261 ::
 262
 263    arn:aws:sns:<zone-group>:<tenant>:<topic>
 264
 265 Get Topic Attributes
 266 ````````````````````
 267
 268 This returns information about a specific topic. This includes push-endpoint
 269 information, if provided.
 270
 271 ::
 272
 273    POST
 274
 275    Action=GetTopicAttributes
 276    &TopicArn=<topic-arn>
 277
 278 The response has the following format:
 279
 280 ::
 281
 282     <GetTopicAttributesResponse>
 283         <GetTopicAttributesResult>
 284             <Attributes>
 285                 <entry>
 286                     <key>User</key>
 287                     <value></value>
 288                 </entry>
 289                 <entry>
 290                     <key>Name</key>
 291                     <value></value>
 292                 </entry>
 293                 <entry>
 294                     <key>EndPoint</key>
 295                     <value></value>
 296                 </entry>
 297                 <entry>
 298                     <key>TopicArn</key>
 299                     <value></value>
 300                 </entry>
 301                 <entry>
 302                     <key>OpaqueData</key>
 303                     <value></value>
 304                 </entry>
 305             </Attributes>
 306         </GetTopicAttributesResult>
 307         <ResponseMetadata>
 308             <RequestId></RequestId>
 309         </ResponseMetadata>
 310     </GetTopicAttributesResponse>
 311
 312 - User: the name of the user that created the topic.
 313 - Name: the name of the topic.
 314 - EndPoint: The JSON-formatted endpoint parameters, including:
 315    - EndpointAddress: The push-endpoint URL.
 316    - EndpointArgs: The push-endpoint args.
 317    - EndpointTopic: The topic name to be sent to the endpoint (can be different
 318      than the above topic name).
 319    - HasStoredSecret: This is "true" if the endpoint URL contains user/password
 320      information. In this case, the request must be made over HTTPS. The "topic
 321      get" request will otherwise be rejected.
 322    - Persistent: This is "true" if the topic is persistent.
 323 - TopicArn: topic `ARN
 324   <https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html>`_.
 325 - OpaqueData: The opaque data set on the topic.
 326
 327 Get Topic Information
 328 `````````````````````
 329
 330 This returns information about a specific topic. This includes push-endpoint
 331 information, if provided.  Note that this API is now deprecated in favor of the
 332 AWS compliant `GetTopicAttributes` API.
 333
 334 ::
 335
 336    POST
 337
 338    Action=GetTopic
 339    &TopicArn=<topic-arn>
 340
 341 The response has the following format:
 342
 343 ::
 344
 345     <GetTopicResponse>
 346         <GetTopicResult>
 347             <Topic>
 348                 <User></User>
 349                 <Name></Name>
 350                 <EndPoint>
 351                     <EndpointAddress></EndpointAddress>
 352                     <EndpointArgs></EndpointArgs>
 353                     <EndpointTopic></EndpointTopic>
 354                     <HasStoredSecret></HasStoredSecret>
 355                     <Persistent></Persistent>
 356                 </EndPoint>
 357                 <TopicArn></TopicArn>
 358                 <OpaqueData></OpaqueData>
 359             </Topic>
 360         </GetTopicResult>
 361         <ResponseMetadata>
 362             <RequestId></RequestId>
 363         </ResponseMetadata>
 364     </GetTopicResponse>
 365
 366 - User: The name of the user that created the topic.
 367 - Name: The name of the topic.
 368 - EndpointAddress: The push-endpoint URL.
 369 - EndpointArgs: The push-endpoint args.
 370 - EndpointTopic: The topic name to be sent to the endpoint (which can be
 371   different than the above topic name).
 372 - HasStoredSecret: This is "true" if the endpoint URL contains user/password
 373   information. In this case, the request must be made over HTTPS. The "topic
 374   get" request will otherwise be rejected.
 375 - Persistent: "true" if topic is persistent.
 376 - TopicArn: topic `ARN
 377   <https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html>`_.
 378 - OpaqueData: the opaque data set on the topic.
 379
 380 Delete Topic
 381 ````````````
 382
 383 ::
 384
 385    POST
 386
 387    Action=DeleteTopic
 388    &TopicArn=<topic-arn>
 389
 390 This deletes the specified topic.
 391
 392 .. note::
 393
 394   - Deleting an unknown notification (for example, double delete) is not
 395     considered an error.
 396   - Deleting a topic does not automatically delete all notifications associated
 397     with it.
 398
 399 The response has the following format:
 400
 401 ::
 402
 403     <DeleteTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
 404         <ResponseMetadata>
 405             <RequestId></RequestId>
 406         </ResponseMetadata>
 407     </DeleteTopicResponse>
 408
 409 List Topics
 410 ```````````
 411
 412 List all topics associated with a tenant.
 413
 414 ::
 415
 416    POST
 417
 418    Action=ListTopics
 419
 420 The response has the following format:
 421
 422 ::
 423
 424     <ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
 425         <ListTopicsResult>
 426             <Topics>
 427                 <member>
 428                     <User></User>
 429                     <Name></Name>
 430                     <EndPoint>
 431                         <EndpointAddress></EndpointAddress>
 432                         <EndpointArgs></EndpointArgs>
 433                         <EndpointTopic></EndpointTopic>
 434                     </EndPoint>
 435                     <TopicArn></TopicArn>
 436                     <OpaqueData></OpaqueData>
 437                 </member>
 438             </Topics>
 439         </ListTopicsResult>
 440         <ResponseMetadata>
 441             <RequestId></RequestId>
 442         </ResponseMetadata>
 443     </ListTopicsResponse>
 444
 445 - If the endpoint URL contains user/password information in any part of the
 446   topic, the request must be made over HTTPS. The "topic list" request will
 447   otherwise be rejected.
 448
 449 Notifications
 450 ~~~~~~~~~~~~~
 451
 452 Detailed under: `Bucket Operations`_.
 453
 454 .. note::
 455
 456     - "Abort Multipart Upload" request does not emit a notification
 457     - Both "Initiate Multipart Upload" and "POST Object" requests will emit an ``s3:ObjectCreated:Post`` notification
 458
 459 Events
 460 ~~~~~~
 461
 462 Events are in JSON format (regardless of the actual endpoint), and are S3-compatible.
 463 For example:
 464
 465 ::
 466
 467    {"Records":[
 468        {
 469            "eventVersion":"2.1",
 470            "eventSource":"ceph:s3",
 471            "awsRegion":"zonegroup1",
 472            "eventTime":"2019-11-22T13:47:35.124724Z",
 473            "eventName":"ObjectCreated:Put",
 474            "userIdentity":{
 475                "principalId":"tester"
 476            },
 477            "requestParameters":{
 478                "sourceIPAddress":""
 479            },
 480            "responseElements":{
 481                "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
 482                "x-amz-id-2":"14d2-zone1-zonegroup1"
 483            },
 484            "s3":{
 485                "s3SchemaVersion":"1.0",
 486                "configurationId":"mynotif1",
 487                "bucket":{
 488                    "name":"mybucket1",
 489                    "ownerIdentity":{
 490                        "principalId":"tester"
 491                    },
 492                    "arn":"arn:aws:s3:zonegroup1::mybucket1",
 493                    "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
 494                },
 495                "object":{
 496                    "key":"myimage1.jpg",
 497                    "size":"1024",
 498                    "eTag":"37b51d194a7513e45b56f6524f2d51f2",
 499                    "versionId":"",
 500                    "sequencer": "F7E6D75DC742D108",
 501                    "metadata":[],
 502                    "tags":[]
 503                }
 504            },
 505            "eventId":"",
 506            "opaqueData":"me@example.com"
 507        }
 508    ]}
 509
 510 - awsRegion: The zonegroup.
 511 - eventTime: The timestamp, indicating when the event was triggered.
 512 - eventName: For the list of supported events see: `S3 Notification
 513   Compatibility`_. Note that eventName values do not start with the `s3:`
 514   prefix.
 515 - userIdentity.principalId: The user that triggered the change.
 516 - requestParameters.sourceIPAddress: not supported
 517 - responseElements.x-amz-request-id: The request ID of the original change.
 518 - responseElements.x_amz_id_2: The RGW on which the change was made.
 519 - s3.configurationId: The notification ID that created the event.
 520 - s3.bucket.name: The name of the bucket.
 521 - s3.bucket.ownerIdentity.principalId: The owner of the bucket.
 522 - s3.bucket.arn: The ARN of the bucket.
 523 - s3.bucket.id: The ID of the bucket. (This is an extension to the S3
 524   notification API.)
 525 - s3.object.key: The object key.
 526 - s3.object.size: The object size.
 527 - s3.object.eTag: The object etag.
 528 - s3.object.versionId: The object version, if the bucket is versioned. When a
 529   copy is made, it includes the version of the target object. When a delete
 530   marker is created, it includes the version of the delete marker.
 531 - s3.object.sequencer: The monotonically-increasing identifier of the "change
 532   per object" (hexadecimal format).
 533 - s3.object.metadata: Any metadata set on the object that is sent as
 534   ``x-amz-meta-`` (that is, any metadata set on the object that is sent as an
 535   extension to the S3 notification API).
 536 - s3.object.tags: Any tags set on the object. (This is an extension to the S3
 537   notification API.)
 538 - s3.eventId: The unique ID of the event, which could be used for acking. (This
 539   is an extension to the S3 notification API.)
 540 - s3.opaqueData: This means that "opaque data" is set in the topic configuration
 541   and is added to all notifications triggered by the topic. (This is an
 542   extension to the S3 notification API.)
 543
 544 .. _S3 Notification Compatibility: ../s3-notification-compatibility
 545 .. _AWS Create Topic: https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html
 546 .. _Bucket Operations: ../s3/bucketops
 547 .. _S3 CloudEvents Spec: https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md