ceph/doc/radosgw/pubsub-module.rst

   1 ==================
   2 PubSub Sync Module
   3 ==================
   4
   5 .. versionadded:: Nautilus
   6
   7 .. contents::
   8
   9 This sync module provides a publish and subscribe mechanism for the object store modification
  10 events. Events are published into predefined topics. Topics can be subscribed to, and events
  11 can be pulled from them. Events need to be acked. Also, events will expire and disappear
  12 after a period of time.
  13
  14 A push notification mechanism exists too, currently supporting HTTP and
  15 AMQP0.9.1 endpoints, on top of storing the events in Ceph. If events should only be pushed to an endpoint
  16 and do not need to be stored in Ceph, the `Bucket Notification`_ mechanism should be used instead of pubsub sync module.
  17
  18 A user can create different topics. A topic entity is defined by its user and its name. A
  19 user can only manage its own topics, and can only subscribe to events published by buckets
  20 it owns.
  21
  22 In order to publish events for specific bucket a notification entity needs to be created. A
  23 notification can be created on a subset of event types, or for all event types (default).
  24 There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications.
  25
  26 A subscription to a topic can also be defined. There can be multiple subscriptions for any
  27 specific topic.
  28
  29 REST API has been defined to provide configuration and control interfaces for the pubsub
  30 mechanisms. This API has two flavors, one is S3-compatible and one is not. The two flavors can be used
  31 together, although it is recommended to use the S3-compatible one.
  32 The S3-compatible API is similar to the one used in the bucket notification mechanism.
  33
  34 Events are stored as RGW objects in a special bucket, under a special user. Events cannot
  35 be accessed directly, but need to be pulled and acked using the new REST API.
  36
  37 .. toctree::
  38    :maxdepth: 1
  39
  40    S3 Bucket Notification Compatibility <s3-notification-compatibility>
  41
  42 PubSub Zone Configuration
  43 -------------------------
  44
  45 The pubsub sync module requires the creation of a new zone in a `Multisite`_ environment.
  46 First, a master zone must exist, then a secondary zone should be created.
  47 In the creation of the secondary zone, its tier type must be set to ``pubsub``:
  48
  49 ::
  50
  51    # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
  52                                --rgw-zone={zone-name} \
  53                                --endpoints={http://fqdn}[,{http://fqdn}] \
  54                                --sync-from-all=0 \
  55                                --sync-from={master-zone-name} \
  56                                --tier-type=pubsub
  57
  58
  59 PubSub Zone Configuration Parameters
  60 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  61
  62 ::
  63
  64    {
  65        "tenant": <tenant>,             # default: <empty>
  66        "uid": <uid>,                   # default: "pubsub"
  67        "data_bucket_prefix": <prefix>  # default: "pubsub-"
  68        "data_oid_prefix": <prefix>     #
  69        "events_retention_days": <days> # default: 7
  70    }
  71
  72 * ``tenant`` (string)
  73
  74 The tenant of the pubsub control user.
  75
  76 * ``uid`` (string)
  77
  78 The uid of the pubsub control user.
  79
  80 * ``data_bucket_prefix`` (string)
  81
  82 The prefix of the bucket name that will be created to store events for specific topic.
  83
  84 * ``data_oid_prefix`` (string)
  85
  86 The oid prefix for the stored events.
  87
  88 * ``events_retention_days`` (integer)
  89
  90 How many days to keep events that weren't acked.
  91
  92 Configuring Parameters via CLI
  93 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  94
  95 The tier configuration could be set using the following command:
  96
  97 ::
  98
  99    # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
 100                                 --rgw-zone={zone-name} \
 101                                 --tier-config={key}={val}[,{key}={val}]
 102
 103 Where the ``key`` in the configuration specifies the configuration variable that needs to be updated (from the list above), and
 104 the ``val`` specifies its new value. For example, setting the pubsub control user ``uid`` to ``user_ps``:
 105
 106 ::
 107
 108    # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
 109                                 --rgw-zone={zone-name} \
 110                                 --tier-config=uid=pubsub
 111
 112 A configuration field can be removed by using ``--tier-config-rm={key}``.
 113
 114 PubSub Performance Stats
 115 -------------------------
 116 Same counters are shared between the pubsub sync module and the notification mechanism.
 117
 118 - ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them
 119 - ``pubsub_event_lost``: running counter of events that had topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions
 120 - ``pubsub_store_ok``: running counter, for all subscriptions, of stored events
 121 - ``pubsub_store_fail``: running counter, for all subscriptions, of events failed to be stored
 122 - ``pubsub_push_ok``: running counter, for all subscriptions, of events successfully pushed to their endpoint
 123 - ``pubsub_push_fail``: running counter, for all subscriptions, of events failed to be pushed to their endpoint
 124 - ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet
 125
 126 .. note::
 127
 128     ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while:
 129     ``pubsub_store_ok``, ``pubsub_store_fail``, ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per store/push action on each subscriptions.
 130
 131 PubSub REST API
 132 ---------------
 133
 134 .. tip:: PubSub REST calls, and only them, should be sent to an RGW which belong to a PubSub zone
 135
 136 Topics
 137 ~~~~~~
 138
 139 Create a Topic
 140 ``````````````
 141
 142 This will create a new topic. Topic creation is needed both for both flavors of the API.
 143 Optionally the topic could be provided with push endpoint parameters that would be used later
 144 when an S3-compatible notification is created.
 145 Upon successful request, the response will include the topic ARN that could be later used to reference this topic in an S3-compatible notification request.
 146 To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values.
 147
 148 .. tip:: Any S3-compatible notification already associated with the topic needs to be re-created for the topic update to take effect
 149
 150 ::
 151
 152    PUT /topics/<topic-name>[?push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=<level>][&verify-ssl=true|false]]
 153
 154 Request parameters:
 155
 156 - push-endpoint: URI of endpoint to send push notification to
 157
 158  - URI schema is: ``http[s]|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]``
 159  - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP)
 160  - Default values for HTTP/S: no user/password, port 80/443
 161  - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/"
 162
 163 - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default)
 164 - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics
 165 - amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist:
 166
 167  - "none" - message is considered "delivered" if sent to broker
 168  - "broker" message is considered "delivered" if acked by broker
 169
 170 The topic ARN in the response will have the following format:
 171
 172 ::
 173
 174    arn:aws:sns:<zone-group>:<tenant>:<topic>
 175
 176 Get Topic Information
 177 `````````````````````
 178
 179 Returns information about specific topic. This includes subscriptions to that topic, and push-endpoint information, if provided.
 180
 181 ::
 182
 183    GET /topics/<topic-name>
 184
 185 Response will have the following format (JSON):
 186
 187 ::
 188
 189    {
 190        "topic":{
 191            "user":"",
 192            "name":"",
 193            "dest":{
 194                "bucket_name":"",
 195                "oid_prefix":"",
 196                "push_endpoint":"",
 197                "push_endpoint_args":""
 198            },
 199            "arn":""
 200        },
 201        "subs":[]
 202    }
 203
 204 - topic.user: name of the user that created the topic
 205 - name: name of the topic
 206 - dest.bucket_name: not used
 207 - dest.oid_prefix: not used
 208 - dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL
 209 - dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args
 210 - topic.arn: topic ARN
 211 - subs: list of subscriptions associated with this topic
 212
 213 Delete Topic
 214 ````````````
 215
 216 ::
 217
 218    DELETE /topics/<topic-name>
 219
 220 Delete the specified topic.
 221
 222 List Topics
 223 ```````````
 224
 225 List all topics that user defined.
 226
 227 ::
 228
 229    GET /topics
 230
 231 S3-Compliant Notifications
 232 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 233
 234 Detailed under: `Bucket Operations`_.
 235
 236 .. note::
 237
 238     - Notification creation will also create a subscription for pushing/pulling events
 239     - The generated subscription's name will have the same as the notification Id, and could be used later to fetch and ack events with the subscription API.
 240     - Notification deletion will deletes all generated subscriptions
 241     - In case that bucket deletion implicitly deletes the notification,
 242       the associated subscription will not be deleted automatically (any events of the deleted bucket could still be access),
 243       and will have to be deleted explicitly with the subscription deletion API
 244     - Filtering based on metadata (which is an extension to S3) is not supported, and such rules will be ignored
 245
 246
 247 Non S3-Compliant Notifications
 248 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 249
 250 Create a Notification
 251 `````````````````````
 252
 253 This will create a publisher for a specific bucket into a topic.
 254
 255 ::
 256
 257    PUT /notifications/bucket/<bucket>?topic=<topic-name>[&events=<event>[,<event>]]
 258
 259 Request parameters:
 260
 261 - topic-name: name of topic
 262 - event: event type (string), one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE``
 263
 264 Delete Notification Information
 265 ```````````````````````````````
 266
 267 Delete publisher from a specific bucket into a specific topic.
 268
 269 ::
 270
 271    DELETE /notifications/bucket/<bucket>?topic=<topic-name>
 272
 273 Request parameters:
 274
 275 - topic-name: name of topic
 276
 277 .. note:: When the bucket is deleted, any notification defined on it is also deleted
 278
 279 List Notifications
 280 ``````````````````
 281
 282 List all topics with associated events defined on a bucket.
 283
 284 ::
 285
 286    GET /notifications/bucket/<bucket>
 287
 288 Response will have the following format (JSON):
 289
 290 ::
 291
 292    {"topics":[
 293       {
 294          "topic":{
 295             "user":"",
 296             "name":"",
 297             "dest":{
 298                "bucket_name":"",
 299                "oid_prefix":"",
 300                "push_endpoint":"",
 301                "push_endpoint_args":""
 302             }
 303             "arn":""
 304          },
 305          "events":[]
 306       }
 307     ]}
 308
 309 Subscriptions
 310 ~~~~~~~~~~~~~
 311
 312 Create a Subscription
 313 `````````````````````
 314
 315 Creates a new subscription.
 316
 317 ::
 318
 319    PUT /subscriptions/<sub-name>?topic=<topic-name>[&push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=<level>][&verify-ssl=true|false]]
 320
 321 Request parameters:
 322
 323 - topic-name: name of topic
 324 - push-endpoint: URI of endpoint to send push notification to
 325
 326  - URI schema is: ``http[s]|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]``
 327  - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP)
 328  - Default values for HTTP/S: no user/password, port 80/443
 329  - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/"
 330
 331 - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default)
 332 - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics
 333 - amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist:
 334
 335  - "none": message is considered "delivered" if sent to broker
 336  - "broker": message is considered "delivered" if acked by broker
 337
 338 Get Subscription Information
 339 ````````````````````````````
 340
 341 Returns information about specific subscription.
 342
 343 ::
 344
 345    GET /subscriptions/<sub-name>
 346
 347 Response will have the following format (JSON):
 348
 349 ::
 350
 351    {
 352        "user":"",
 353        "name":"",
 354        "topic":"",
 355        "dest":{
 356            "bucket_name":"",
 357            "oid_prefix":"",
 358            "push_endpoint":"",
 359            "push_endpoint_args":""
 360        }
 361        "s3_id":""
 362    }
 363
 364 - user: name of the user that created the subscription
 365 - name: name of the subscription
 366 - topic: name of the topic the subscription is associated with
 367
 368 Delete Subscription
 369 ```````````````````
 370
 371 Removes a subscription.
 372
 373 ::
 374
 375    DELETE /subscriptions/<sub-name>
 376
 377 Events
 378 ~~~~~~
 379
 380 Pull Events
 381 ```````````
 382
 383 Pull events sent to a specific subscription.
 384
 385 ::
 386
 387    GET /subscriptions/<sub-name>?events[&max-entries=<max-entries>][&marker=<marker>]
 388
 389 Request parameters:
 390
 391 - marker: pagination marker for list of events, if not specified will start from the oldest
 392 - max-entries: max number of events to return
 393
 394 The response will hold information on the current marker and whether there are more events not fetched:
 395
 396 ::
 397
 398    {"next_marker":"","is_truncated":"",...}
 399
 400
 401 The actual content of the response is depended with how the subscription was created.
 402 In case that the subscription was created via an S3-compatible notification,
 403 the events will have an S3-compatible record format (JSON):
 404
 405 ::
 406
 407    {"Records":[
 408        {
 409            "eventVersion":"2.1"
 410            "eventSource":"aws:s3",
 411            "awsRegion":"",
 412            "eventTime":"",
 413            "eventName":"",
 414            "userIdentity":{
 415                "principalId":""
 416            },
 417            "requestParameters":{
 418                "sourceIPAddress":""
 419            },
 420            "responseElements":{
 421                "x-amz-request-id":"",
 422                "x-amz-id-2":""
 423            },
 424            "s3":{
 425                "s3SchemaVersion":"1.0",
 426                "configurationId":"",
 427                "bucket":{
 428                    "name":"",
 429                    "ownerIdentity":{
 430                        "principalId":""
 431                    },
 432                    "arn":"",
 433                    "id":""
 434                },
 435                "object":{
 436                    "key":"",
 437                    "size":"0",
 438                    "eTag":"",
 439                    "versionId":"",
 440                    "sequencer":"",
 441                    "metadata":""
 442                }
 443            },
 444            "eventId":"",
 445        }
 446    ]}
 447
 448 - awsRegion: zonegroup
 449 - eventTime: timestamp indicating when the event was triggered
 450 - eventName: either ``s3:ObjectCreated:``, or ``s3:ObjectRemoved:``
 451 - userIdentity: not supported
 452 - requestParameters: not supported
 453 - responseElements: not supported
 454 - s3.configurationId: notification ID that created the subscription for the event
 455 - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API)
 456 - s3.bucket.name: name of the bucket
 457 - s3.bucket.ownerIdentity.principalId: owner of the bucket
 458 - s3.bucket.arn: ARN of the bucket
 459 - s3.bucket.id: Id of the bucket (an extension to the S3 notification API)
 460 - s3.object.key: object key
 461 - s3.object.size: not supported
 462 - s3.object.eTag: object etag
 463 - s3.object.version: object version in case of versioned bucket
 464 - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format)
 465 - s3.object.metadata: not supported (an extension to the S3 notification API)
 466 - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API)
 467
 468 In case that the subscription was not created via a non S3-compatible notification,
 469 the events will have the following event format (JSON):
 470
 471 ::
 472
 473     {"events":[
 474        {
 475            "id":"",
 476            "event":"",
 477            "timestamp":"",
 478            "info":{
 479                "attrs":{
 480                    "mtime":""
 481                },
 482                "bucket":{
 483                    "bucket_id":"",
 484                    "name":"",
 485                    "tenant":""
 486                },
 487                "key":{
 488                    "instance":"",
 489                    "name":""
 490                }
 491            }
 492        }
 493    ]}
 494
 495 - id: unique ID of the event, that could be used for acking
 496 - event: one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE``
 497 - timestamp: timestamp indicating when the event was sent
 498 - info.attrs.mtime: timestamp indicating when the event was triggered
 499 - info.bucket.bucket_id: id of the bucket
 500 - info.bucket.name: name of the bucket
 501 - info.bucket.tenant: tenant the bucket belongs to
 502 - info.key.instance: object version in case of versioned bucket
 503 - info.key.name: object key
 504
 505 Ack Event
 506 `````````
 507
 508 Ack event so that it can be removed from the subscription history.
 509
 510 ::
 511
 512    POST /subscriptions/<sub-name>?ack&event-id=<event-id>
 513
 514 Request parameters:
 515
 516 - event-id: id of event to be acked
 517
 518 .. _Multisite : ../multisite
 519 .. _Bucket Notification : ../notifications
 520 .. _Bucket Operations: ../s3/bucketops