5 .. versionadded:: Nautilus
9 This sync module provides a publish and subscribe mechanism for the object store modification
10 events. Events are published into predefined topics. Topics can be subscribed to, and events
11 can be pulled from them. Events need to be acked. Also, events will expire and disappear
12 after a period of time.
14 A push notification mechanism exists too, currently supporting HTTP,
15 AMQP0.9.1 and Kafka endpoints. In this case, the events are pushed to an endpoint on top of storing them in Ceph. If events should only be pushed to an endpoint
16 and do not need to be stored in Ceph, the `Bucket Notification`_ mechanism should be used instead of pubsub sync module.
18 A user can create different topics. A topic entity is defined by its user and its name. A
19 user can only manage its own topics, and can only subscribe to events published by buckets
22 In order to publish events for specific bucket a notification entity needs to be created. A
23 notification can be created on a subset of event types, or for all event types (default).
24 There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications.
26 A subscription to a topic can also be defined. There can be multiple subscriptions for any
29 REST API has been defined to provide configuration and control interfaces for the pubsub
30 mechanisms. This API has two flavors, one is S3-compatible and one is not. The two flavors can be used
31 together, although it is recommended to use the S3-compatible one.
32 The S3-compatible API is similar to the one used in the bucket notification mechanism.
34 Events are stored as RGW objects in a special bucket, under a special user. Events cannot
35 be accessed directly, but need to be pulled and acked using the new REST API.
40 S3 Bucket Notification Compatibility <s3-notification-compatibility>
42 PubSub Zone Configuration
43 -------------------------
45 The pubsub sync module requires the creation of a new zone in a `Multisite`_ environment.
46 First, a master zone must exist (see: :ref:`master-zone-label`),
47 then a secondary zone should be created (see :ref:`secondary-zone-label`).
48 In the creation of the secondary zone, its tier type must be set to ``pubsub``:
52 # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
53 --rgw-zone={zone-name} \
54 --endpoints={http://fqdn}[,{http://fqdn}] \
56 --sync-from={master-zone-name} \
60 PubSub Zone Configuration Parameters
61 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
66 "tenant": <tenant>, # default: <empty>
67 "uid": <uid>, # default: "pubsub"
68 "data_bucket_prefix": <prefix> # default: "pubsub-"
69 "data_oid_prefix": <prefix> #
70 "events_retention_days": <days> # default: 7
75 The tenant of the pubsub control user.
79 The uid of the pubsub control user.
81 * ``data_bucket_prefix`` (string)
83 The prefix of the bucket name that will be created to store events for specific topic.
85 * ``data_oid_prefix`` (string)
87 The oid prefix for the stored events.
89 * ``events_retention_days`` (integer)
91 How many days to keep events that weren't acked.
93 Configuring Parameters via CLI
94 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
96 The tier configuration could be set using the following command:
100 # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
101 --rgw-zone={zone-name} \
102 --tier-config={key}={val}[,{key}={val}]
104 Where the ``key`` in the configuration specifies the configuration variable that needs to be updated (from the list above), and
105 the ``val`` specifies its new value. For example, setting the pubsub control user ``uid`` to ``user_ps``:
109 # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
110 --rgw-zone={zone-name} \
111 --tier-config=uid=pubsub
113 A configuration field can be removed by using ``--tier-config-rm={key}``.
115 PubSub Performance Stats
116 -------------------------
117 Same counters are shared between the pubsub sync module and the notification mechanism.
119 - ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them
120 - ``pubsub_event_lost``: running counter of events that had topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions
121 - ``pubsub_store_ok``: running counter, for all subscriptions, of stored events
122 - ``pubsub_store_fail``: running counter, for all subscriptions, of events failed to be stored
123 - ``pubsub_push_ok``: running counter, for all subscriptions, of events successfully pushed to their endpoint
124 - ``pubsub_push_fail``: running counter, for all subscriptions, of events failed to be pushed to their endpoint
125 - ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet
129 ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while:
130 ``pubsub_store_ok``, ``pubsub_store_fail``, ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per store/push action on each subscriptions.
135 .. tip:: PubSub REST calls, and only them, should be sent to an RGW which belong to a PubSub zone
143 This will create a new topic. Topic creation is needed both for both flavors of the API.
144 Optionally the topic could be provided with push endpoint parameters that would be used later
145 when an S3-compatible notification is created.
146 Upon successful request, the response will include the topic ARN that could be later used to reference this topic in an S3-compatible notification request.
147 To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values.
149 .. tip:: Any S3-compatible notification already associated with the topic needs to be re-created for the topic update to take effect
153 PUT /topics/<topic-name>[?OpaqueData=<opaque data>][&push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=none|broker][&verify-ssl=true|false][&kafka-ack-level=none|broker][&use-ssl=true|false][&ca-location=<file path>]]
157 - push-endpoint: URI of an endpoint to send push notification to
158 - OpaqueData: opaque data is set in the topic configuration and added to all notifications triggered by the ropic
160 The endpoint URI may include parameters depending with the type of endpoint:
164 - URI: ``http[s]://<fqdn>[:<port]``
165 - port defaults to: 80/443 for HTTP/S accordingly
166 - verify-ssl: indicate whether the server certificate is validated by the client or not ("true" by default)
170 - URI: ``amqp://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]``
171 - user/password defaults to: guest/guest
172 - user/password may only be provided over HTTPS. Topic creation request will be rejected if not
173 - port defaults to: 5672
174 - vhost defaults to: "/"
175 - amqp-exchange: the exchanges must exist and be able to route messages based on topics (mandatory parameter for AMQP0.9.1)
176 - amqp-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Two ack methods exist:
178 - "none": message is considered "delivered" if sent to broker
179 - "broker": message is considered "delivered" if acked by broker (default)
183 - URI: ``kafka://[<user>:<password>@]<fqdn>[:<port]``
184 - if ``use-ssl`` is set to "true", secure connection will be used for connecting with the broker ("false" by default)
185 - if ``ca-location`` is provided, and secure connection is used, the specified CA will be used, instead of the default one, to authenticate the broker
186 - user/password may only be provided over HTTPS. Topic creation request will be rejected if not
187 - user/password may only be provided together with ``use-ssl``, connection to the broker would fail if not
188 - port defaults to: 9092
189 - kafka-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Two ack methods exist:
191 - "none": message is considered "delivered" if sent to broker
192 - "broker": message is considered "delivered" if acked by broker (default)
194 The topic ARN in the response will have the following format:
198 arn:aws:sns:<zone-group>:<tenant>:<topic>
200 Get Topic Information
201 `````````````````````
203 Returns information about specific topic. This includes subscriptions to that topic, and push-endpoint information, if provided.
207 GET /topics/<topic-name>
209 Response will have the following format (JSON):
221 "push_endpoint_args":"",
222 "push_endpoint_topic":""
230 - topic.user: name of the user that created the topic
231 - name: name of the topic
232 - dest.bucket_name: not used
233 - dest.oid_prefix: not used
234 - dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL
235 - if push-endpoint URL contain user/password information, request must be made over HTTPS. Topic get request will be rejected if not
236 - dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args
237 - dest.push_endpoint_topic: in case of S3-compliant notifications, this value will hold the topic name as sent to the endpoint (may be different than the internal topic name)
238 - topic.arn: topic ARN
239 - subs: list of subscriptions associated with this topic
246 DELETE /topics/<topic-name>
248 Delete the specified topic.
253 List all topics that user defined.
259 - if push-endpoint URL contain user/password information, in any of the topic, request must be made over HTTPS. Topic list request will be rejected if not
261 S3-Compliant Notifications
262 ~~~~~~~~~~~~~~~~~~~~~~~~~~
264 Detailed under: `Bucket Operations`_.
268 - Notification creation will also create a subscription for pushing/pulling events
269 - The generated subscription's name will have the same as the notification Id, and could be used later to fetch and ack events with the subscription API.
270 - Notification deletion will deletes all generated subscriptions
271 - In case that bucket deletion implicitly deletes the notification,
272 the associated subscription will not be deleted automatically (any events of the deleted bucket could still be access),
273 and will have to be deleted explicitly with the subscription deletion API
274 - Filtering based on metadata (which is an extension to S3) is not supported, and such rules will be ignored
275 - Filtering based on tags (which is an extension to S3) is not supported, and such rules will be ignored
278 Non S3-Compliant Notifications
279 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
281 Create a Notification
282 `````````````````````
284 This will create a publisher for a specific bucket into a topic.
288 PUT /notifications/bucket/<bucket>?topic=<topic-name>[&events=<event>[,<event>]]
292 - topic-name: name of topic
293 - event: event type (string), one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE``
295 Delete Notification Information
296 ```````````````````````````````
298 Delete publisher from a specific bucket into a specific topic.
302 DELETE /notifications/bucket/<bucket>?topic=<topic-name>
306 - topic-name: name of topic
308 .. note:: When the bucket is deleted, any notification defined on it is also deleted
313 List all topics with associated events defined on a bucket.
317 GET /notifications/bucket/<bucket>
319 Response will have the following format (JSON):
332 "push_endpoint_args":"",
333 "push_endpoint_topic":""
344 Create a Subscription
345 `````````````````````
347 Creates a new subscription.
351 PUT /subscriptions/<sub-name>?topic=<topic-name>[?push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=none|broker][&verify-ssl=true|false][&kafka-ack-level=none|broker][&ca-location=<file path>]]
355 - topic-name: name of topic
356 - push-endpoint: URI of endpoint to send push notification to
358 The endpoint URI may include parameters depending with the type of endpoint:
362 - URI: ``http[s]://<fqdn>[:<port]``
363 - port defaults to: 80/443 for HTTP/S accordingly
364 - verify-ssl: indicate whether the server certificate is validated by the client or not ("true" by default)
368 - URI: ``amqp://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]``
369 - user/password defaults to : guest/guest
370 - port defaults to: 5672
371 - vhost defaults to: "/"
372 - amqp-exchange: the exchanges must exist and be able to route messages based on topics (mandatory parameter for AMQP0.9.1)
373 - amqp-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Two ack methods exist:
375 - "none": message is considered "delivered" if sent to broker
376 - "broker": message is considered "delivered" if acked by broker (default)
380 - URI: ``kafka://[<user>:<password>@]<fqdn>[:<port]``
381 - if ``ca-location`` is provided, secure connection will be used for connection with the broker
382 - user/password may only be provided over HTTPS. Topic creation request will be rejected if not
383 - user/password may only be provided together with ``ca-location``. Topic creation request will be rejected if not
384 - port defaults to: 9092
385 - kafka-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Two ack methods exist:
387 - "none": message is considered "delivered" if sent to broker
388 - "broker": message is considered "delivered" if acked by broker (default)
391 Get Subscription Information
392 ````````````````````````````
394 Returns information about specific subscription.
398 GET /subscriptions/<sub-name>
400 Response will have the following format (JSON):
412 "push_endpoint_args":"",
413 "push_endpoint_topic":""
418 - user: name of the user that created the subscription
419 - name: name of the subscription
420 - topic: name of the topic the subscription is associated with
421 - dest.bucket_name: name of the bucket storing the events
422 - dest.oid_prefix: oid prefix for the events stored in the bucket
423 - dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL
424 - if push-endpoint URL contain user/password information, request must be made over HTTPS. Topic get request will be rejected if not
425 - dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args
426 - dest.push_endpoint_topic: in case of S3-compliant notifications, this value will hold the topic name as sent to the endpoint (may be different than the internal topic name)
427 - s3_id: in case of S3-compliant notifications, this will hold the notification name that created the subscription
432 Removes a subscription.
436 DELETE /subscriptions/<sub-name>
444 Pull events sent to a specific subscription.
448 GET /subscriptions/<sub-name>?events[&max-entries=<max-entries>][&marker=<marker>]
452 - marker: pagination marker for list of events, if not specified will start from the oldest
453 - max-entries: max number of events to return
455 The response will hold information on the current marker and whether there are more events not fetched:
459 {"next_marker":"","is_truncated":"",...}
462 The actual content of the response is depended with how the subscription was created.
463 In case that the subscription was created via an S3-compatible notification,
464 the events will have an S3-compatible record format (JSON):
471 "eventSource":"aws:s3",
478 "requestParameters":{
482 "x-amz-request-id":"",
486 "s3SchemaVersion":"1.0",
487 "configurationId":"",
511 - awsRegion: zonegroup
512 - eventTime: timestamp indicating when the event was triggered
513 - eventName: either ``s3:ObjectCreated:``, or ``s3:ObjectRemoved:``
514 - userIdentity: not supported
515 - requestParameters: not supported
516 - responseElements: not supported
517 - s3.configurationId: notification ID that created the subscription for the event
518 - s3.bucket.name: name of the bucket
519 - s3.bucket.ownerIdentity.principalId: owner of the bucket
520 - s3.bucket.arn: ARN of the bucket
521 - s3.bucket.id: Id of the bucket (an extension to the S3 notification API)
522 - s3.object.key: object key
523 - s3.object.size: not supported
524 - s3.object.eTag: object etag
525 - s3.object.version: object version in case of versioned bucket
526 - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format)
527 - s3.object.metadata: not supported (an extension to the S3 notification API)
528 - s3.object.tags: not supported (an extension to the S3 notification API)
529 - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API)
530 - s3.opaqueData: opaque data is set in the topic configuration and added to all notifications triggered by the ropic (an extension to the S3 notification API)
532 In case that the subscription was not created via a non S3-compatible notification,
533 the events will have the following event format (JSON):
559 - id: unique ID of the event, that could be used for acking
560 - event: one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE``
561 - timestamp: timestamp indicating when the event was sent
562 - info.attrs.mtime: timestamp indicating when the event was triggered
563 - info.bucket.bucket_id: id of the bucket
564 - info.bucket.name: name of the bucket
565 - info.bucket.tenant: tenant the bucket belongs to
566 - info.key.instance: object version in case of versioned bucket
567 - info.key.name: object key
572 Ack event so that it can be removed from the subscription history.
576 POST /subscriptions/<sub-name>?ack&event-id=<event-id>
580 - event-id: id of event to be acked
582 .. _Multisite : ../multisite
583 .. _Bucket Notification : ../notifications
584 .. _Bucket Operations: ../s3/bucketops