]>
Commit | Line | Data |
---|---|---|
eafe8130 | 1 | ================== |
11fdf7f2 | 2 | PubSub Sync Module |
eafe8130 | 3 | ================== |
11fdf7f2 TL |
4 | |
5 | .. versionadded:: Nautilus | |
6 | ||
eafe8130 TL |
7 | .. contents:: |
8 | ||
11fdf7f2 | 9 | This sync module provides a publish and subscribe mechanism for the object store modification |
eafe8130 | 10 | events. Events are published into predefined topics. Topics can be subscribed to, and events |
11fdf7f2 | 11 | can be pulled from them. Events need to be acked. Also, events will expire and disappear |
eafe8130 TL |
12 | after a period of time. |
13 | ||
9f95a23c TL |
14 | A push notification mechanism exists too, currently supporting HTTP, |
15 | AMQP0.9.1 and Kafka endpoints. In this case, the events are pushed to an endpoint on top of storing them in Ceph. If events should only be pushed to an endpoint | |
eafe8130 | 16 | and do not need to be stored in Ceph, the `Bucket Notification`_ mechanism should be used instead of pubsub sync module. |
11fdf7f2 | 17 | |
f67539c2 TL |
18 | A user can create different topics. A topic entity is defined by its name and is per tenant. A |
19 | user can only associate its topics (via notification configuration) with buckets it owns. | |
11fdf7f2 | 20 | |
eafe8130 TL |
21 | In order to publish events for specific bucket a notification entity needs to be created. A |
22 | notification can be created on a subset of event types, or for all event types (default). | |
23 | There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. | |
11fdf7f2 TL |
24 | |
25 | A subscription to a topic can also be defined. There can be multiple subscriptions for any | |
26 | specific topic. | |
27 | ||
eafe8130 TL |
28 | REST API has been defined to provide configuration and control interfaces for the pubsub |
29 | mechanisms. This API has two flavors, one is S3-compatible and one is not. The two flavors can be used | |
30 | together, although it is recommended to use the S3-compatible one. | |
31 | The S3-compatible API is similar to the one used in the bucket notification mechanism. | |
11fdf7f2 | 32 | |
f67539c2 | 33 | Events are stored as RGW objects in a special bucket, under a special user (pubsub control user). Events cannot |
eafe8130 | 34 | be accessed directly, but need to be pulled and acked using the new REST API. |
11fdf7f2 | 35 | |
eafe8130 TL |
36 | .. toctree:: |
37 | :maxdepth: 1 | |
11fdf7f2 | 38 | |
eafe8130 TL |
39 | S3 Bucket Notification Compatibility <s3-notification-compatibility> |
40 | ||
f67539c2 TL |
41 | .. note:: To enable bucket notifications API, the `rgw_enable_apis` configuration parameter should contain: "notifications". |
42 | ||
eafe8130 TL |
43 | PubSub Zone Configuration |
44 | ------------------------- | |
11fdf7f2 | 45 | |
f67539c2 | 46 | The pubsub sync module requires the creation of a new zone in a :ref:`multisite` environment... |
9f95a23c TL |
47 | First, a master zone must exist (see: :ref:`master-zone-label`), |
48 | then a secondary zone should be created (see :ref:`secondary-zone-label`). | |
eafe8130 | 49 | In the creation of the secondary zone, its tier type must be set to ``pubsub``: |
11fdf7f2 TL |
50 | |
51 | :: | |
52 | ||
eafe8130 TL |
53 | # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ |
54 | --rgw-zone={zone-name} \ | |
55 | --endpoints={http://fqdn}[,{http://fqdn}] \ | |
56 | --sync-from-all=0 \ | |
57 | --sync-from={master-zone-name} \ | |
58 | --tier-type=pubsub | |
11fdf7f2 | 59 | |
11fdf7f2 | 60 | |
eafe8130 TL |
61 | PubSub Zone Configuration Parameters |
62 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
63 | ||
64 | :: | |
11fdf7f2 | 65 | |
eafe8130 TL |
66 | { |
67 | "tenant": <tenant>, # default: <empty> | |
68 | "uid": <uid>, # default: "pubsub" | |
69 | "data_bucket_prefix": <prefix> # default: "pubsub-" | |
70 | "data_oid_prefix": <prefix> # | |
71 | "events_retention_days": <days> # default: 7 | |
72 | } | |
11fdf7f2 TL |
73 | |
74 | * ``tenant`` (string) | |
75 | ||
76 | The tenant of the pubsub control user. | |
77 | ||
78 | * ``uid`` (string) | |
79 | ||
80 | The uid of the pubsub control user. | |
81 | ||
82 | * ``data_bucket_prefix`` (string) | |
83 | ||
84 | The prefix of the bucket name that will be created to store events for specific topic. | |
85 | ||
86 | * ``data_oid_prefix`` (string) | |
87 | ||
88 | The oid prefix for the stored events. | |
89 | ||
90 | * ``events_retention_days`` (integer) | |
91 | ||
92 | How many days to keep events that weren't acked. | |
93 | ||
eafe8130 TL |
94 | Configuring Parameters via CLI |
95 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
11fdf7f2 | 96 | |
eafe8130 | 97 | The tier configuration could be set using the following command: |
11fdf7f2 TL |
98 | |
99 | :: | |
100 | ||
eafe8130 | 101 | # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ |
11fdf7f2 TL |
102 | --rgw-zone={zone-name} \ |
103 | --tier-config={key}={val}[,{key}={val}] | |
104 | ||
eafe8130 TL |
105 | Where the ``key`` in the configuration specifies the configuration variable that needs to be updated (from the list above), and |
106 | the ``val`` specifies its new value. For example, setting the pubsub control user ``uid`` to ``user_ps``: | |
11fdf7f2 TL |
107 | |
108 | :: | |
109 | ||
eafe8130 | 110 | # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ |
11fdf7f2 TL |
111 | --rgw-zone={zone-name} \ |
112 | --tier-config=uid=pubsub | |
113 | ||
11fdf7f2 TL |
114 | A configuration field can be removed by using ``--tier-config-rm={key}``. |
115 | ||
f67539c2 TL |
116 | |
117 | Topic and Subscription Management via CLI | |
118 | ----------------------------------------- | |
119 | ||
120 | Configuration of all topics, associated with a tenant, could be fetched using the following command: | |
121 | ||
122 | :: | |
123 | ||
124 | # radosgw-admin topic list [--tenant={tenant}] | |
125 | ||
126 | ||
127 | Configuration of a specific topic could be fetched using: | |
128 | ||
129 | :: | |
130 | ||
131 | # radosgw-admin topic get --topic={topic-name} [--tenant={tenant}] | |
132 | ||
133 | ||
134 | And removed using: | |
135 | ||
136 | :: | |
137 | ||
138 | # radosgw-admin topic rm --topic={topic-name} [--tenant={tenant}] | |
139 | ||
140 | ||
141 | Configuration of a subscription could be fetched using: | |
142 | ||
143 | :: | |
144 | ||
145 | # radosgw-admin subscription get --subscription={topic-name} [--tenant={tenant}] | |
146 | ||
147 | And removed using: | |
148 | ||
149 | :: | |
150 | ||
151 | # radosgw-admin subscription rm --subscription={topic-name} [--tenant={tenant}] | |
152 | ||
153 | ||
154 | To fetch all of the events stored in a subcription, use: | |
155 | ||
156 | :: | |
157 | ||
158 | # radosgw-admin subscription pull --subscription={topic-name} [--marker={last-marker}] [--tenant={tenant}] | |
159 | ||
160 | ||
161 | To ack (and remove) an event from a subscription, use: | |
162 | ||
163 | :: | |
164 | ||
165 | # radosgw-admin subscription ack --subscription={topic-name} --event-id={event-id} [--tenant={tenant}] | |
166 | ||
167 | ||
11fdf7f2 TL |
168 | PubSub Performance Stats |
169 | ------------------------- | |
eafe8130 TL |
170 | Same counters are shared between the pubsub sync module and the notification mechanism. |
171 | ||
172 | - ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them | |
173 | - ``pubsub_event_lost``: running counter of events that had topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions | |
174 | - ``pubsub_store_ok``: running counter, for all subscriptions, of stored events | |
175 | - ``pubsub_store_fail``: running counter, for all subscriptions, of events failed to be stored | |
176 | - ``pubsub_push_ok``: running counter, for all subscriptions, of events successfully pushed to their endpoint | |
177 | - ``pubsub_push_fail``: running counter, for all subscriptions, of events failed to be pushed to their endpoint | |
178 | - ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet | |
11fdf7f2 | 179 | |
eafe8130 TL |
180 | .. note:: |
181 | ||
182 | ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while: | |
183 | ``pubsub_store_ok``, ``pubsub_store_fail``, ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per store/push action on each subscriptions. | |
11fdf7f2 TL |
184 | |
185 | PubSub REST API | |
eafe8130 | 186 | --------------- |
11fdf7f2 | 187 | |
eafe8130 | 188 | .. tip:: PubSub REST calls, and only them, should be sent to an RGW which belong to a PubSub zone |
11fdf7f2 TL |
189 | |
190 | Topics | |
191 | ~~~~~~ | |
eafe8130 | 192 | |
f67539c2 TL |
193 | .. _radosgw-create-a-topic: |
194 | ||
11fdf7f2 | 195 | Create a Topic |
eafe8130 TL |
196 | `````````````` |
197 | ||
198 | This will create a new topic. Topic creation is needed both for both flavors of the API. | |
199 | Optionally the topic could be provided with push endpoint parameters that would be used later | |
200 | when an S3-compatible notification is created. | |
201 | Upon successful request, the response will include the topic ARN that could be later used to reference this topic in an S3-compatible notification request. | |
202 | To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values. | |
11fdf7f2 | 203 | |
eafe8130 | 204 | .. tip:: Any S3-compatible notification already associated with the topic needs to be re-created for the topic update to take effect |
11fdf7f2 TL |
205 | |
206 | :: | |
207 | ||
e306af50 | 208 | PUT /topics/<topic-name>[?OpaqueData=<opaque data>][&push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=none|broker|routable][&verify-ssl=true|false][&kafka-ack-level=none|broker][&use-ssl=true|false][&ca-location=<file path>]] |
eafe8130 TL |
209 | |
210 | Request parameters: | |
211 | ||
9f95a23c | 212 | - push-endpoint: URI of an endpoint to send push notification to |
20effc67 | 213 | - OpaqueData: opaque data is set in the topic configuration and added to all notifications triggered by the topic |
9f95a23c TL |
214 | |
215 | The endpoint URI may include parameters depending with the type of endpoint: | |
216 | ||
217 | - HTTP endpoint | |
218 | ||
219 | - URI: ``http[s]://<fqdn>[:<port]`` | |
220 | - port defaults to: 80/443 for HTTP/S accordingly | |
221 | - verify-ssl: indicate whether the server certificate is validated by the client or not ("true" by default) | |
222 | ||
223 | - AMQP0.9.1 endpoint | |
eafe8130 | 224 | |
f67539c2 | 225 | - URI: ``amqp[s]://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]`` |
9f95a23c TL |
226 | - user/password defaults to: guest/guest |
227 | - user/password may only be provided over HTTPS. Topic creation request will be rejected if not | |
f67539c2 | 228 | - port defaults to: 5672/5671 for unencrypted/SSL-encrypted connections |
9f95a23c | 229 | - vhost defaults to: "/" |
f67539c2 TL |
230 | - verify-ssl: indicate whether the server certificate is validated by the client or not ("true" by default) |
231 | - if ``ca-location`` is provided, and secure connection is used, the specified CA will be used, instead of the default one, to authenticate the broker | |
232 | - amqp-exchange: the exchanges must exist and be able to route messages based on topics (mandatory parameter for AMQP0.9.1). Different topics pointing to the same endpoint must use the same exchange | |
e306af50 | 233 | - amqp-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Three ack methods exist: |
eafe8130 | 234 | |
9f95a23c TL |
235 | - "none": message is considered "delivered" if sent to broker |
236 | - "broker": message is considered "delivered" if acked by broker (default) | |
e306af50 | 237 | - "routable": message is considered "delivered" if broker can route to a consumer |
eafe8130 | 238 | |
f67539c2 TL |
239 | .. tip:: The topic-name (see :ref:`radosgw-create-a-topic`) is used for the AMQP topic ("routing key" for a topic exchange) |
240 | ||
9f95a23c TL |
241 | - Kafka endpoint |
242 | ||
243 | - URI: ``kafka://[<user>:<password>@]<fqdn>[:<port]`` | |
244 | - if ``use-ssl`` is set to "true", secure connection will be used for connecting with the broker ("false" by default) | |
245 | - if ``ca-location`` is provided, and secure connection is used, the specified CA will be used, instead of the default one, to authenticate the broker | |
246 | - user/password may only be provided over HTTPS. Topic creation request will be rejected if not | |
247 | - user/password may only be provided together with ``use-ssl``, connection to the broker would fail if not | |
248 | - port defaults to: 9092 | |
249 | - kafka-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Two ack methods exist: | |
250 | ||
251 | - "none": message is considered "delivered" if sent to broker | |
252 | - "broker": message is considered "delivered" if acked by broker (default) | |
eafe8130 TL |
253 | |
254 | The topic ARN in the response will have the following format: | |
11fdf7f2 | 255 | |
eafe8130 TL |
256 | :: |
257 | ||
258 | arn:aws:sns:<zone-group>:<tenant>:<topic> | |
11fdf7f2 TL |
259 | |
260 | Get Topic Information | |
eafe8130 | 261 | ````````````````````` |
11fdf7f2 | 262 | |
eafe8130 | 263 | Returns information about specific topic. This includes subscriptions to that topic, and push-endpoint information, if provided. |
11fdf7f2 TL |
264 | |
265 | :: | |
266 | ||
267 | GET /topics/<topic-name> | |
268 | ||
eafe8130 TL |
269 | Response will have the following format (JSON): |
270 | ||
271 | :: | |
11fdf7f2 | 272 | |
eafe8130 TL |
273 | { |
274 | "topic":{ | |
275 | "user":"", | |
276 | "name":"", | |
277 | "dest":{ | |
278 | "bucket_name":"", | |
279 | "oid_prefix":"", | |
280 | "push_endpoint":"", | |
9f95a23c | 281 | "push_endpoint_args":"", |
f67539c2 TL |
282 | "push_endpoint_topic":"", |
283 | "stored_secret":"", | |
284 | "persistent":"" | |
eafe8130 TL |
285 | }, |
286 | "arn":"" | |
9f95a23c | 287 | "opaqueData":"" |
eafe8130 TL |
288 | }, |
289 | "subs":[] | |
290 | } | |
291 | ||
292 | - topic.user: name of the user that created the topic | |
293 | - name: name of the topic | |
294 | - dest.bucket_name: not used | |
295 | - dest.oid_prefix: not used | |
296 | - dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL | |
9f95a23c | 297 | - if push-endpoint URL contain user/password information, request must be made over HTTPS. Topic get request will be rejected if not |
eafe8130 | 298 | - dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args |
9f95a23c | 299 | - dest.push_endpoint_topic: in case of S3-compliant notifications, this value will hold the topic name as sent to the endpoint (may be different than the internal topic name) |
eafe8130 TL |
300 | - topic.arn: topic ARN |
301 | - subs: list of subscriptions associated with this topic | |
11fdf7f2 TL |
302 | |
303 | Delete Topic | |
eafe8130 | 304 | ```````````` |
11fdf7f2 TL |
305 | |
306 | :: | |
307 | ||
308 | DELETE /topics/<topic-name> | |
309 | ||
310 | Delete the specified topic. | |
311 | ||
312 | List Topics | |
eafe8130 | 313 | ``````````` |
11fdf7f2 | 314 | |
f67539c2 | 315 | List all topics associated with a tenant. |
11fdf7f2 TL |
316 | |
317 | :: | |
318 | ||
319 | GET /topics | |
eafe8130 | 320 | |
9f95a23c TL |
321 | - if push-endpoint URL contain user/password information, in any of the topic, request must be made over HTTPS. Topic list request will be rejected if not |
322 | ||
eafe8130 TL |
323 | S3-Compliant Notifications |
324 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
11fdf7f2 | 325 | |
eafe8130 | 326 | Detailed under: `Bucket Operations`_. |
11fdf7f2 | 327 | |
eafe8130 TL |
328 | .. note:: |
329 | ||
330 | - Notification creation will also create a subscription for pushing/pulling events | |
331 | - The generated subscription's name will have the same as the notification Id, and could be used later to fetch and ack events with the subscription API. | |
332 | - Notification deletion will deletes all generated subscriptions | |
333 | - In case that bucket deletion implicitly deletes the notification, | |
334 | the associated subscription will not be deleted automatically (any events of the deleted bucket could still be access), | |
335 | and will have to be deleted explicitly with the subscription deletion API | |
336 | - Filtering based on metadata (which is an extension to S3) is not supported, and such rules will be ignored | |
9f95a23c | 337 | - Filtering based on tags (which is an extension to S3) is not supported, and such rules will be ignored |
11fdf7f2 | 338 | |
eafe8130 TL |
339 | |
340 | Non S3-Compliant Notifications | |
341 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
11fdf7f2 TL |
342 | |
343 | Create a Notification | |
eafe8130 | 344 | ````````````````````` |
11fdf7f2 TL |
345 | |
346 | This will create a publisher for a specific bucket into a topic. | |
347 | ||
348 | :: | |
349 | ||
350 | PUT /notifications/bucket/<bucket>?topic=<topic-name>[&events=<event>[,<event>]] | |
351 | ||
eafe8130 | 352 | Request parameters: |
11fdf7f2 | 353 | |
eafe8130 TL |
354 | - topic-name: name of topic |
355 | - event: event type (string), one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE`` | |
356 | ||
11fdf7f2 | 357 | Delete Notification Information |
eafe8130 | 358 | ``````````````````````````````` |
11fdf7f2 TL |
359 | |
360 | Delete publisher from a specific bucket into a specific topic. | |
361 | ||
362 | :: | |
363 | ||
364 | DELETE /notifications/bucket/<bucket>?topic=<topic-name> | |
365 | ||
eafe8130 TL |
366 | Request parameters: |
367 | ||
368 | - topic-name: name of topic | |
369 | ||
370 | .. note:: When the bucket is deleted, any notification defined on it is also deleted | |
371 | ||
372 | List Notifications | |
373 | `````````````````` | |
374 | ||
375 | List all topics with associated events defined on a bucket. | |
376 | ||
377 | :: | |
378 | ||
379 | GET /notifications/bucket/<bucket> | |
380 | ||
381 | Response will have the following format (JSON): | |
382 | ||
383 | :: | |
11fdf7f2 | 384 | |
eafe8130 TL |
385 | {"topics":[ |
386 | { | |
387 | "topic":{ | |
388 | "user":"", | |
389 | "name":"", | |
390 | "dest":{ | |
391 | "bucket_name":"", | |
392 | "oid_prefix":"", | |
393 | "push_endpoint":"", | |
9f95a23c TL |
394 | "push_endpoint_args":"", |
395 | "push_endpoint_topic":"" | |
eafe8130 TL |
396 | } |
397 | "arn":"" | |
398 | }, | |
399 | "events":[] | |
400 | } | |
401 | ]} | |
11fdf7f2 | 402 | |
eafe8130 TL |
403 | Subscriptions |
404 | ~~~~~~~~~~~~~ | |
11fdf7f2 | 405 | |
eafe8130 TL |
406 | Create a Subscription |
407 | ````````````````````` | |
11fdf7f2 TL |
408 | |
409 | Creates a new subscription. | |
410 | ||
411 | :: | |
412 | ||
e306af50 | 413 | PUT /subscriptions/<sub-name>?topic=<topic-name>[?push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=none|broker|routable][&verify-ssl=true|false][&kafka-ack-level=none|broker][&ca-location=<file path>]] |
11fdf7f2 | 414 | |
eafe8130 | 415 | Request parameters: |
11fdf7f2 | 416 | |
eafe8130 TL |
417 | - topic-name: name of topic |
418 | - push-endpoint: URI of endpoint to send push notification to | |
11fdf7f2 | 419 | |
9f95a23c TL |
420 | The endpoint URI may include parameters depending with the type of endpoint: |
421 | ||
422 | - HTTP endpoint | |
11fdf7f2 | 423 | |
9f95a23c TL |
424 | - URI: ``http[s]://<fqdn>[:<port]`` |
425 | - port defaults to: 80/443 for HTTP/S accordingly | |
426 | - verify-ssl: indicate whether the server certificate is validated by the client or not ("true" by default) | |
427 | ||
428 | - AMQP0.9.1 endpoint | |
429 | ||
430 | - URI: ``amqp://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]`` | |
431 | - user/password defaults to : guest/guest | |
432 | - port defaults to: 5672 | |
433 | - vhost defaults to: "/" | |
434 | - amqp-exchange: the exchanges must exist and be able to route messages based on topics (mandatory parameter for AMQP0.9.1) | |
e306af50 | 435 | - amqp-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Three ack methods exist: |
9f95a23c TL |
436 | |
437 | - "none": message is considered "delivered" if sent to broker | |
438 | - "broker": message is considered "delivered" if acked by broker (default) | |
e306af50 | 439 | - "routable": message is considered "delivered" if broker can route to a consumer |
9f95a23c TL |
440 | |
441 | - Kafka endpoint | |
442 | ||
443 | - URI: ``kafka://[<user>:<password>@]<fqdn>[:<port]`` | |
444 | - if ``ca-location`` is provided, secure connection will be used for connection with the broker | |
445 | - user/password may only be provided over HTTPS. Topic creation request will be rejected if not | |
446 | - user/password may only be provided together with ``ca-location``. Topic creation request will be rejected if not | |
447 | - port defaults to: 9092 | |
448 | - kafka-ack-level: no end2end acking is required, as messages may persist in the broker before delivered into their final destination. Two ack methods exist: | |
449 | ||
450 | - "none": message is considered "delivered" if sent to broker | |
451 | - "broker": message is considered "delivered" if acked by broker (default) | |
11fdf7f2 | 452 | |
11fdf7f2 | 453 | |
eafe8130 TL |
454 | Get Subscription Information |
455 | ```````````````````````````` | |
456 | ||
457 | Returns information about specific subscription. | |
11fdf7f2 TL |
458 | |
459 | :: | |
460 | ||
461 | GET /subscriptions/<sub-name> | |
462 | ||
eafe8130 TL |
463 | Response will have the following format (JSON): |
464 | ||
465 | :: | |
466 | ||
467 | { | |
468 | "user":"", | |
469 | "name":"", | |
470 | "topic":"", | |
471 | "dest":{ | |
472 | "bucket_name":"", | |
473 | "oid_prefix":"", | |
474 | "push_endpoint":"", | |
9f95a23c TL |
475 | "push_endpoint_args":"", |
476 | "push_endpoint_topic":"" | |
eafe8130 TL |
477 | } |
478 | "s3_id":"" | |
479 | } | |
480 | ||
481 | - user: name of the user that created the subscription | |
482 | - name: name of the subscription | |
483 | - topic: name of the topic the subscription is associated with | |
9f95a23c TL |
484 | - dest.bucket_name: name of the bucket storing the events |
485 | - dest.oid_prefix: oid prefix for the events stored in the bucket | |
486 | - dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL | |
487 | - if push-endpoint URL contain user/password information, request must be made over HTTPS. Topic get request will be rejected if not | |
488 | - dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args | |
489 | - dest.push_endpoint_topic: in case of S3-compliant notifications, this value will hold the topic name as sent to the endpoint (may be different than the internal topic name) | |
490 | - s3_id: in case of S3-compliant notifications, this will hold the notification name that created the subscription | |
11fdf7f2 TL |
491 | |
492 | Delete Subscription | |
eafe8130 | 493 | ``````````````````` |
11fdf7f2 | 494 | |
eafe8130 | 495 | Removes a subscription. |
11fdf7f2 TL |
496 | |
497 | :: | |
498 | ||
499 | DELETE /subscriptions/<sub-name> | |
500 | ||
11fdf7f2 TL |
501 | Events |
502 | ~~~~~~ | |
503 | ||
504 | Pull Events | |
eafe8130 | 505 | ``````````` |
11fdf7f2 | 506 | |
eafe8130 | 507 | Pull events sent to a specific subscription. |
11fdf7f2 TL |
508 | |
509 | :: | |
510 | ||
511 | GET /subscriptions/<sub-name>?events[&max-entries=<max-entries>][&marker=<marker>] | |
512 | ||
eafe8130 TL |
513 | Request parameters: |
514 | ||
515 | - marker: pagination marker for list of events, if not specified will start from the oldest | |
516 | - max-entries: max number of events to return | |
517 | ||
518 | The response will hold information on the current marker and whether there are more events not fetched: | |
519 | ||
520 | :: | |
521 | ||
522 | {"next_marker":"","is_truncated":"",...} | |
523 | ||
524 | ||
525 | The actual content of the response is depended with how the subscription was created. | |
526 | In case that the subscription was created via an S3-compatible notification, | |
527 | the events will have an S3-compatible record format (JSON): | |
528 | ||
529 | :: | |
530 | ||
531 | {"Records":[ | |
532 | { | |
533 | "eventVersion":"2.1" | |
534 | "eventSource":"aws:s3", | |
535 | "awsRegion":"", | |
536 | "eventTime":"", | |
537 | "eventName":"", | |
538 | "userIdentity":{ | |
539 | "principalId":"" | |
540 | }, | |
541 | "requestParameters":{ | |
542 | "sourceIPAddress":"" | |
543 | }, | |
544 | "responseElements":{ | |
545 | "x-amz-request-id":"", | |
546 | "x-amz-id-2":"" | |
547 | }, | |
548 | "s3":{ | |
549 | "s3SchemaVersion":"1.0", | |
550 | "configurationId":"", | |
551 | "bucket":{ | |
552 | "name":"", | |
553 | "ownerIdentity":{ | |
554 | "principalId":"" | |
555 | }, | |
556 | "arn":"", | |
557 | "id":"" | |
558 | }, | |
559 | "object":{ | |
560 | "key":"", | |
561 | "size":"0", | |
562 | "eTag":"", | |
563 | "versionId":"", | |
564 | "sequencer":"", | |
9f95a23c TL |
565 | "metadata":[], |
566 | "tags":[] | |
eafe8130 TL |
567 | } |
568 | }, | |
569 | "eventId":"", | |
9f95a23c | 570 | "opaqueData":"", |
eafe8130 TL |
571 | } |
572 | ]} | |
573 | ||
574 | - awsRegion: zonegroup | |
575 | - eventTime: timestamp indicating when the event was triggered | |
576 | - eventName: either ``s3:ObjectCreated:``, or ``s3:ObjectRemoved:`` | |
577 | - userIdentity: not supported | |
578 | - requestParameters: not supported | |
579 | - responseElements: not supported | |
580 | - s3.configurationId: notification ID that created the subscription for the event | |
eafe8130 TL |
581 | - s3.bucket.name: name of the bucket |
582 | - s3.bucket.ownerIdentity.principalId: owner of the bucket | |
583 | - s3.bucket.arn: ARN of the bucket | |
584 | - s3.bucket.id: Id of the bucket (an extension to the S3 notification API) | |
585 | - s3.object.key: object key | |
586 | - s3.object.size: not supported | |
587 | - s3.object.eTag: object etag | |
588 | - s3.object.version: object version in case of versioned bucket | |
589 | - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) | |
590 | - s3.object.metadata: not supported (an extension to the S3 notification API) | |
9f95a23c | 591 | - s3.object.tags: not supported (an extension to the S3 notification API) |
eafe8130 | 592 | - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API) |
20effc67 | 593 | - s3.opaqueData: opaque data is set in the topic configuration and added to all notifications triggered by the topic (an extension to the S3 notification API) |
eafe8130 TL |
594 | |
595 | In case that the subscription was not created via a non S3-compatible notification, | |
596 | the events will have the following event format (JSON): | |
597 | ||
598 | :: | |
11fdf7f2 | 599 | |
eafe8130 TL |
600 | {"events":[ |
601 | { | |
602 | "id":"", | |
603 | "event":"", | |
604 | "timestamp":"", | |
605 | "info":{ | |
606 | "attrs":{ | |
607 | "mtime":"" | |
608 | }, | |
609 | "bucket":{ | |
610 | "bucket_id":"", | |
611 | "name":"", | |
612 | "tenant":"" | |
613 | }, | |
614 | "key":{ | |
615 | "instance":"", | |
616 | "name":"" | |
617 | } | |
618 | } | |
619 | } | |
620 | ]} | |
621 | ||
622 | - id: unique ID of the event, that could be used for acking | |
623 | - event: one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE`` | |
624 | - timestamp: timestamp indicating when the event was sent | |
625 | - info.attrs.mtime: timestamp indicating when the event was triggered | |
626 | - info.bucket.bucket_id: id of the bucket | |
627 | - info.bucket.name: name of the bucket | |
628 | - info.bucket.tenant: tenant the bucket belongs to | |
629 | - info.key.instance: object version in case of versioned bucket | |
630 | - info.key.name: object key | |
11fdf7f2 TL |
631 | |
632 | Ack Event | |
eafe8130 | 633 | ````````` |
11fdf7f2 TL |
634 | |
635 | Ack event so that it can be removed from the subscription history. | |
636 | ||
637 | :: | |
638 | ||
639 | POST /subscriptions/<sub-name>?ack&event-id=<event-id> | |
640 | ||
eafe8130 | 641 | Request parameters: |
11fdf7f2 | 642 | |
eafe8130 | 643 | - event-id: id of event to be acked |
11fdf7f2 | 644 | |
eafe8130 TL |
645 | .. _Bucket Notification : ../notifications |
646 | .. _Bucket Operations: ../s3/bucketops |