]>
Commit | Line | Data |
---|---|---|
eafe8130 | 1 | ================== |
11fdf7f2 | 2 | PubSub Sync Module |
eafe8130 | 3 | ================== |
11fdf7f2 TL |
4 | |
5 | .. versionadded:: Nautilus | |
6 | ||
eafe8130 TL |
7 | .. contents:: |
8 | ||
11fdf7f2 | 9 | This sync module provides a publish and subscribe mechanism for the object store modification |
eafe8130 | 10 | events. Events are published into predefined topics. Topics can be subscribed to, and events |
11fdf7f2 | 11 | can be pulled from them. Events need to be acked. Also, events will expire and disappear |
eafe8130 TL |
12 | after a period of time. |
13 | ||
14 | A push notification mechanism exists too, currently supporting HTTP and | |
15 | AMQP0.9.1 endpoints, on top of storing the events in Ceph. If events should only be pushed to an endpoint | |
16 | and do not need to be stored in Ceph, the `Bucket Notification`_ mechanism should be used instead of pubsub sync module. | |
11fdf7f2 TL |
17 | |
18 | A user can create different topics. A topic entity is defined by its user and its name. A | |
19 | user can only manage its own topics, and can only subscribe to events published by buckets | |
20 | it owns. | |
21 | ||
eafe8130 TL |
22 | In order to publish events for specific bucket a notification entity needs to be created. A |
23 | notification can be created on a subset of event types, or for all event types (default). | |
24 | There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. | |
11fdf7f2 TL |
25 | |
26 | A subscription to a topic can also be defined. There can be multiple subscriptions for any | |
27 | specific topic. | |
28 | ||
eafe8130 TL |
29 | REST API has been defined to provide configuration and control interfaces for the pubsub |
30 | mechanisms. This API has two flavors, one is S3-compatible and one is not. The two flavors can be used | |
31 | together, although it is recommended to use the S3-compatible one. | |
32 | The S3-compatible API is similar to the one used in the bucket notification mechanism. | |
11fdf7f2 | 33 | |
eafe8130 TL |
34 | Events are stored as RGW objects in a special bucket, under a special user. Events cannot |
35 | be accessed directly, but need to be pulled and acked using the new REST API. | |
11fdf7f2 | 36 | |
eafe8130 TL |
37 | .. toctree:: |
38 | :maxdepth: 1 | |
11fdf7f2 | 39 | |
eafe8130 TL |
40 | S3 Bucket Notification Compatibility <s3-notification-compatibility> |
41 | ||
42 | PubSub Zone Configuration | |
43 | ------------------------- | |
11fdf7f2 | 44 | |
eafe8130 TL |
45 | The pubsub sync module requires the creation of a new zone in a `Multisite`_ environment. |
46 | First, a master zone must exist, then a secondary zone should be created. | |
47 | In the creation of the secondary zone, its tier type must be set to ``pubsub``: | |
11fdf7f2 TL |
48 | |
49 | :: | |
50 | ||
eafe8130 TL |
51 | # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ |
52 | --rgw-zone={zone-name} \ | |
53 | --endpoints={http://fqdn}[,{http://fqdn}] \ | |
54 | --sync-from-all=0 \ | |
55 | --sync-from={master-zone-name} \ | |
56 | --tier-type=pubsub | |
11fdf7f2 | 57 | |
11fdf7f2 | 58 | |
eafe8130 TL |
59 | PubSub Zone Configuration Parameters |
60 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
61 | ||
62 | :: | |
11fdf7f2 | 63 | |
eafe8130 TL |
64 | { |
65 | "tenant": <tenant>, # default: <empty> | |
66 | "uid": <uid>, # default: "pubsub" | |
67 | "data_bucket_prefix": <prefix> # default: "pubsub-" | |
68 | "data_oid_prefix": <prefix> # | |
69 | "events_retention_days": <days> # default: 7 | |
70 | } | |
11fdf7f2 TL |
71 | |
72 | * ``tenant`` (string) | |
73 | ||
74 | The tenant of the pubsub control user. | |
75 | ||
76 | * ``uid`` (string) | |
77 | ||
78 | The uid of the pubsub control user. | |
79 | ||
80 | * ``data_bucket_prefix`` (string) | |
81 | ||
82 | The prefix of the bucket name that will be created to store events for specific topic. | |
83 | ||
84 | * ``data_oid_prefix`` (string) | |
85 | ||
86 | The oid prefix for the stored events. | |
87 | ||
88 | * ``events_retention_days`` (integer) | |
89 | ||
90 | How many days to keep events that weren't acked. | |
91 | ||
eafe8130 TL |
92 | Configuring Parameters via CLI |
93 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
11fdf7f2 | 94 | |
eafe8130 | 95 | The tier configuration could be set using the following command: |
11fdf7f2 TL |
96 | |
97 | :: | |
98 | ||
eafe8130 | 99 | # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ |
11fdf7f2 TL |
100 | --rgw-zone={zone-name} \ |
101 | --tier-config={key}={val}[,{key}={val}] | |
102 | ||
eafe8130 TL |
103 | Where the ``key`` in the configuration specifies the configuration variable that needs to be updated (from the list above), and |
104 | the ``val`` specifies its new value. For example, setting the pubsub control user ``uid`` to ``user_ps``: | |
11fdf7f2 TL |
105 | |
106 | :: | |
107 | ||
eafe8130 | 108 | # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ |
11fdf7f2 TL |
109 | --rgw-zone={zone-name} \ |
110 | --tier-config=uid=pubsub | |
111 | ||
11fdf7f2 TL |
112 | A configuration field can be removed by using ``--tier-config-rm={key}``. |
113 | ||
114 | PubSub Performance Stats | |
115 | ------------------------- | |
eafe8130 TL |
116 | Same counters are shared between the pubsub sync module and the notification mechanism. |
117 | ||
118 | - ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them | |
119 | - ``pubsub_event_lost``: running counter of events that had topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions | |
120 | - ``pubsub_store_ok``: running counter, for all subscriptions, of stored events | |
121 | - ``pubsub_store_fail``: running counter, for all subscriptions, of events failed to be stored | |
122 | - ``pubsub_push_ok``: running counter, for all subscriptions, of events successfully pushed to their endpoint | |
123 | - ``pubsub_push_fail``: running counter, for all subscriptions, of events failed to be pushed to their endpoint | |
124 | - ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet | |
11fdf7f2 | 125 | |
eafe8130 TL |
126 | .. note:: |
127 | ||
128 | ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while: | |
129 | ``pubsub_store_ok``, ``pubsub_store_fail``, ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per store/push action on each subscriptions. | |
11fdf7f2 TL |
130 | |
131 | PubSub REST API | |
eafe8130 | 132 | --------------- |
11fdf7f2 | 133 | |
eafe8130 | 134 | .. tip:: PubSub REST calls, and only them, should be sent to an RGW which belong to a PubSub zone |
11fdf7f2 TL |
135 | |
136 | Topics | |
137 | ~~~~~~ | |
eafe8130 | 138 | |
11fdf7f2 | 139 | Create a Topic |
eafe8130 TL |
140 | `````````````` |
141 | ||
142 | This will create a new topic. Topic creation is needed both for both flavors of the API. | |
143 | Optionally the topic could be provided with push endpoint parameters that would be used later | |
144 | when an S3-compatible notification is created. | |
145 | Upon successful request, the response will include the topic ARN that could be later used to reference this topic in an S3-compatible notification request. | |
146 | To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values. | |
11fdf7f2 | 147 | |
eafe8130 | 148 | .. tip:: Any S3-compatible notification already associated with the topic needs to be re-created for the topic update to take effect |
11fdf7f2 TL |
149 | |
150 | :: | |
151 | ||
eafe8130 TL |
152 | PUT /topics/<topic-name>[?push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=<level>][&verify-ssl=true|false]] |
153 | ||
154 | Request parameters: | |
155 | ||
156 | - push-endpoint: URI of endpoint to send push notification to | |
157 | ||
158 | - URI schema is: ``http[s]|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]`` | |
159 | - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) | |
160 | - Default values for HTTP/S: no user/password, port 80/443 | |
161 | - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" | |
162 | ||
163 | - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) | |
164 | - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics | |
165 | - amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: | |
166 | ||
167 | - "none" - message is considered "delivered" if sent to broker | |
168 | - "broker" message is considered "delivered" if acked by broker | |
169 | ||
170 | The topic ARN in the response will have the following format: | |
11fdf7f2 | 171 | |
eafe8130 TL |
172 | :: |
173 | ||
174 | arn:aws:sns:<zone-group>:<tenant>:<topic> | |
11fdf7f2 TL |
175 | |
176 | Get Topic Information | |
eafe8130 | 177 | ````````````````````` |
11fdf7f2 | 178 | |
eafe8130 | 179 | Returns information about specific topic. This includes subscriptions to that topic, and push-endpoint information, if provided. |
11fdf7f2 TL |
180 | |
181 | :: | |
182 | ||
183 | GET /topics/<topic-name> | |
184 | ||
eafe8130 TL |
185 | Response will have the following format (JSON): |
186 | ||
187 | :: | |
11fdf7f2 | 188 | |
eafe8130 TL |
189 | { |
190 | "topic":{ | |
191 | "user":"", | |
192 | "name":"", | |
193 | "dest":{ | |
194 | "bucket_name":"", | |
195 | "oid_prefix":"", | |
196 | "push_endpoint":"", | |
197 | "push_endpoint_args":"" | |
198 | }, | |
199 | "arn":"" | |
200 | }, | |
201 | "subs":[] | |
202 | } | |
203 | ||
204 | - topic.user: name of the user that created the topic | |
205 | - name: name of the topic | |
206 | - dest.bucket_name: not used | |
207 | - dest.oid_prefix: not used | |
208 | - dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL | |
209 | - dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args | |
210 | - topic.arn: topic ARN | |
211 | - subs: list of subscriptions associated with this topic | |
11fdf7f2 TL |
212 | |
213 | Delete Topic | |
eafe8130 | 214 | ```````````` |
11fdf7f2 TL |
215 | |
216 | :: | |
217 | ||
218 | DELETE /topics/<topic-name> | |
219 | ||
220 | Delete the specified topic. | |
221 | ||
222 | List Topics | |
eafe8130 | 223 | ``````````` |
11fdf7f2 TL |
224 | |
225 | List all topics that user defined. | |
226 | ||
227 | :: | |
228 | ||
229 | GET /topics | |
eafe8130 TL |
230 | |
231 | S3-Compliant Notifications | |
232 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
11fdf7f2 | 233 | |
eafe8130 | 234 | Detailed under: `Bucket Operations`_. |
11fdf7f2 | 235 | |
eafe8130 TL |
236 | .. note:: |
237 | ||
238 | - Notification creation will also create a subscription for pushing/pulling events | |
239 | - The generated subscription's name will have the same as the notification Id, and could be used later to fetch and ack events with the subscription API. | |
240 | - Notification deletion will deletes all generated subscriptions | |
241 | - In case that bucket deletion implicitly deletes the notification, | |
242 | the associated subscription will not be deleted automatically (any events of the deleted bucket could still be access), | |
243 | and will have to be deleted explicitly with the subscription deletion API | |
244 | - Filtering based on metadata (which is an extension to S3) is not supported, and such rules will be ignored | |
11fdf7f2 | 245 | |
eafe8130 TL |
246 | |
247 | Non S3-Compliant Notifications | |
248 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
11fdf7f2 TL |
249 | |
250 | Create a Notification | |
eafe8130 | 251 | ````````````````````` |
11fdf7f2 TL |
252 | |
253 | This will create a publisher for a specific bucket into a topic. | |
254 | ||
255 | :: | |
256 | ||
257 | PUT /notifications/bucket/<bucket>?topic=<topic-name>[&events=<event>[,<event>]] | |
258 | ||
eafe8130 | 259 | Request parameters: |
11fdf7f2 | 260 | |
eafe8130 TL |
261 | - topic-name: name of topic |
262 | - event: event type (string), one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE`` | |
263 | ||
11fdf7f2 | 264 | Delete Notification Information |
eafe8130 | 265 | ``````````````````````````````` |
11fdf7f2 TL |
266 | |
267 | Delete publisher from a specific bucket into a specific topic. | |
268 | ||
269 | :: | |
270 | ||
271 | DELETE /notifications/bucket/<bucket>?topic=<topic-name> | |
272 | ||
eafe8130 TL |
273 | Request parameters: |
274 | ||
275 | - topic-name: name of topic | |
276 | ||
277 | .. note:: When the bucket is deleted, any notification defined on it is also deleted | |
278 | ||
279 | List Notifications | |
280 | `````````````````` | |
281 | ||
282 | List all topics with associated events defined on a bucket. | |
283 | ||
284 | :: | |
285 | ||
286 | GET /notifications/bucket/<bucket> | |
287 | ||
288 | Response will have the following format (JSON): | |
289 | ||
290 | :: | |
11fdf7f2 | 291 | |
eafe8130 TL |
292 | {"topics":[ |
293 | { | |
294 | "topic":{ | |
295 | "user":"", | |
296 | "name":"", | |
297 | "dest":{ | |
298 | "bucket_name":"", | |
299 | "oid_prefix":"", | |
300 | "push_endpoint":"", | |
301 | "push_endpoint_args":"" | |
302 | } | |
303 | "arn":"" | |
304 | }, | |
305 | "events":[] | |
306 | } | |
307 | ]} | |
11fdf7f2 | 308 | |
eafe8130 TL |
309 | Subscriptions |
310 | ~~~~~~~~~~~~~ | |
11fdf7f2 | 311 | |
eafe8130 TL |
312 | Create a Subscription |
313 | ````````````````````` | |
11fdf7f2 TL |
314 | |
315 | Creates a new subscription. | |
316 | ||
317 | :: | |
318 | ||
319 | PUT /subscriptions/<sub-name>?topic=<topic-name>[&push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=<level>][&verify-ssl=true|false]] | |
320 | ||
eafe8130 | 321 | Request parameters: |
11fdf7f2 | 322 | |
eafe8130 TL |
323 | - topic-name: name of topic |
324 | - push-endpoint: URI of endpoint to send push notification to | |
11fdf7f2 | 325 | |
eafe8130 TL |
326 | - URI schema is: ``http[s]|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]`` |
327 | - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) | |
328 | - Default values for HTTP/S: no user/password, port 80/443 | |
329 | - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" | |
11fdf7f2 | 330 | |
eafe8130 TL |
331 | - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) |
332 | - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics | |
333 | - amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: | |
11fdf7f2 | 334 | |
eafe8130 TL |
335 | - "none": message is considered "delivered" if sent to broker |
336 | - "broker": message is considered "delivered" if acked by broker | |
11fdf7f2 | 337 | |
eafe8130 TL |
338 | Get Subscription Information |
339 | ```````````````````````````` | |
340 | ||
341 | Returns information about specific subscription. | |
11fdf7f2 TL |
342 | |
343 | :: | |
344 | ||
345 | GET /subscriptions/<sub-name> | |
346 | ||
eafe8130 TL |
347 | Response will have the following format (JSON): |
348 | ||
349 | :: | |
350 | ||
351 | { | |
352 | "user":"", | |
353 | "name":"", | |
354 | "topic":"", | |
355 | "dest":{ | |
356 | "bucket_name":"", | |
357 | "oid_prefix":"", | |
358 | "push_endpoint":"", | |
359 | "push_endpoint_args":"" | |
360 | } | |
361 | "s3_id":"" | |
362 | } | |
363 | ||
364 | - user: name of the user that created the subscription | |
365 | - name: name of the subscription | |
366 | - topic: name of the topic the subscription is associated with | |
11fdf7f2 TL |
367 | |
368 | Delete Subscription | |
eafe8130 | 369 | ``````````````````` |
11fdf7f2 | 370 | |
eafe8130 | 371 | Removes a subscription. |
11fdf7f2 TL |
372 | |
373 | :: | |
374 | ||
375 | DELETE /subscriptions/<sub-name> | |
376 | ||
11fdf7f2 TL |
377 | Events |
378 | ~~~~~~ | |
379 | ||
380 | Pull Events | |
eafe8130 | 381 | ``````````` |
11fdf7f2 | 382 | |
eafe8130 | 383 | Pull events sent to a specific subscription. |
11fdf7f2 TL |
384 | |
385 | :: | |
386 | ||
387 | GET /subscriptions/<sub-name>?events[&max-entries=<max-entries>][&marker=<marker>] | |
388 | ||
eafe8130 TL |
389 | Request parameters: |
390 | ||
391 | - marker: pagination marker for list of events, if not specified will start from the oldest | |
392 | - max-entries: max number of events to return | |
393 | ||
394 | The response will hold information on the current marker and whether there are more events not fetched: | |
395 | ||
396 | :: | |
397 | ||
398 | {"next_marker":"","is_truncated":"",...} | |
399 | ||
400 | ||
401 | The actual content of the response is depended with how the subscription was created. | |
402 | In case that the subscription was created via an S3-compatible notification, | |
403 | the events will have an S3-compatible record format (JSON): | |
404 | ||
405 | :: | |
406 | ||
407 | {"Records":[ | |
408 | { | |
409 | "eventVersion":"2.1" | |
410 | "eventSource":"aws:s3", | |
411 | "awsRegion":"", | |
412 | "eventTime":"", | |
413 | "eventName":"", | |
414 | "userIdentity":{ | |
415 | "principalId":"" | |
416 | }, | |
417 | "requestParameters":{ | |
418 | "sourceIPAddress":"" | |
419 | }, | |
420 | "responseElements":{ | |
421 | "x-amz-request-id":"", | |
422 | "x-amz-id-2":"" | |
423 | }, | |
424 | "s3":{ | |
425 | "s3SchemaVersion":"1.0", | |
426 | "configurationId":"", | |
427 | "bucket":{ | |
428 | "name":"", | |
429 | "ownerIdentity":{ | |
430 | "principalId":"" | |
431 | }, | |
432 | "arn":"", | |
433 | "id":"" | |
434 | }, | |
435 | "object":{ | |
436 | "key":"", | |
437 | "size":"0", | |
438 | "eTag":"", | |
439 | "versionId":"", | |
440 | "sequencer":"", | |
92f5a8d4 | 441 | "metadata":[] |
eafe8130 TL |
442 | } |
443 | }, | |
444 | "eventId":"", | |
445 | } | |
446 | ]} | |
447 | ||
448 | - awsRegion: zonegroup | |
449 | - eventTime: timestamp indicating when the event was triggered | |
450 | - eventName: either ``s3:ObjectCreated:``, or ``s3:ObjectRemoved:`` | |
451 | - userIdentity: not supported | |
452 | - requestParameters: not supported | |
453 | - responseElements: not supported | |
454 | - s3.configurationId: notification ID that created the subscription for the event | |
eafe8130 TL |
455 | - s3.bucket.name: name of the bucket |
456 | - s3.bucket.ownerIdentity.principalId: owner of the bucket | |
457 | - s3.bucket.arn: ARN of the bucket | |
458 | - s3.bucket.id: Id of the bucket (an extension to the S3 notification API) | |
459 | - s3.object.key: object key | |
460 | - s3.object.size: not supported | |
461 | - s3.object.eTag: object etag | |
462 | - s3.object.version: object version in case of versioned bucket | |
463 | - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) | |
464 | - s3.object.metadata: not supported (an extension to the S3 notification API) | |
465 | - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API) | |
466 | ||
467 | In case that the subscription was not created via a non S3-compatible notification, | |
468 | the events will have the following event format (JSON): | |
469 | ||
470 | :: | |
11fdf7f2 | 471 | |
eafe8130 TL |
472 | {"events":[ |
473 | { | |
474 | "id":"", | |
475 | "event":"", | |
476 | "timestamp":"", | |
477 | "info":{ | |
478 | "attrs":{ | |
479 | "mtime":"" | |
480 | }, | |
481 | "bucket":{ | |
482 | "bucket_id":"", | |
483 | "name":"", | |
484 | "tenant":"" | |
485 | }, | |
486 | "key":{ | |
487 | "instance":"", | |
488 | "name":"" | |
489 | } | |
490 | } | |
491 | } | |
492 | ]} | |
493 | ||
494 | - id: unique ID of the event, that could be used for acking | |
495 | - event: one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE`` | |
496 | - timestamp: timestamp indicating when the event was sent | |
497 | - info.attrs.mtime: timestamp indicating when the event was triggered | |
498 | - info.bucket.bucket_id: id of the bucket | |
499 | - info.bucket.name: name of the bucket | |
500 | - info.bucket.tenant: tenant the bucket belongs to | |
501 | - info.key.instance: object version in case of versioned bucket | |
502 | - info.key.name: object key | |
11fdf7f2 TL |
503 | |
504 | Ack Event | |
eafe8130 | 505 | ````````` |
11fdf7f2 TL |
506 | |
507 | Ack event so that it can be removed from the subscription history. | |
508 | ||
509 | :: | |
510 | ||
511 | POST /subscriptions/<sub-name>?ack&event-id=<event-id> | |
512 | ||
eafe8130 | 513 | Request parameters: |
11fdf7f2 | 514 | |
eafe8130 | 515 | - event-id: id of event to be acked |
11fdf7f2 | 516 | |
eafe8130 TL |
517 | .. _Multisite : ../multisite |
518 | .. _Bucket Notification : ../notifications | |
519 | .. _Bucket Operations: ../s3/bucketops |