]>
Commit | Line | Data |
---|---|---|
11fdf7f2 TL |
1 | ========================= |
2 | PubSub Sync Module | |
3 | ========================= | |
4 | ||
5 | .. versionadded:: Nautilus | |
6 | ||
7 | This sync module provides a publish and subscribe mechanism for the object store modification | |
8 | events. Events are published into defined topics. Topics can be subscribed to, and events | |
9 | can be pulled from them. Events need to be acked. Also, events will expire and disappear | |
10 | after a period of time. A push notification mechanism exists too, currently supporting HTTP and | |
11 | AMQP0.9.1 endpoints. | |
12 | ||
13 | A user can create different topics. A topic entity is defined by its user and its name. A | |
14 | user can only manage its own topics, and can only subscribe to events published by buckets | |
15 | it owns. | |
16 | ||
17 | In order to publish events for specific bucket a notification needs to be created. A | |
18 | notification can be created only on subset of event types, or for all event types (default). | |
19 | There can be multiple notifications for any specific topic. | |
20 | ||
21 | A subscription to a topic can also be defined. There can be multiple subscriptions for any | |
22 | specific topic. | |
23 | ||
24 | A new REST api has been defined to provide configuration and control interfaces for the pubsub | |
25 | mechanisms. | |
26 | ||
27 | Events are stored as rgw objects in a special bucket, under a special user. Events cannot | |
28 | be accessed directly, but need to be pulled and acked using the new REST api. | |
29 | ||
30 | ||
31 | ||
32 | PubSub Tier Type Configuration | |
33 | ------------------------------------- | |
34 | ||
35 | :: | |
36 | ||
37 | { | |
38 | "tenant": <tenant>, # default: <empty> | |
39 | "uid": <uid>, # default: "pubsub" | |
40 | "data_bucket_prefix": <prefix> # default: "pubsub-" | |
41 | "data_oid_prefix": <prefix> # | |
42 | ||
43 | "events_retention_days": <days> # default: 7 | |
44 | } | |
45 | ||
46 | ||
47 | ||
48 | * ``tenant`` (string) | |
49 | ||
50 | The tenant of the pubsub control user. | |
51 | ||
52 | * ``uid`` (string) | |
53 | ||
54 | The uid of the pubsub control user. | |
55 | ||
56 | * ``data_bucket_prefix`` (string) | |
57 | ||
58 | The prefix of the bucket name that will be created to store events for specific topic. | |
59 | ||
60 | * ``data_oid_prefix`` (string) | |
61 | ||
62 | The oid prefix for the stored events. | |
63 | ||
64 | * ``events_retention_days`` (integer) | |
65 | ||
66 | How many days to keep events that weren't acked. | |
67 | ||
68 | How to Configure | |
69 | ~~~~~~~~~~~~~~~~ | |
70 | ||
71 | See `Multisite Configuration`_ for how to multisite config instructions. The pubsub sync module requires a creation of a new zone. The zone | |
72 | tier type needs to be defined as ``pubsub``: | |
73 | ||
74 | :: | |
75 | ||
76 | # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ | |
77 | --rgw-zone={zone-name} \ | |
78 | --endpoints={http://fqdn}[,{http://fqdn}] | |
79 | --tier-type=pubsub | |
80 | ||
81 | ||
82 | The tier configuration can be then done using the following command | |
83 | ||
84 | :: | |
85 | ||
86 | # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ | |
87 | --rgw-zone={zone-name} \ | |
88 | --tier-config={key}={val}[,{key}={val}] | |
89 | ||
90 | The ``key`` in the configuration specifies the config variable that needs to be updated, and | |
91 | the ``val`` specifies its new value. Nested values can be accessed using period. For example: | |
92 | ||
93 | :: | |
94 | ||
95 | # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ | |
96 | --rgw-zone={zone-name} \ | |
97 | --tier-config=uid=pubsub | |
98 | ||
99 | ||
100 | A configuration field can be removed by using ``--tier-config-rm={key}``. | |
101 | ||
102 | PubSub Performance Stats | |
103 | ------------------------- | |
104 | - **pubsub_event_triggered**: running counter of events with at lease one pubsub topic associated with them | |
105 | - **pubsub_event_lost**: running counter of events that had pubsub topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions | |
106 | - **pubsub_store_ok**: running counter, for all subscriptions, of stored pubsub events | |
107 | - **pubsub_store_fail**: running counter, for all subscriptions, of pubsub events that needed to be stored but failed | |
108 | - **pubsub_push_ok**: running counter, for all subscriptions, of pubsub events successfully pushed to their endpoint | |
109 | - **pubsub_push_fail**: running counter, for all subscriptions, of pubsub events failed to be pushed to their endpoint | |
110 | - **pubsub_push_pending**: gauge value of pubsub events pushed to a endpoined but not acked or nacked yet | |
111 | ||
112 | Note that **pubsub_event_triggered** and **pubsub_event_lost** are incremented per event, while: **pubsub_store_ok**, **pubsub_store_fail**, **pubsub_push_ok**, **pubsub_push_fail**, are incremented per store/push action on each subscriptions. | |
113 | ||
114 | PubSub REST API | |
115 | ------------------------- | |
116 | ||
117 | ||
118 | Topics | |
119 | ~~~~~~ | |
120 | ||
121 | Create a Topic | |
122 | `````````````````````````` | |
123 | ||
124 | This will create a new topic. | |
125 | ||
126 | :: | |
127 | ||
128 | PUT /topics/<topic-name> | |
129 | ||
130 | ||
131 | Get Topic Information | |
132 | ```````````````````````````````` | |
133 | ||
134 | Returns information about specific topic. This includes subscriptions to that topic. | |
135 | ||
136 | :: | |
137 | ||
138 | GET /topics/<topic-name> | |
139 | ||
140 | ||
141 | ||
142 | Delete Topic | |
143 | ```````````````````````````````````` | |
144 | ||
145 | :: | |
146 | ||
147 | DELETE /topics/<topic-name> | |
148 | ||
149 | Delete the specified topic. | |
150 | ||
151 | List Topics | |
152 | ```````````````````````````````````` | |
153 | ||
154 | List all topics that user defined. | |
155 | ||
156 | :: | |
157 | ||
158 | GET /topics | |
159 | ||
160 | ||
161 | ||
162 | Notifications | |
163 | ~~~~~~~~~~~~~ | |
164 | ||
165 | Create a Notification | |
166 | `````````````````````````` | |
167 | ||
168 | This will create a publisher for a specific bucket into a topic. | |
169 | ||
170 | :: | |
171 | ||
172 | PUT /notifications/bucket/<bucket>?topic=<topic-name>[&events=<event>[,<event>]] | |
173 | ||
174 | ||
175 | Request Params: | |
176 | - topic-name: name of topic | |
177 | - event: event type (string), one of: OBJECT_CREATE, OBJECT_DELETE | |
178 | ||
179 | ||
180 | ||
181 | Delete Notification Information | |
182 | ```````````````````````````````` | |
183 | ||
184 | Delete publisher from a specific bucket into a specific topic. | |
185 | ||
186 | :: | |
187 | ||
188 | DELETE /notifications/bucket/<bucket>?topic=<topic-name> | |
189 | ||
190 | Request Params: | |
191 | - topic-name: name of topic | |
192 | ||
193 | ||
194 | ||
195 | Create Subscription | |
196 | ```````````````````````````````````` | |
197 | ||
198 | Creates a new subscription. | |
199 | ||
200 | :: | |
201 | ||
202 | PUT /subscriptions/<sub-name>?topic=<topic-name>[&push-endpoint=<endpoint>[&amqp-exchange=<exchange>][&amqp-ack-level=<level>][&verify-ssl=true|false]] | |
203 | ||
204 | Request Params: | |
205 | ||
206 | - topic-name: name of topic | |
207 | - push-endpoint: URI of endpoint to send push notification to | |
208 | ||
209 | - URI schema is: ``http|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]`` | |
210 | - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) | |
211 | - Default values for HTTP: no user/password, port 80 | |
212 | - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" | |
213 | ||
214 | - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) | |
215 | - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics | |
216 | - amqp-ack-level: 2 ack levels exist: "none" - message is considered "delivered" if sent to broker; | |
217 | "broker" message is considered "delivered" if acked by broker. | |
218 | No end2end acking is required, as messages may persist in the broker before delivered into their final destination | |
219 | ||
220 | Get Subscription Info | |
221 | ```````````````````````````````````` | |
222 | ||
223 | Returns info about specific subscription | |
224 | ||
225 | :: | |
226 | ||
227 | GET /subscriptions/<sub-name> | |
228 | ||
229 | ||
230 | Delete Subscription | |
231 | ````````````````````````````````` | |
232 | ||
233 | Removes a subscription | |
234 | ||
235 | :: | |
236 | ||
237 | DELETE /subscriptions/<sub-name> | |
238 | ||
239 | ||
240 | Events | |
241 | ~~~~~~ | |
242 | ||
243 | Pull Events | |
244 | ````````````````````````````````` | |
245 | ||
246 | Pull events sent to a specific subscription | |
247 | ||
248 | :: | |
249 | ||
250 | GET /subscriptions/<sub-name>?events[&max-entries=<max-entries>][&marker=<marker>] | |
251 | ||
252 | Request Params: | |
253 | - marker: pagination marker for list of events, if not specified will start from the oldest | |
254 | - max-entries: max number of events to return | |
255 | ||
256 | ||
257 | Ack Event | |
258 | ````````````````````````````````` | |
259 | ||
260 | Ack event so that it can be removed from the subscription history. | |
261 | ||
262 | :: | |
263 | ||
264 | POST /subscriptions/<sub-name>?ack&event-id=<event-id> | |
265 | ||
266 | ||
267 | Request Params: | |
268 | - event-id: id of event to be acked | |
269 | ||
270 | .. _Multisite Configuration: ./multisite.rst |