]>
Commit | Line | Data |
---|---|---|
eafe8130 TL |
1 | ==================== |
2 | Bucket Notifications | |
3 | ==================== | |
4 | ||
5 | .. versionadded:: Nautilus | |
6 | ||
7 | .. contents:: | |
8 | ||
9 | Bucket notifications provide a mechanism for sending information out of the radosgw when certain events are happening on the bucket. | |
10 | Currently, notifications could be sent to HTTP and AMQP0.9.1 endpoints. | |
11 | ||
12 | Note, that if the events should be stored in Ceph, in addition, or instead of being pushed to an endpoint, | |
13 | the `PubSub Module`_ should be used instead of the bucket notification mechanism. | |
14 | ||
15 | A user can create different topics. A topic entity is defined by its user and its name. A | |
16 | user can only manage its own topics, and can only associate them with buckets it owns. | |
17 | ||
18 | In order to send notifications for events for a specific bucket, a notification entity needs to be created. A | |
19 | notification can be created on a subset of event types, or for all event types (default). | |
20 | The notification may also filter out events based on preffix/suffix and/or regular expression matching of the keys. As well as, | |
21 | on the metadata attributes attached to the object. | |
22 | There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. | |
23 | ||
24 | REST API has been defined to provide configuration and control interfaces for the bucket notification | |
25 | mechanism. This API is similar to the one defined as S3-compatible API of the pubsub sync module. | |
26 | ||
27 | .. toctree:: | |
28 | :maxdepth: 1 | |
29 | ||
30 | S3 Bucket Notification Compatibility <s3-notification-compatibility> | |
31 | ||
32 | Notificatios Performance Stats | |
33 | ------------------------------ | |
34 | Same counters are shared between the pubsub sync module and the bucket notification mechanism. | |
35 | ||
36 | - ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them | |
37 | - ``pubsub_event_lost``: running counter of events that had topics associated with them but that were not pushed to any of the endpoints | |
38 | - ``pubsub_push_ok``: running counter, for all notifications, of events successfully pushed to their endpoint | |
39 | - ``pubsub_push_fail``: running counter, for all notifications, of events failed to be pushed to their endpoint | |
40 | - ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet | |
41 | ||
42 | .. note:: | |
43 | ||
44 | ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while: | |
45 | ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per push action on each notification. | |
46 | ||
47 | Bucket Notification REST API | |
48 | ---------------------------- | |
49 | ||
50 | Topics | |
51 | ~~~~~~ | |
52 | ||
53 | Create a Topic | |
54 | `````````````` | |
55 | ||
56 | This will create a new topic. The topic should be provided with push endpoint parameters that would be used later | |
57 | when a notification is created. | |
58 | Upon successful request, the response will include the topic ARN that could be later used to reference this topic in the notification request. | |
59 | To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values. | |
60 | ||
61 | .. tip:: Any notification already associated with the topic needs to be re-created for the topic update to take effect | |
62 | ||
63 | :: | |
64 | ||
65 | POST | |
66 | Action=CreateTopic | |
67 | &Name=<topic-name> | |
68 | &push-endpoint=<endpoint> | |
69 | [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>] | |
70 | [&Attributes.entry.2.key=amqp-sck-level&Attributes.entry.2.value=ack-level] | |
71 | &Attributes.entry.3.key=verify-sll&Attributes.entry.3.value=true|false] | |
72 | ||
73 | Request parameters: | |
74 | ||
75 | - push-endpoint: URI of endpoint to send push notification to | |
76 | ||
77 | - URI schema is: ``http[s]|amqp://[<user>:<password>@]<fqdn>[:<port>][/<amqp-vhost>]`` | |
78 | - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) | |
79 | - Default values for HTTP/S: no user/password, port 80/443 | |
80 | - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" | |
81 | ||
82 | - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) | |
83 | - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics | |
84 | - amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: | |
85 | ||
86 | - "none" - message is considered "delivered" if sent to broker | |
87 | - "broker" message is considered "delivered" if acked by broker | |
88 | ||
89 | .. note:: | |
90 | ||
91 | - The key/value of a specific parameter does not have to reside in the same line, or in any specific order, but must use the same index | |
92 | - Attribute indexing does not need to be sequntial or start from any specific value | |
93 | - `AWS Create Topic`_ has detailed explanation on endpoint attributes format. However, in our case different keys and values are used | |
94 | ||
95 | The response will have the following format: | |
96 | ||
97 | :: | |
98 | ||
99 | <CreateTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> | |
100 | <CreateTopicResult> | |
101 | <TopicArn></TopicArn> | |
102 | </CreateTopicResult> | |
103 | <ResponseMetadata> | |
104 | <RequestId></RequestId> | |
105 | </ResponseMetadata> | |
106 | </CreateTopicResponse> | |
107 | ||
108 | The topic ARN in the response will have the following format: | |
109 | ||
110 | :: | |
111 | ||
112 | arn:aws:sns:<zone-group>:<tenant>:<topic> | |
113 | ||
114 | Get Topic Information | |
115 | ````````````````````` | |
116 | ||
117 | Returns information about specific topic. This includes push-endpoint information, if provided. | |
118 | ||
119 | :: | |
120 | ||
121 | POST | |
122 | Action=GetTopic&TopicArn=<topic-arn> | |
123 | ||
124 | Response will have the following format: | |
125 | ||
126 | :: | |
127 | ||
128 | <GetTopicResponse> | |
129 | <GetTopicRersult> | |
130 | <Topic> | |
131 | <User></User> | |
132 | <Name></Name> | |
133 | <EndPoint> | |
134 | <EndpointAddress></EndpointAddress> | |
135 | <EndpointArgs></EndpointArgs> | |
136 | <EndpointTopic></EndpointTopic> | |
137 | </EndPoint> | |
138 | <TopicArn></TopicArn> | |
139 | </Topic> | |
140 | </GetTopicResult> | |
141 | <ResponseMetadata> | |
142 | <RequestId></RequestId> | |
143 | </ResponseMetadata> | |
144 | </GetTopicResponse> | |
145 | ||
146 | - User: name of the user that created the topic | |
147 | - Name: name of the topic | |
148 | - EndPoinjtAddress: the push-endpoint URL | |
149 | - EndPointArgs: the push-endpoint args | |
150 | - EndpointTopic: the topic name that should be sent to the endpoint (mat be different than the above topic name) | |
151 | - TopicArn: topic ARN | |
152 | ||
153 | Delete Topic | |
154 | ```````````` | |
155 | ||
156 | :: | |
157 | ||
158 | POST | |
159 | Action=DeleteTopic&TopicArn=<topic-arn> | |
160 | ||
161 | Delete the specified topic. Note that deleting a deleted topic should result with no-op and not a failure. | |
162 | ||
163 | The response will have the following format: | |
164 | ||
165 | :: | |
166 | ||
167 | <DeleteTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> | |
168 | <ResponseMetadata> | |
169 | <RequestId></RequestId> | |
170 | </ResponseMetadata> | |
171 | </DeleteTopicResponse> | |
172 | ||
173 | List Topics | |
174 | ``````````` | |
175 | ||
176 | List all topics that user defined. | |
177 | ||
178 | :: | |
179 | ||
180 | POST | |
181 | Action=ListTopics | |
182 | ||
183 | Response will have the following format: | |
184 | ||
185 | :: | |
186 | ||
187 | <ListTopicdResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> | |
188 | <ListTopicsRersult> | |
189 | <Topics> | |
190 | <member> | |
191 | <User></User> | |
192 | <Name></Name> | |
193 | <EndPoint> | |
194 | <EndpointAddress></EndpointAddress> | |
195 | <EndpointArgs></EndpointArgs> | |
196 | <EndpointTopic></EndpointTopic> | |
197 | </EndPoint> | |
198 | <TopicArn></TopicArn> | |
199 | </member> | |
200 | </Topics> | |
201 | </ListTopicsResult> | |
202 | <ResponseMetadata> | |
203 | <RequestId></RequestId> | |
204 | </ResponseMetadata> | |
205 | </ListTopicsResponse> | |
206 | ||
207 | Notifications | |
208 | ~~~~~~~~~~~~~ | |
209 | ||
210 | Detailed under: `Bucket Operations`_. | |
211 | ||
212 | .. note:: | |
213 | ||
214 | - "Abort Multipart Upload" request does not emit a notification | |
215 | - "Delete Multiple Objects" request does not emit a notification | |
216 | - Both "Initiate Multipart Upload" and "POST Object" requests will emit an ``s3:ObjectCreated:Post`` notification | |
217 | ||
218 | ||
219 | Events | |
220 | ~~~~~~ | |
221 | ||
222 | The events are in JSON format (regardless of the actual endpoint), and share the same structure as the S3-compatible events | |
223 | pushed or pulled using the pubsub sync module. | |
224 | ||
225 | :: | |
226 | ||
227 | {"Records":[ | |
228 | { | |
229 | "eventVersion":"2.1" | |
230 | "eventSource":"aws:s3", | |
231 | "awsRegion":"", | |
232 | "eventTime":"", | |
233 | "eventName":"", | |
234 | "userIdentity":{ | |
235 | "principalId":"" | |
236 | }, | |
237 | "requestParameters":{ | |
238 | "sourceIPAddress":"" | |
239 | }, | |
240 | "responseElements":{ | |
241 | "x-amz-request-id":"", | |
242 | "x-amz-id-2":"" | |
243 | }, | |
244 | "s3":{ | |
245 | "s3SchemaVersion":"1.0", | |
246 | "configurationId":"", | |
247 | "bucket":{ | |
248 | "name":"", | |
249 | "ownerIdentity":{ | |
250 | "principalId":"" | |
251 | }, | |
252 | "arn":"", | |
253 | "id:"" | |
254 | }, | |
255 | "object":{ | |
256 | "key":"", | |
257 | "size":"", | |
258 | "eTag":"", | |
259 | "versionId":"", | |
260 | "sequencer": "", | |
92f5a8d4 | 261 | "metadata":[] |
eafe8130 TL |
262 | } |
263 | }, | |
264 | "eventId":"", | |
265 | } | |
266 | ]} | |
267 | ||
268 | - awsRegion: zonegroup | |
269 | - eventTime: timestamp indicating when the event was triggered | |
270 | - eventName: for list of supported events see: `S3 Notification Compatibility`_ | |
271 | - userIdentity.principalId: user that triggered the change | |
272 | - requestParameters.sourceIPAddress: not supported | |
273 | - responseElements.x-amz-request-id: request ID of the original change | |
274 | - responseElements.x_amz_id_2: RGW on which the change was made | |
275 | - s3.configurationId: notification ID that created the event | |
276 | - s3.bucket.name: name of the bucket | |
277 | - s3.bucket.ownerIdentity.principalId: owner of the bucket | |
278 | - s3.bucket.arn: ARN of the bucket | |
279 | - s3.bucket.id: Id of the bucket (an extension to the S3 notification API) | |
280 | - s3.object.key: object key | |
281 | - s3.object.size: object size | |
282 | - s3.object.eTag: object etag | |
283 | - s3.object.version: object version in case of versioned bucket | |
284 | - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) | |
285 | - s3.object.metadata: any metadata set on the object sent as: ``x-amz-meta-`` (an extension to the S3 notification API) | |
92f5a8d4 | 286 | - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API) |
eafe8130 TL |
287 | |
288 | .. _PubSub Module : ../pubsub-module | |
289 | .. _S3 Notification Compatibility: ../s3-notification-compatibility | |
290 | .. _AWS Create Topic: https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html | |
291 | .. _Bucket Operations: ../s3/bucketops |