]>
Commit | Line | Data |
---|---|---|
11fdf7f2 TL |
1 | ========================= |
2 | ElasticSearch Sync Module | |
3 | ========================= | |
4 | ||
5 | .. versionadded:: Kraken | |
6 | ||
7 | This sync module writes the metadata from other zones to `ElasticSearch`_. As of | |
8 | luminous this is a json of data fields we currently store in ElasticSearch. | |
9 | ||
10 | :: | |
11 | ||
12 | { | |
13 | "_index" : "rgw-gold-ee5863d6", | |
14 | "_type" : "object", | |
15 | "_id" : "34137443-8592-48d9-8ca7-160255d52ade.34137.1:object1:null", | |
16 | "_score" : 1.0, | |
17 | "_source" : { | |
18 | "bucket" : "testbucket123", | |
19 | "name" : "object1", | |
20 | "instance" : "null", | |
21 | "versioned_epoch" : 0, | |
22 | "owner" : { | |
23 | "id" : "user1", | |
24 | "display_name" : "user1" | |
25 | }, | |
26 | "permissions" : [ | |
27 | "user1" | |
28 | ], | |
29 | "meta" : { | |
30 | "size" : 712354, | |
31 | "mtime" : "2017-05-04T12:54:16.462Z", | |
32 | "etag" : "7ac66c0f148de9519b8bd264312c4d64" | |
33 | } | |
34 | } | |
35 | } | |
36 | ||
37 | ||
38 | ||
39 | ElasticSearch tier type configurables | |
40 | ------------------------------------- | |
41 | ||
42 | * ``endpoint`` | |
43 | ||
44 | Specifies the Elasticsearch server endpoint to access | |
45 | ||
46 | * ``num_shards`` (integer) | |
47 | ||
48 | The number of shards that Elasticsearch will be configured with on | |
49 | data sync initialization. Note that this cannot be changed after init. | |
50 | Any change here requires rebuild of the Elasticsearch index and reinit | |
51 | of the data sync process. | |
52 | ||
53 | * ``num_replicas`` (integer) | |
54 | ||
55 | The number of the replicas that Elasticsearch will be configured with | |
56 | on data sync initialization. | |
57 | ||
58 | * ``explicit_custom_meta`` (true | false) | |
59 | ||
60 | Specifies whether all user custom metadata will be indexed, or whether | |
61 | user will need to configure (at the bucket level) what custom | |
62 | metadata entries should be indexed. This is false by default | |
63 | ||
64 | * ``index_buckets_list`` (comma separated list of strings) | |
65 | ||
66 | If empty, all buckets will be indexed. Otherwise, only buckets | |
67 | specified here will be indexed. It is possible to provide bucket | |
68 | prefixes (e.g., foo\*), or bucket suffixes (e.g., \*bar). | |
69 | ||
70 | * ``approved_owners_list`` (comma separated list of strings) | |
71 | ||
72 | If empty, buckets of all owners will be indexed (subject to other | |
73 | restrictions), otherwise, only buckets owned by specified owners will | |
74 | be indexed. Suffixes and prefixes can also be provided. | |
75 | ||
76 | * ``override_index_path`` (string) | |
77 | ||
78 | if not empty, this string will be used as the elasticsearch index | |
79 | path. Otherwise the index path will be determined and generated on | |
80 | sync initialization. | |
81 | ||
82 | ||
83 | End user metadata queries | |
84 | ------------------------- | |
85 | ||
86 | .. versionadded:: Luminous | |
87 | ||
88 | Since the ElasticSearch cluster now stores object metadata, it is important that | |
89 | the ElasticSearch endpoint is not exposed to the public and only accessible to | |
90 | the cluster administrators. For exposing metadata queries to the end user itself | |
91 | this poses a problem since we'd want the user to only query their metadata and | |
92 | not of any other users, this would require the ElasticSearch cluster to | |
93 | authenticate users in a way similar to RGW does which poses a problem. | |
94 | ||
95 | As of Luminous RGW in the metadata master zone can now service end user | |
96 | requests. This allows for not exposing the elasticsearch endpoint in public and | |
97 | also solves the authentication and authorization problem since RGW itself can | |
98 | authenticate the end user requests. For this purpose RGW introduces a new query | |
9f95a23c | 99 | in the bucket APIs that can service elasticsearch requests. All these requests |
11fdf7f2 TL |
100 | must be sent to the metadata master zone. |
101 | ||
102 | Syntax | |
103 | ~~~~~~ | |
104 | ||
105 | Get an elasticsearch query | |
106 | `````````````````````````` | |
107 | ||
108 | :: | |
109 | ||
110 | GET /{bucket}?query={query-expr} | |
111 | ||
112 | request params: | |
113 | - max-keys: max number of entries to return | |
114 | - marker: pagination marker | |
115 | ||
116 | ``expression := [(]<arg> <op> <value> [)][<and|or> ...]`` | |
117 | ||
118 | op is one of the following: | |
119 | <, <=, ==, >=, > | |
120 | ||
121 | For example :: | |
122 | ||
123 | GET /?query=name==foo | |
124 | ||
125 | Will return all the indexed keys that user has read permission to, and | |
126 | are named 'foo'. | |
127 | ||
128 | Will return all the indexed keys that user has read permission to, and | |
129 | are named 'foo'. | |
130 | ||
131 | The output will be a list of keys in XML that is similar to the S3 | |
132 | list buckets response. | |
133 | ||
134 | Configure custom metadata fields | |
135 | ```````````````````````````````` | |
136 | ||
137 | Define which custom metadata entries should be indexed (under the | |
138 | specified bucket), and what are the types of these keys. If explicit | |
139 | custom metadata indexing is configured, this is needed so that rgw | |
140 | will index the specified custom metadata values. Otherwise it is | |
141 | needed in cases where the indexed metadata keys are of a type other | |
142 | than string. | |
143 | ||
144 | :: | |
145 | ||
146 | POST /{bucket}?mdsearch | |
147 | x-amz-meta-search: <key [; type]> [, ...] | |
148 | ||
149 | Multiple metadata fields must be comma separated, a type can be forced for a | |
150 | field with a `;`. The currently allowed types are string(default), integer and | |
151 | date | |
152 | ||
153 | eg. if you want to index a custom object metadata x-amz-meta-year as int, | |
154 | x-amz-meta-date as type date and x-amz-meta-title as string, you'd do | |
155 | ||
156 | :: | |
157 | ||
158 | POST /mybooks?mdsearch | |
159 | x-amz-meta-search: x-amz-meta-year;int, x-amz-meta-release-date;date, x-amz-meta-title;string | |
160 | ||
161 | ||
162 | Delete custom metadata configuration | |
163 | ```````````````````````````````````` | |
164 | ||
165 | Delete custom metadata bucket configuration. | |
166 | ||
167 | :: | |
168 | ||
169 | DELETE /<bucket>?mdsearch | |
170 | ||
171 | Get custom metadata configuration | |
172 | ````````````````````````````````` | |
173 | ||
174 | Retrieve custom metadata bucket configuration. | |
175 | ||
176 | :: | |
177 | ||
178 | GET /<bucket>?mdsearch | |
179 | ||
180 | ||
181 | .. _`Elasticsearch`: https://github.com/elastic/elasticsearch |