ceph/doc/radosgw/elastic-sync-module.rst

   1 =========================
   2 ElasticSearch Sync Module
   3 =========================
   4
   5 .. versionadded:: Kraken
   6
   7 This sync module writes the metadata from other zones to `ElasticSearch`_. As of
   8 luminous this is a json of data fields we currently store in ElasticSearch.
   9
  10 ::
  11
  12    {
  13         "_index" : "rgw-gold-ee5863d6",
  14         "_type" : "object",
  15         "_id" : "34137443-8592-48d9-8ca7-160255d52ade.34137.1:object1:null",
  16         "_score" : 1.0,
  17         "_source" : {
  18           "bucket" : "testbucket123",
  19           "name" : "object1",
  20           "instance" : "null",
  21           "versioned_epoch" : 0,
  22           "owner" : {
  23             "id" : "user1",
  24             "display_name" : "user1"
  25           },
  26           "permissions" : [
  27             "user1"
  28           ],
  29           "meta" : {
  30             "size" : 712354,
  31             "mtime" : "2017-05-04T12:54:16.462Z",
  32             "etag" : "7ac66c0f148de9519b8bd264312c4d64"
  33           }
  34         }
  35       }
  36
  37
  38
  39 ElasticSearch tier type configurables
  40 -------------------------------------
  41
  42 * ``endpoint``
  43
  44 Specifies the Elasticsearch server endpoint to access
  45
  46 * ``num_shards`` (integer)
  47
  48 The number of shards that Elasticsearch will be configured with on
  49 data sync initialization. Note that this cannot be changed after init.
  50 Any change here requires rebuild of the Elasticsearch index and reinit
  51 of the data sync process.
  52
  53 * ``num_replicas`` (integer)
  54
  55 The number of the replicas that Elasticsearch will be configured with
  56 on data sync initialization.
  57
  58 * ``explicit_custom_meta`` (true | false)
  59
  60 Specifies whether all user custom metadata will be indexed, or whether
  61 user will need to configure (at the bucket level) what custom
  62 metadata entries should be indexed. This is false by default
  63
  64 * ``index_buckets_list`` (comma separated list of strings)
  65
  66 If empty, all buckets will be indexed. Otherwise, only buckets
  67 specified here will be indexed. It is possible to provide bucket
  68 prefixes (e.g., foo\*), or bucket suffixes (e.g., \*bar).
  69
  70 * ``approved_owners_list`` (comma separated list of strings)
  71
  72 If empty, buckets of all owners will be indexed (subject to other
  73 restrictions), otherwise, only buckets owned by specified owners will
  74 be indexed. Suffixes and prefixes can also be provided.
  75
  76 * ``override_index_path`` (string)
  77
  78 if not empty, this string will be used as the elasticsearch index
  79 path. Otherwise the index path will be determined and generated on
  80 sync initialization.
  81
  82
  83 End user metadata queries
  84 -------------------------
  85
  86 .. versionadded:: Luminous
  87
  88 Since the ElasticSearch cluster now stores object metadata, it is important that
  89 the ElasticSearch endpoint is not exposed to the public and only accessible to
  90 the cluster administrators. For exposing metadata queries to the end user itself
  91 this poses a problem since we'd want the user to only query their metadata and
  92 not of any other users, this would require the ElasticSearch cluster to
  93 authenticate users in a way similar to RGW does which poses a problem.
  94
  95 As of Luminous RGW in the metadata master zone can now service end user
  96 requests. This allows for not exposing the elasticsearch endpoint in public and
  97 also solves the authentication and authorization problem since RGW itself can
  98 authenticate the end user requests. For this purpose RGW introduces a new query
  99 in the bucket apis that can service elasticsearch requests. All these requests
 100 must be sent to the metadata master zone.
 101
 102 Syntax
 103 ~~~~~~
 104
 105 Get an elasticsearch query
 106 ``````````````````````````
 107
 108 ::
 109
 110    GET /{bucket}?query={query-expr}
 111
 112 request params:
 113  - max-keys: max number of entries to return
 114  - marker: pagination marker
 115
 116 ``expression := [(]<arg> <op> <value> [)][<and|or> ...]``
 117
 118 op is one of the following:
 119 <, <=, ==, >=, >
 120
 121 For example ::
 122
 123   GET /?query=name==foo
 124
 125 Will return all the indexed keys that user has read permission to, and
 126 are named 'foo'.
 127
 128 Will return all the indexed keys that user has read permission to, and
 129 are named 'foo'.
 130
 131 The output will be a list of keys in XML that is similar to the S3
 132 list buckets response.
 133
 134 Configure custom metadata fields
 135 ````````````````````````````````
 136
 137 Define which custom metadata entries should be indexed (under the
 138 specified bucket), and what are the types of these keys. If explicit
 139 custom metadata indexing is configured, this is needed so that rgw
 140 will index the specified custom metadata values. Otherwise it is
 141 needed in cases where the indexed metadata keys are of a type other
 142 than string.
 143
 144 ::
 145
 146    POST /{bucket}?mdsearch
 147    x-amz-meta-search: <key [; type]> [, ...]
 148
 149 Multiple metadata fields must be comma separated, a type can be forced for a
 150 field with a `;`. The currently allowed types are string(default), integer and
 151 date
 152
 153 eg. if you want to index a custom object metadata x-amz-meta-year as int,
 154 x-amz-meta-date as type date and x-amz-meta-title as string, you'd do
 155
 156 ::
 157
 158    POST /mybooks?mdsearch
 159    x-amz-meta-search: x-amz-meta-year;int, x-amz-meta-release-date;date, x-amz-meta-title;string
 160
 161
 162 Delete custom metadata configuration
 163 ````````````````````````````````````
 164
 165 Delete custom metadata bucket configuration.
 166
 167 ::
 168
 169    DELETE /<bucket>?mdsearch
 170
 171 Get custom metadata configuration
 172 `````````````````````````````````
 173
 174 Retrieve custom metadata bucket configuration.
 175
 176 ::
 177
 178    GET /<bucket>?mdsearch
 179
 180
 181 .. _`Elasticsearch`: https://github.com/elastic/elasticsearch