1 ===========================
2 Install Ceph Object Gateway
3 ===========================
5 As of `firefly` (v0.80), Ceph Object Gateway is running on Civetweb (embedded
6 into the ``ceph-radosgw`` daemon) instead of Apache and FastCGI. Using Civetweb
7 simplifies the Ceph Object Gateway installation and configuration.
9 .. note:: To run the Ceph Object Gateway service, you should have a running
10 Ceph storage cluster, and the gateway host should have access to the
13 .. note:: In version 0.80, the Ceph Object Gateway does not support SSL. You
14 may setup a reverse proxy server with SSL to dispatch HTTPS requests
15 as HTTP requests to CivetWeb.
17 Execute the Pre-Installation Procedure
18 --------------------------------------
20 See Preflight_ and execute the pre-installation procedures on your Ceph Object
21 Gateway node. Specifically, you should disable ``requiretty`` on your Ceph
22 Deploy user, set SELinux to ``Permissive`` and set up a Ceph Deploy user with
23 password-less ``sudo``. For Ceph Object Gateways, you will need to open the
24 port that Civetweb will use in production.
26 .. note:: Civetweb runs on port ``7480`` by default.
28 Install Ceph Object Gateway
29 ---------------------------
31 From the working directory of your administration server, install the Ceph
32 Object Gateway package on the Ceph Object Gateway node. For example::
34 ceph-deploy install --rgw <gateway-node1> [<gateway-node2> ...]
36 The ``ceph-common`` package is a dependency, so ``ceph-deploy`` will install
37 this too. The ``ceph`` CLI tools are intended for administrators. To make your
38 Ceph Object Gateway node an administrator node, execute the following from the
39 working directory of your administration server::
41 ceph-deploy admin <node-name>
43 Create a Gateway Instance
44 -------------------------
46 From the working directory of your administration server, create an instance of
47 the Ceph Object Gateway on the Ceph Object Gateway. For example::
49 ceph-deploy rgw create <gateway-node1>
51 Once the gateway is running, you should be able to access it on port ``7480``
52 with an unauthenticated request like this::
54 http://client-node:7480
56 If the gateway instance is working properly, you should receive a response like
59 <?xml version="1.0" encoding="UTF-8"?>
60 <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
63 <DisplayName></DisplayName>
67 </ListAllMyBucketsResult>
69 If at any point you run into trouble and you want to start over, execute the
70 following to purge the configuration::
72 ceph-deploy purge <gateway-node1> [<gateway-node2>]
73 ceph-deploy purgedata <gateway-node1> [<gateway-node2>]
75 If you execute ``purge``, you must re-install Ceph.
77 Change the Default Port
78 -----------------------
80 Civetweb runs on port ``7480`` by default. To change the default port (e.g., to
81 port ``80``), modify your Ceph configuration file in the working directory of
82 your administration server. Add a section entitled
83 ``[client.rgw.<gateway-node>]``, replacing ``<gateway-node>`` with the short
84 node name of your Ceph Object Gateway node (i.e., ``hostname -s``).
86 .. note:: As of version 11.0.1, the Ceph Object Gateway **does** support SSL.
87 See `Using SSL with Civetweb`_ for information on how to set that up.
89 For example, if your node name is ``gateway-node1``, add a section like this
90 after the ``[global]`` section::
92 [client.rgw.gateway-node1]
93 rgw_frontends = "civetweb port=80"
95 .. note:: Ensure that you leave no whitespace between ``port=<port-number>`` in
96 the ``rgw_frontends`` key/value pair. The ``[client.rgw.gateway-node1]``
97 heading identifies this portion of the Ceph configuration file as
98 configuring a Ceph Storage Cluster client where the client type is a Ceph
99 Object Gateway (i.e., ``rgw``), and the name of the instance is
102 Push the updated configuration file to your Ceph Object Gateway node
103 (and other Ceph nodes)::
105 ceph-deploy --overwrite-conf config push <gateway-node> [<other-nodes>]
107 To make the new port setting take effect, restart the Ceph Object
110 sudo systemctl restart ceph-radosgw.service
112 Finally, check to ensure that the port you selected is open on the node's
113 firewall (e.g., port ``80``). If it is not open, add the port and reload the
114 firewall configuration. If you use the ``firewalld`` daemon, execute::
116 sudo firewall-cmd --list-all
117 sudo firewall-cmd --zone=public --add-port 80/tcp --permanent
118 sudo firewall-cmd --reload
120 If you use ``iptables``, execute::
123 sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
125 Replace ``<iface>`` and ``<ip-address>/<netmask>`` with the relevant values for
126 your Ceph Object Gateway node.
128 Once you have finished configuring ``iptables``, ensure that you make the
129 change persistent so that it will be in effect when your Ceph Object Gateway
130 node reboots. Execute::
132 sudo apt-get install iptables-persistent
134 A terminal UI will open up. Select ``yes`` for the prompts to save current
135 ``IPv4`` iptables rules to ``/etc/iptables/rules.v4`` and current ``IPv6``
136 iptables rules to ``/etc/iptables/rules.v6``.
138 The ``IPv4`` iptables rule that you set in the earlier step will be loaded in
139 ``/etc/iptables/rules.v4`` and will be persistent across reboots.
141 If you add a new ``IPv4`` iptables rule after installing
142 ``iptables-persistent`` you will have to add it to the rule file. In such case,
143 execute the following as the ``root`` user::
145 iptables-save > /etc/iptables/rules.v4
147 Using SSL with Civetweb
148 -----------------------
149 .. _Using SSL with Civetweb:
151 Before using SSL with civetweb, you will need a certificate that will match
152 the host name that that will be used to access the Ceph Object Gateway.
153 You may wish to obtain one that has `subject alternate name` fields for
154 more flexibility. If you intend to use S3-style subdomains
155 (`Add Wildcard to DNS`_), you will need a `wildcard` certificate.
157 Civetweb requires that the server key, server certificate, and any other
158 CA or intermediate certificates be supplied in one file. Each of these
159 items must be in `pem` form. Because the combined file contains the
160 secret key, it should be protected from unauthorized access.
162 To configure ssl operation, append ``s`` to the port number. Currently
163 it is not possible to configure the radosgw to listen on both
164 http and https, you must pick only one. So::
166 [client.rgw.gateway-node1]
167 rgw_frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/keyandcert.pem
169 Migrating from Apache to Civetweb
170 ---------------------------------
172 If you're running the Ceph Object Gateway on Apache and FastCGI with Ceph
173 Storage v0.80 or above, you're already running Civetweb--it starts with the
174 ``ceph-radosgw`` daemon and it's running on port 7480 by default so that it
175 doesn't conflict with your Apache and FastCGI installation and other commonly
176 used web service ports. Migrating to use Civetweb basically involves removing
177 your Apache installation. Then, you must remove Apache and FastCGI settings
178 from your Ceph configuration file and reset ``rgw_frontends`` to Civetweb.
180 Referring back to the description for installing a Ceph Object Gateway with
181 ``ceph-deploy``, notice that the configuration file only has one setting
182 ``rgw_frontends`` (and that's assuming you elected to change the default port).
183 The ``ceph-deploy`` utility generates the data directory and the keyring for
184 you--placing the keyring in ``/var/lib/ceph/radosgw/{rgw-intance}``. The daemon
185 looks in default locations, whereas you may have specified different settings
186 in your Ceph configuration file. Since you already have keys and a data
187 directory, you will want to maintain those paths in your Ceph configuration
188 file if you used something other than default paths.
190 A typical Ceph Object Gateway configuration file for an Apache-based deployment
191 looks something similar as the following:
193 On Red Hat Enterprise Linux::
195 [client.radosgw.gateway-node1]
197 keyring = /etc/ceph/ceph.client.radosgw.keyring
199 log file = /var/log/radosgw/client.radosgw.gateway-node1.log
200 rgw frontends = fastcgi socket\_port=9000 socket\_host=0.0.0.0
201 rgw print continue = false
205 [client.radosgw.gateway-node]
207 keyring = /etc/ceph/ceph.client.radosgw.keyring
208 rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
209 log file = /var/log/radosgw/client.radosgw.gateway-node1.log
211 To modify it for use with Civetweb, simply remove the Apache-specific settings
212 such as ``rgw_socket_path`` and ``rgw_print_continue``. Then, change the
213 ``rgw_frontends`` setting to reflect Civetweb rather than the Apache FastCGI
214 front end and specify the port number you intend to use. For example::
216 [client.radosgw.gateway-node1]
218 keyring = /etc/ceph/ceph.client.radosgw.keyring
219 log file = /var/log/radosgw/client.radosgw.gateway-node1.log
220 rgw_frontends = civetweb port=80
222 Finally, restart the Ceph Object Gateway. On Red Hat Enterprise Linux execute::
224 sudo systemctl restart ceph-radosgw.service
228 sudo service radosgw restart id=rgw.<short-hostname>
230 If you used a port number that is not open, you will also need to open that
231 port on your firewall.
233 Configure Bucket Sharding
234 -------------------------
236 A Ceph Object Gateway stores bucket index data in the ``index_pool``, which
237 defaults to ``.rgw.buckets.index``. Sometimes users like to put many objects
238 (hundreds of thousands to millions of objects) in a single bucket. If you do
239 not use the gateway administration interface to set quotas for the maximum
240 number of objects per bucket, the bucket index can suffer significant
241 performance degradation when users place large numbers of objects into a
244 In Ceph 0.94, you may shard bucket indices to help prevent performance
245 bottlenecks when you allow a high number of objects per bucket. The
246 ``rgw_override_bucket_index_max_shards`` setting allows you to set a maximum
247 number of shards per bucket. The default value is ``0``, which means bucket
248 index sharding is off by default.
250 To turn bucket index sharding on, set ``rgw_override_bucket_index_max_shards``
251 to a value greater than ``0``.
253 For simple configurations, you may add ``rgw_override_bucket_index_max_shards``
254 to your Ceph configuration file. Add it under ``[global]`` to create a
255 system-wide value. You can also set it for each instance in your Ceph
258 Once you have changed your bucket sharding configuration in your Ceph
259 configuration file, restart your gateway. On Red Hat Enteprise Linux execute::
261 sudo systemctl restart ceph-radosgw.service
265 sudo service radosgw restart id=rgw.<short-hostname>
267 For federated configurations, each zone may have a different ``index_pool``
268 setting for failover. To make the value consistent for a region's zones, you
269 may set ``rgw_override_bucket_index_max_shards`` in a gateway's region
270 configuration. For example::
272 radosgw-admin region get > region.json
274 Open the ``region.json`` file and edit the ``bucket_index_max_shards`` setting
275 for each named zone. Save the ``region.json`` file and reset the region. For
278 radosgw-admin region set < region.json
280 Once you've updated your region, update the region map. For example::
282 radosgw-admin regionmap update --name client.rgw.ceph-client
284 Where ``client.rgw.ceph-client`` is the name of the gateway user.
286 .. note:: Mapping the index pool (for each zone, if applicable) to a CRUSH
287 ruleset of SSD-based OSDs may also help with bucket index performance.
291 .. _Add Wildcard to DNS:
293 To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you
294 need to add a wildcard to the DNS record of the DNS server you use with the
295 ``ceph-radosgw`` daemon.
297 The address of the DNS must also be specified in the Ceph configuration file
298 with the ``rgw dns name = {hostname}`` setting.
300 For ``dnsmasq``, add the following address setting with a dot (.) prepended to
303 address=/.{hostname-or-fqdn}/{host-ip-address}
307 address=/.gateway-node1/192.168.122.75
310 For ``bind``, add a wildcard to the DNS record. For example::
313 @ IN SOA gateway-node1. root.gateway-node1. (
318 604800 ) ; Negative Cache TTL
320 @ IN NS gateway-node1.
321 @ IN A 192.168.122.113
324 Restart your DNS server and ping your server with a subdomain to ensure that
325 your DNS configuration works as expected::
327 ping mybucket.{hostname}
331 ping mybucket.gateway-node1
333 Add Debugging (if needed)
334 -------------------------
336 Once you finish the setup procedure, if you encounter issues with your
337 configuration, you can add debugging to the ``[global]`` section of your Ceph
338 configuration file and restart the gateway(s) to help troubleshoot any
339 configuration issues. For example::
342 #append the following in the global section.
349 To use the REST interfaces, first create an initial Ceph Object Gateway user
350 for the S3 interface. Then, create a subuser for the Swift interface. You then
351 need to verify if the created users are able to access the gateway.
353 Create a RADOSGW User for S3 Access
354 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
356 A ``radosgw`` user needs to be created and granted access. The command ``man
357 radosgw-admin`` will provide information on additional command options.
359 To create the user, execute the following on the ``gateway host``::
361 sudo radosgw-admin user create --uid="testuser" --display-name="First User"
363 The output of the command will be something like the following::
366 "user_id": "testuser",
367 "display_name": "First User",
375 "access_key": "I0PJDPCIYZ665MW88W9R",
376 "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
380 "op_mask": "read, write, delete",
381 "default_placement": "",
382 "placement_tags": [],
396 .. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are
397 needed for access validation.
399 .. important:: Check the key output. Sometimes ``radosgw-admin`` generates a
400 JSON escape character ``\`` in ``access_key`` or ``secret_key``
401 and some clients do not know how to handle JSON escape
402 characters. Remedies include removing the JSON escape character
403 ``\``, encapsulating the string in quotes, regenerating the key
404 and ensuring that it does not have a JSON escape character or
405 specify the key and secret manually. Also, if ``radosgw-admin``
406 generates a JSON escape character ``\`` and a forward slash ``/``
407 together in a key, like ``\/``, only remove the JSON escape
408 character ``\``. Do not remove the forward slash ``/`` as it is
409 a valid character in the key.
414 A Swift subuser needs to be created if this kind of access is needed. Creating
415 a Swift user is a two step process. The first step is to create the user. The
416 second is to create the secret key.
418 Execute the following steps on the ``gateway host``:
420 Create the Swift user::
422 sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
424 The output will be something like the following::
427 "user_id": "testuser",
428 "display_name": "First User",
434 "id": "testuser:swift",
435 "permissions": "full-control"
438 "user": "testuser:swift",
439 "access_key": "3Y1LNW4Q6X0Y53A52DET",
443 "access_key": "I0PJDPCIYZ665MW88W9R",
444 "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
448 "op_mask": "read, write, delete",
449 "default_placement": "",
450 "placement_tags": [],
464 Create the secret key::
466 sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
468 The output will be something like the following::
471 "user_id": "testuser",
472 "display_name": "First User",
478 "id": "testuser:swift",
479 "permissions": "full-control"
482 "user": "testuser:swift",
483 "access_key": "3Y1LNW4Q6X0Y53A52DET",
487 "access_key": "I0PJDPCIYZ665MW88W9R",
488 "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
491 "user": "testuser:swift",
492 "secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA"
495 "op_mask": "read, write, delete",
496 "default_placement": "",
497 "placement_tags": [],
517 You need to write and run a Python test script for verifying S3 access. The S3
518 access test script will connect to the ``radosgw``, create a new bucket and
519 list all buckets. The values for ``aws_access_key_id`` and
520 ``aws_secret_access_key`` are taken from the values of ``access_key`` and
521 ``secret_key`` returned by the ``radosgw-admin`` command.
523 Execute the following steps:
525 #. You will need to install the ``python-boto`` package::
527 sudo yum install python-boto
529 #. Create the Python script::
533 #. Add the following contents to the file::
535 import boto.s3.connection
537 access_key = 'I0PJDPCIYZ665MW88W9R'
538 secret_key = 'dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA'
539 conn = boto.connect_s3(
540 aws_access_key_id=access_key,
541 aws_secret_access_key=secret_key,
542 host='{hostname}', port={port},
543 is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
546 bucket = conn.create_bucket('my-new-bucket')
547 for bucket in conn.get_all_buckets():
548 print "{name} {created}".format(
550 created=bucket.creation_date,
554 Replace ``{hostname}`` with the hostname of the host where you have
555 configured the gateway service i.e., the ``gateway host``. Replace ``{port}``
556 with the port number you are using with Civetweb.
562 The output will be something like the following::
564 my-new-bucket 2015-02-16T17:09:10.000Z
569 Swift access can be verified via the ``swift`` command line client. The command
570 ``man swift`` will provide more information on available command line options.
572 To install ``swift`` client, execute the following commands. On Red Hat
575 sudo yum install python-setuptools
576 sudo easy_install pip
577 sudo pip install --upgrade setuptools
578 sudo pip install --upgrade python-swiftclient
580 On Debian-based distributions::
582 sudo apt-get install python-setuptools
583 sudo easy_install pip
584 sudo pip install --upgrade setuptools
585 sudo pip install --upgrade python-swiftclient
587 To test swift access, execute the following::
589 swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list
591 Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and
592 ``{swift_secret_key}`` with its value from the output of ``radosgw-admin key
593 create`` command executed for the ``swift`` user. Replace {port} with the port
594 number you are using with Civetweb (e.g., ``7480`` is the default). If you
595 don't replace the port, it will default to port ``80``.
599 swift -A http://10.19.143.116:7480/auth/1.0 -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list
601 The output should be::
605 .. _Preflight: ../../start/quick-start-preflight