1 ===========================
2 Install Ceph Object Gateway
3 ===========================
5 As of `firefly` (v0.80), Ceph Object Gateway is running on Civetweb (embedded
6 into the ``ceph-radosgw`` daemon) instead of Apache and FastCGI. Using Civetweb
7 simplifies the Ceph Object Gateway installation and configuration.
9 .. note:: To run the Ceph Object Gateway service, you should have a running
10 Ceph storage cluster, and the gateway host should have access to the
13 .. note:: In version 0.80, the Ceph Object Gateway does not support SSL. You
14 may setup a reverse proxy server with SSL to dispatch HTTPS requests
15 as HTTP requests to CivetWeb.
17 Execute the Pre-Installation Procedure
18 --------------------------------------
20 See Preflight_ and execute the pre-installation procedures on your Ceph Object
21 Gateway node. Specifically, you should disable ``requiretty`` on your Ceph
22 Deploy user, set SELinux to ``Permissive`` and set up a Ceph Deploy user with
23 password-less ``sudo``. For Ceph Object Gateways, you will need to open the
24 port that Civetweb will use in production.
26 .. note:: Civetweb runs on port ``7480`` by default.
28 Install Ceph Object Gateway
29 ---------------------------
31 From the working directory of your administration server, install the Ceph
32 Object Gateway package on the Ceph Object Gateway node. For example::
34 ceph-deploy install --rgw <gateway-node1> [<gateway-node2> ...]
36 The ``ceph-common`` package is a dependency, so ``ceph-deploy`` will install
37 this too. The ``ceph`` CLI tools are intended for administrators. To make your
38 Ceph Object Gateway node an administrator node, execute the following from the
39 working directory of your administration server::
41 ceph-deploy admin <node-name>
43 Create a Gateway Instance
44 -------------------------
46 From the working directory of your administration server, create an instance of
47 the Ceph Object Gateway on the Ceph Object Gateway. For example::
49 ceph-deploy rgw create <gateway-node1>
51 Once the gateway is running, you should be able to access it on port ``7480``
52 with an unauthenticated request like this::
54 http://client-node:7480
56 If the gateway instance is working properly, you should receive a response like
59 <?xml version="1.0" encoding="UTF-8"?>
60 <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
63 <DisplayName></DisplayName>
67 </ListAllMyBucketsResult>
69 If at any point you run into trouble and you want to start over, execute the
70 following to purge the configuration::
72 ceph-deploy purge <gateway-node1> [<gateway-node2>]
73 ceph-deploy purgedata <gateway-node1> [<gateway-node2>]
75 If you execute ``purge``, you must re-install Ceph.
77 Change the Default Port
78 -----------------------
80 Civetweb runs on port ``7480`` by default. To change the default port (e.g., to
81 port ``80``), modify your Ceph configuration file in the working directory of
82 your administration server. Add a section entitled
83 ``[client.rgw.<gateway-node>]``, replacing ``<gateway-node>`` with the short
84 node name of your Ceph Object Gateway node (i.e., ``hostname -s``).
86 .. note:: As of version 11.0.1, the Ceph Object Gateway **does** support SSL.
87 See `Using SSL with Civetweb`_ for information on how to set that up.
89 For example, if your node name is ``gateway-node1``, add a section like this
90 after the ``[global]`` section::
92 [client.rgw.gateway-node1]
93 rgw_frontends = "civetweb port=80"
95 .. note:: Ensure that you leave no whitespace between ``port=<port-number>`` in
96 the ``rgw_frontends`` key/value pair. The ``[client.rgw.gateway-node1]``
97 heading identifies this portion of the Ceph configuration file as
98 configuring a Ceph Storage Cluster client where the client type is a Ceph
99 Object Gateway (i.e., ``rgw``), and the name of the instance is
102 Push the updated configuration file to your Ceph Object Gateway node
103 (and other Ceph nodes)::
105 ceph-deploy --overwrite-conf config push <gateway-node> [<other-nodes>]
107 To make the new port setting take effect, restart the Ceph Object
110 sudo systemctl restart ceph-radosgw.service
112 Finally, check to ensure that the port you selected is open on the node's
113 firewall (e.g., port ``80``). If it is not open, add the port and reload the
114 firewall configuration. If you use the ``firewalld`` daemon, execute::
116 sudo firewall-cmd --list-all
117 sudo firewall-cmd --zone=public --add-port 80/tcp --permanent
118 sudo firewall-cmd --reload
120 If you use ``iptables``, execute::
123 sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
125 Replace ``<iface>`` and ``<ip-address>/<netmask>`` with the relevant values for
126 your Ceph Object Gateway node.
128 Once you have finished configuring ``iptables``, ensure that you make the
129 change persistent so that it will be in effect when your Ceph Object Gateway
130 node reboots. Execute::
132 sudo apt-get install iptables-persistent
134 A terminal UI will open up. Select ``yes`` for the prompts to save current
135 ``IPv4`` iptables rules to ``/etc/iptables/rules.v4`` and current ``IPv6``
136 iptables rules to ``/etc/iptables/rules.v6``.
138 The ``IPv4`` iptables rule that you set in the earlier step will be loaded in
139 ``/etc/iptables/rules.v4`` and will be persistent across reboots.
141 If you add a new ``IPv4`` iptables rule after installing
142 ``iptables-persistent`` you will have to add it to the rule file. In such case,
143 execute the following as the ``root`` user::
145 iptables-save > /etc/iptables/rules.v4
147 Using SSL with Civetweb
148 -----------------------
149 .. _Using SSL with Civetweb:
151 Before using SSL with civetweb, you will need a certificate that will match
152 the host name that that will be used to access the Ceph Object Gateway.
153 You may wish to obtain one that has `subject alternate name` fields for
154 more flexibility. If you intend to use S3-style subdomains
155 (`Add Wildcard to DNS`_), you will need a `wildcard` certificate.
157 Civetweb requires that the server key, server certificate, and any other
158 CA or intermediate certificates be supplied in one file. Each of these
159 items must be in `pem` form. Because the combined file contains the
160 secret key, it should be protected from unauthorized access.
162 To configure ssl operation, append ``s`` to the port number. For eg::
164 [client.rgw.gateway-node1]
165 rgw_frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/keyandcert.pem
167 .. versionadded :: Luminous
169 Furthermore, civetweb can be made to bind to multiple ports, by separating them
170 with ``+`` in the configuration. This allows for use cases where both ssl and
171 non-ssl connections are hosted by a single rgw instance. For eg::
173 [client.rgw.gateway-node1]
174 rgw_frontends = civetweb port=80+443s ssl_certificate=/etc/ceph/private/keyandcert.pem
176 Additional Civetweb Configuration Options
177 -----------------------------------------
178 Some additional configuration options can be adjusted for the embedded Civetweb web server
179 in the **Ceph Object Gateway** section of the ``ceph.conf`` file.
180 A list of supported options, including an example, can be found in the `HTTP Frontends`_.
182 Migrating from Apache to Civetweb
183 ---------------------------------
185 If you are running the Ceph Object Gateway on Apache and FastCGI with Ceph
186 Storage v0.80 or above, you are already running Civetweb--it starts with the
187 ``ceph-radosgw`` daemon and it's running on port 7480 by default so that it
188 doesn't conflict with your Apache and FastCGI installation and other commonly
189 used web service ports. Migrating to use Civetweb basically involves removing
190 your Apache installation. Then, you must remove Apache and FastCGI settings
191 from your Ceph configuration file and reset ``rgw_frontends`` to Civetweb.
193 Referring back to the description for installing a Ceph Object Gateway with
194 ``ceph-deploy``, notice that the configuration file only has one setting
195 ``rgw_frontends`` (and that's assuming you elected to change the default port).
196 The ``ceph-deploy`` utility generates the data directory and the keyring for
197 you--placing the keyring in ``/var/lib/ceph/radosgw/{rgw-instance}``. The daemon
198 looks in default locations, whereas you may have specified different settings
199 in your Ceph configuration file. Since you already have keys and a data
200 directory, you will want to maintain those paths in your Ceph configuration
201 file if you used something other than default paths.
203 A typical Ceph Object Gateway configuration file for an Apache-based deployment
204 looks something similar as the following:
206 On Red Hat Enterprise Linux::
208 [client.radosgw.gateway-node1]
210 keyring = /etc/ceph/ceph.client.radosgw.keyring
212 log file = /var/log/radosgw/client.radosgw.gateway-node1.log
213 rgw frontends = fastcgi socket\_port=9000 socket\_host=0.0.0.0
214 rgw print continue = false
218 [client.radosgw.gateway-node]
220 keyring = /etc/ceph/ceph.client.radosgw.keyring
221 rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
222 log file = /var/log/radosgw/client.radosgw.gateway-node1.log
224 To modify it for use with Civetweb, simply remove the Apache-specific settings
225 such as ``rgw_socket_path`` and ``rgw_print_continue``. Then, change the
226 ``rgw_frontends`` setting to reflect Civetweb rather than the Apache FastCGI
227 front end and specify the port number you intend to use. For example::
229 [client.radosgw.gateway-node1]
231 keyring = /etc/ceph/ceph.client.radosgw.keyring
232 log file = /var/log/radosgw/client.radosgw.gateway-node1.log
233 rgw_frontends = civetweb port=80
235 Finally, restart the Ceph Object Gateway. On Red Hat Enterprise Linux execute::
237 sudo systemctl restart ceph-radosgw.service
241 sudo service radosgw restart id=rgw.<short-hostname>
243 If you used a port number that is not open, you will also need to open that
244 port on your firewall.
246 Configure Bucket Sharding
247 -------------------------
249 A Ceph Object Gateway stores bucket index data in the ``index_pool``, which
250 defaults to ``.rgw.buckets.index``. Sometimes users like to put many objects
251 (hundreds of thousands to millions of objects) in a single bucket. If you do
252 not use the gateway administration interface to set quotas for the maximum
253 number of objects per bucket, the bucket index can suffer significant
254 performance degradation when users place large numbers of objects into a
257 In Ceph 0.94, you may shard bucket indices to help prevent performance
258 bottlenecks when you allow a high number of objects per bucket. The
259 ``rgw_override_bucket_index_max_shards`` setting allows you to set a maximum
260 number of shards per bucket. The default value is ``0``, which means bucket
261 index sharding is off by default.
263 To turn bucket index sharding on, set ``rgw_override_bucket_index_max_shards``
264 to a value greater than ``0``.
266 For simple configurations, you may add ``rgw_override_bucket_index_max_shards``
267 to your Ceph configuration file. Add it under ``[global]`` to create a
268 system-wide value. You can also set it for each instance in your Ceph
271 Once you have changed your bucket sharding configuration in your Ceph
272 configuration file, restart your gateway. On Red Hat Enterprise Linux execute::
274 sudo systemctl restart ceph-radosgw.service
278 sudo service radosgw restart id=rgw.<short-hostname>
280 For federated configurations, each zone may have a different ``index_pool``
281 setting for failover. To make the value consistent for a zonegroup's zones, you
282 may set ``rgw_override_bucket_index_max_shards`` in a gateway's zonegroup
283 configuration. For example::
285 radosgw-admin zonegroup get > zonegroup.json
287 Open the ``zonegroup.json`` file and edit the ``bucket_index_max_shards`` setting
288 for each named zone. Save the ``zonegroup.json`` file and reset the zonegroup.
291 radosgw-admin zonegroup set < zonegroup.json
293 Once you have updated your zonegroup, update and commit the period.
296 radosgw-admin period update --commit
298 .. note:: Mapping the index pool (for each zone, if applicable) to a CRUSH
299 rule of SSD-based OSDs may also help with bucket index performance.
303 .. _Add Wildcard to DNS:
305 To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you
306 need to add a wildcard to the DNS record of the DNS server you use with the
307 ``ceph-radosgw`` daemon.
309 The address of the DNS must also be specified in the Ceph configuration file
310 with the ``rgw dns name = {hostname}`` setting.
312 For ``dnsmasq``, add the following address setting with a dot (.) prepended to
315 address=/.{hostname-or-fqdn}/{host-ip-address}
319 address=/.gateway-node1/192.168.122.75
322 For ``bind``, add a wildcard to the DNS record. For example::
325 @ IN SOA gateway-node1. root.gateway-node1. (
330 604800 ) ; Negative Cache TTL
332 @ IN NS gateway-node1.
333 @ IN A 192.168.122.113
336 Restart your DNS server and ping your server with a subdomain to ensure that
337 your DNS configuration works as expected::
339 ping mybucket.{hostname}
343 ping mybucket.gateway-node1
345 Add Debugging (if needed)
346 -------------------------
348 Once you finish the setup procedure, if you encounter issues with your
349 configuration, you can add debugging to the ``[global]`` section of your Ceph
350 configuration file and restart the gateway(s) to help troubleshoot any
351 configuration issues. For example::
354 #append the following in the global section.
361 To use the REST interfaces, first create an initial Ceph Object Gateway user
362 for the S3 interface. Then, create a subuser for the Swift interface. You then
363 need to verify if the created users are able to access the gateway.
365 Create a RADOSGW User for S3 Access
366 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
368 A ``radosgw`` user needs to be created and granted access. The command ``man
369 radosgw-admin`` will provide information on additional command options.
371 To create the user, execute the following on the ``gateway host``::
373 sudo radosgw-admin user create --uid="testuser" --display-name="First User"
375 The output of the command will be something like the following::
378 "user_id": "testuser",
379 "display_name": "First User",
386 "access_key": "I0PJDPCIYZ665MW88W9R",
387 "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
391 "op_mask": "read, write, delete",
392 "default_placement": "",
393 "placement_tags": [],
407 .. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are
408 needed for access validation.
410 .. important:: Check the key output. Sometimes ``radosgw-admin`` generates a
411 JSON escape character ``\`` in ``access_key`` or ``secret_key``
412 and some clients do not know how to handle JSON escape
413 characters. Remedies include removing the JSON escape character
414 ``\``, encapsulating the string in quotes, regenerating the key
415 and ensuring that it does not have a JSON escape character or
416 specify the key and secret manually. Also, if ``radosgw-admin``
417 generates a JSON escape character ``\`` and a forward slash ``/``
418 together in a key, like ``\/``, only remove the JSON escape
419 character ``\``. Do not remove the forward slash ``/`` as it is
420 a valid character in the key.
425 A Swift subuser needs to be created if this kind of access is needed. Creating
426 a Swift user is a two step process. The first step is to create the user. The
427 second is to create the secret key.
429 Execute the following steps on the ``gateway host``:
431 Create the Swift user::
433 sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
435 The output will be something like the following::
438 "user_id": "testuser",
439 "display_name": "First User",
444 "id": "testuser:swift",
445 "permissions": "full-control"
448 "user": "testuser:swift",
449 "access_key": "3Y1LNW4Q6X0Y53A52DET",
453 "access_key": "I0PJDPCIYZ665MW88W9R",
454 "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
458 "op_mask": "read, write, delete",
459 "default_placement": "",
460 "placement_tags": [],
474 Create the secret key::
476 sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
478 The output will be something like the following::
481 "user_id": "testuser",
482 "display_name": "First User",
487 "id": "testuser:swift",
488 "permissions": "full-control"
491 "user": "testuser:swift",
492 "access_key": "3Y1LNW4Q6X0Y53A52DET",
496 "access_key": "I0PJDPCIYZ665MW88W9R",
497 "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
500 "user": "testuser:swift",
501 "secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA"
504 "op_mask": "read, write, delete",
505 "default_placement": "",
506 "placement_tags": [],
526 You need to write and run a Python test script for verifying S3 access. The S3
527 access test script will connect to the ``radosgw``, create a new bucket and
528 list all buckets. The values for ``aws_access_key_id`` and
529 ``aws_secret_access_key`` are taken from the values of ``access_key`` and
530 ``secret_key`` returned by the ``radosgw-admin`` command.
532 Execute the following steps:
534 #. You will need to install the ``python-boto`` package::
536 sudo yum install python-boto
538 #. Create the Python script::
542 #. Add the following contents to the file::
544 import boto.s3.connection
546 access_key = 'I0PJDPCIYZ665MW88W9R'
547 secret_key = 'dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA'
548 conn = boto.connect_s3(
549 aws_access_key_id=access_key,
550 aws_secret_access_key=secret_key,
551 host='{hostname}', port={port},
552 is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
555 bucket = conn.create_bucket('my-new-bucket')
556 for bucket in conn.get_all_buckets():
557 print "{name} {created}".format(
559 created=bucket.creation_date,
563 Replace ``{hostname}`` with the hostname of the host where you have
564 configured the gateway service i.e., the ``gateway host``. Replace ``{port}``
565 with the port number you are using with Civetweb.
571 The output will be something like the following::
573 my-new-bucket 2015-02-16T17:09:10.000Z
578 Swift access can be verified via the ``swift`` command line client. The command
579 ``man swift`` will provide more information on available command line options.
581 To install ``swift`` client, execute the following commands. On Red Hat
584 sudo yum install python-setuptools
585 sudo easy_install pip
586 sudo pip install --upgrade setuptools
587 sudo pip install --upgrade python-swiftclient
589 On Debian-based distributions::
591 sudo apt-get install python-setuptools
592 sudo easy_install pip
593 sudo pip install --upgrade setuptools
594 sudo pip install --upgrade python-swiftclient
596 To test swift access, execute the following::
598 swift -V 1 -A http://{IP ADDRESS}:{port}/auth -U testuser:swift -K '{swift_secret_key}' list
600 Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and
601 ``{swift_secret_key}`` with its value from the output of ``radosgw-admin key
602 create`` command executed for the ``swift`` user. Replace {port} with the port
603 number you are using with Civetweb (e.g., ``7480`` is the default). If you
604 don't replace the port, it will default to port ``80``.
608 swift -V 1 -A http://10.19.143.116:7480/auth -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list
610 The output should be::
614 .. _Preflight: ../../start/quick-start-preflight
615 .. _HTTP Frontends: ../../radosgw/frontends