6 The Gateway Won't Start
7 =======================
9 If you cannot start the gateway (i.e., there is no existing ``pid``),
10 check to see if there is an existing ``.asok`` file from another
11 user. If an ``.asok`` file from another user exists and there is no
12 running ``pid``, remove the ``.asok`` file and try to start the
13 process again. This may occur when you start the process as a ``root`` user and
14 the startup script is trying to start the process as a
15 ``www-data`` or ``apache`` user and an existing ``.asok`` is
16 preventing the script from starting the daemon.
18 The radosgw init script (/etc/init.d/radosgw) also has a verbose argument that
19 can provide some insight as to what could be the issue::
21 /etc/init.d/radosgw start -v
25 /etc/init.d radosgw start --verbose
30 Examining the access and error logs for the web server itself is
31 probably the first step in identifying what is going on. If there is
32 a 500 error, that usually indicates a problem communicating with the
33 ``radosgw`` daemon. Ensure the daemon is running, its socket path is
34 configured, and that the web server is looking for it in the proper
38 Crashed ``radosgw`` process
39 ===========================
41 If the ``radosgw`` process dies, you will normally see a 500 error
42 from the web server (apache, nginx, etc.). In that situation, simply
43 restarting radosgw will restore service.
45 To diagnose the cause of the crash, check the log in ``/var/log/ceph``
46 and/or the core file (if one was generated).
49 Blocked ``radosgw`` Requests
50 ============================
52 If some (or all) radosgw requests appear to be blocked, you can get
53 some insight into the internal state of the ``radosgw`` daemon via
54 its admin socket. By default, there will be a socket configured to
55 reside in ``/var/run/ceph``, and the daemon can be queried with::
57 ceph daemon /var/run/ceph/client.rgw help
59 help list available commands
60 objecter_requests show in-progress osd requests
61 perfcounters_dump dump perfcounters value
62 perfcounters_schema dump perfcounters schema
63 version get protocol version
65 Of particular interest::
67 ceph daemon /var/run/ceph/client.rgw objecter_requests
70 will dump information about current in-progress requests with the
71 RADOS cluster. This allows one to identify if any requests are blocked
72 by a non-responsive OSD. For example, one might see::
78 "last_sent": "2012-03-08 14:56:37.949872",
80 "object_id": "fatty_25647_object1857",
81 "object_locator": "@2",
83 "snap_context": "0=[]",
84 "mtime": "2012-03-08 14:56:37.949813",
90 "last_sent": "2012-03-08 14:56:37.970615",
92 "object_id": "fatty_25647_object1872",
93 "object_locator": "@2",
95 "snap_context": "0=[]",
96 "mtime": "2012-03-08 14:56:37.970555",
104 In this dump, two requests are in progress. The ``last_sent`` field is
105 the time the RADOS request was sent. If this is a while ago, it suggests
106 that the OSD is not responding. For example, for request 1858, you could
107 check the OSD status with::
109 ceph pg map 2.d2041a48
111 osdmap e9 pg 2.d2041a48 (2.0) -> up [1,0] acting [1,0]
113 This tells us to look at ``osd.1``, the primary copy for this PG::
115 ceph daemon osd.1 ops
118 { "description": "osd_op(client.4124.0:1858 fatty_25647_object1857 [write 0~4096] 2.d2041a48)",
119 "received_at": "1331247573.344650",
121 "flag_point": "waiting for sub ops",
122 "client_info": { "client": "client.4124",
126 The ``flag_point`` field indicates that the OSD is currently waiting
127 for replicas to respond, in this case ``osd.0``.
130 Java S3 API Troubleshooting
131 ===========================
134 Peer Not Authenticated
135 ----------------------
137 You may receive an error that looks like this::
139 [java] INFO: Unable to execute HTTP request: peer not authenticated
141 The Java SDK for S3 requires a valid certificate from a recognized certificate
142 authority, because it uses HTTPS by default. If you are just testing the Ceph
143 Object Storage services, you can resolve this problem in a few ways:
145 #. Prepend the IP address or hostname with ``http://``. For example, change this::
147 conn.setEndpoint("myserver");
151 conn.setEndpoint("http://myserver")
153 #. After setting your credentials, add a client configuration and set the
154 protocol to ``Protocol.HTTP``. ::
156 AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
158 ClientConfiguration clientConfig = new ClientConfiguration();
159 clientConfig.setProtocol(Protocol.HTTP);
161 AmazonS3 conn = new AmazonS3Client(credentials, clientConfig);
168 If you receive an 405 error, check to see if you have the S3 subdomain set up correctly.
169 You will need to have a wild card setting in your DNS record for subdomain functionality
172 Also, check to ensure that the default site is disabled. ::
174 [java] Exception in thread "main" Status Code: 405, AWS Service: Amazon S3, AWS Request ID: null, AWS Error Code: MethodNotAllowed, AWS Error Message: null, S3 Extended Request ID: null
178 Numerous objects in default.rgw.meta pool
179 =========================================
181 Clusters created prior to *jewel* have a metadata archival feature enabled by default, using the ``default.rgw.meta`` pool.
182 This archive keeps all old versions of user and bucket metadata, resulting in large numbers of objects in the ``default.rgw.meta`` pool.
184 Disabling the Metadata Heap
185 ---------------------------
187 Users who want to disable this feature going forward should set the ``metadata_heap`` field to an empty string ``""``::
189 $ radosgw-admin zone get --rgw-zone=default > zone.json
190 [edit zone.json, setting "metadata_heap": ""]
191 $ radosgw-admin zone set --rgw-zone=default --infile=zone.json
192 $ radosgw-admin period update --commit
194 This will stop new metadata from being written to the ``default.rgw.meta`` pool, but does not remove any existing objects or pool.
196 Cleaning the Metadata Heap Pool
197 -------------------------------
199 Clusters created prior to *jewel* normally use ``default.rgw.meta`` only for the metadata archival feature.
201 However, from *luminous* onwards, radosgw uses :ref:`Pool Namespaces <radosgw-pool-namespaces>` within ``default.rgw.meta`` for an entirely different purpose, that is, to store ``user_keys`` and other critical metadata.
203 Users should check zone configuration before proceeding any cleanup procedures::
205 $ radosgw-admin zone get --rgw-zone=default | grep default.rgw.meta
206 [should not match any strings]
208 Having confirmed that the pool is not used for any purpose, users may safely delete all objects in the ``default.rgw.meta`` pool, or optionally, delete the entire pool itself.