]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | ======================= |
2 | Logging and Debugging | |
3 | ======================= | |
4 | ||
5 | Typically, when you add debugging to your Ceph configuration, you do so at | |
6 | runtime. You can also add Ceph debug logging to your Ceph configuration file if | |
7 | you are encountering issues when starting your cluster. You may view Ceph log | |
8 | files under ``/var/log/ceph`` (the default location). | |
9 | ||
10 | .. tip:: When debug output slows down your system, the latency can hide | |
11 | race conditions. | |
12 | ||
13 | Logging is resource intensive. If you are encountering a problem in a specific | |
14 | area of your cluster, enable logging for that area of the cluster. For example, | |
15 | if your OSDs are running fine, but your metadata servers are not, you should | |
16 | start by enabling debug logging for the specific metadata server instance(s) | |
17 | giving you trouble. Enable logging for each subsystem as needed. | |
18 | ||
19 | .. important:: Verbose logging can generate over 1GB of data per hour. If your | |
20 | OS disk reaches its capacity, the node will stop working. | |
21 | ||
22 | If you enable or increase the rate of Ceph logging, ensure that you have | |
23 | sufficient disk space on your OS disk. See `Accelerating Log Rotation`_ for | |
24 | details on rotating log files. When your system is running well, remove | |
25 | unnecessary debugging settings to ensure your cluster runs optimally. Logging | |
26 | debug output messages is relatively slow, and a waste of resources when | |
27 | operating your cluster. | |
28 | ||
29 | See `Subsystem, Log and Debug Settings`_ for details on available settings. | |
30 | ||
31 | Runtime | |
32 | ======= | |
33 | ||
34 | If you would like to see the configuration settings at runtime, you must log | |
35 | in to a host with a running daemon and execute the following:: | |
36 | ||
37 | ceph daemon {daemon-name} config show | less | |
38 | ||
39 | For example,:: | |
40 | ||
41 | ceph daemon osd.0 config show | less | |
42 | ||
43 | To activate Ceph's debugging output (*i.e.*, ``dout()``) at runtime, use the | |
44 | ``ceph tell`` command to inject arguments into the runtime configuration:: | |
45 | ||
46 | ceph tell {daemon-type}.{daemon id or *} injectargs --{name} {value} [--{name} {value}] | |
47 | ||
48 | Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply | |
49 | the runtime setting to all daemons of a particular type with ``*``, or specify | |
50 | a specific daemon's ID. For example, to increase | |
51 | debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following:: | |
52 | ||
53 | ceph tell osd.0 injectargs --debug-osd 0/5 | |
54 | ||
55 | The ``ceph tell`` command goes through the monitors. If you cannot bind to the | |
56 | monitor, you can still make the change by logging into the host of the daemon | |
57 | whose configuration you'd like to change using ``ceph daemon``. | |
58 | For example:: | |
59 | ||
60 | sudo ceph daemon osd.0 config set debug_osd 0/5 | |
61 | ||
62 | See `Subsystem, Log and Debug Settings`_ for details on available settings. | |
63 | ||
64 | ||
65 | Boot Time | |
66 | ========= | |
67 | ||
68 | To activate Ceph's debugging output (*i.e.*, ``dout()``) at boot time, you must | |
69 | add settings to your Ceph configuration file. Subsystems common to each daemon | |
70 | may be set under ``[global]`` in your configuration file. Subsystems for | |
71 | particular daemons are set under the daemon section in your configuration file | |
72 | (*e.g.*, ``[mon]``, ``[osd]``, ``[mds]``). For example:: | |
73 | ||
74 | [global] | |
75 | debug ms = 1/5 | |
76 | ||
77 | [mon] | |
78 | debug mon = 20 | |
79 | debug paxos = 1/5 | |
80 | debug auth = 2 | |
81 | ||
82 | [osd] | |
83 | debug osd = 1/5 | |
84 | debug filestore = 1/5 | |
85 | debug journal = 1 | |
86 | debug monc = 5/20 | |
87 | ||
88 | [mds] | |
89 | debug mds = 1 | |
90 | debug mds balancer = 1 | |
91 | ||
92 | ||
93 | See `Subsystem, Log and Debug Settings`_ for details. | |
94 | ||
95 | ||
96 | Accelerating Log Rotation | |
97 | ========================= | |
98 | ||
99 | If your OS disk is relatively full, you can accelerate log rotation by modifying | |
100 | the Ceph log rotation file at ``/etc/logrotate.d/ceph``. Add a size setting | |
101 | after the rotation frequency to accelerate log rotation (via cronjob) if your | |
102 | logs exceed the size setting. For example, the default setting looks like | |
103 | this:: | |
104 | ||
105 | rotate 7 | |
106 | weekly | |
107 | compress | |
108 | sharedscripts | |
109 | ||
110 | Modify it by adding a ``size`` setting. :: | |
111 | ||
112 | rotate 7 | |
113 | weekly | |
114 | size 500M | |
115 | compress | |
116 | sharedscripts | |
117 | ||
118 | Then, start the crontab editor for your user space. :: | |
119 | ||
120 | crontab -e | |
121 | ||
122 | Finally, add an entry to check the ``etc/logrotate.d/ceph`` file. :: | |
123 | ||
124 | 30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1 | |
125 | ||
126 | The preceding example checks the ``etc/logrotate.d/ceph`` file every 30 minutes. | |
127 | ||
128 | ||
129 | Valgrind | |
130 | ======== | |
131 | ||
132 | Debugging may also require you to track down memory and threading issues. | |
133 | You can run a single daemon, a type of daemon, or the whole cluster with | |
134 | Valgrind. You should only use Valgrind when developing or debugging Ceph. | |
135 | Valgrind is computationally expensive, and will slow down your system otherwise. | |
136 | Valgrind messages are logged to ``stderr``. | |
137 | ||
138 | ||
139 | Subsystem, Log and Debug Settings | |
140 | ================================= | |
141 | ||
142 | In most cases, you will enable debug logging output via subsystems. | |
143 | ||
144 | Ceph Subsystems | |
145 | --------------- | |
146 | ||
147 | Each subsystem has a logging level for its output logs, and for its logs | |
148 | in-memory. You may set different values for each of these subsystems by setting | |
149 | a log file level and a memory level for debug logging. Ceph's logging levels | |
150 | operate on a scale of ``1`` to ``20``, where ``1`` is terse and ``20`` is | |
151 | verbose [#]_ . In general, the logs in-memory are not sent to the output log unless: | |
152 | ||
153 | - a fatal signal is raised or | |
154 | - an ``assert`` in source code is triggered or | |
155 | - upon requested. Please consult `document on admin socket <http://docs.ceph.com/docs/master/man/8/ceph/#daemon>`_ for more details. | |
156 | ||
157 | A debug logging setting can take a single value for the log level and the | |
158 | memory level, which sets them both as the same value. For example, if you | |
159 | specify ``debug ms = 5``, Ceph will treat it as a log level and a memory level | |
160 | of ``5``. You may also specify them separately. The first setting is the log | |
161 | level, and the second setting is the memory level. You must separate them with | |
162 | a forward slash (/). For example, if you want to set the ``ms`` subsystem's | |
163 | debug logging level to ``1`` and its memory level to ``5``, you would specify it | |
164 | as ``debug ms = 1/5``. For example: | |
165 | ||
166 | ||
167 | ||
168 | .. code-block:: ini | |
169 | ||
170 | debug {subsystem} = {log-level}/{memory-level} | |
171 | #for example | |
172 | debug mds balancer = 1/20 | |
173 | ||
174 | ||
175 | The following table provides a list of Ceph subsystems and their default log and | |
176 | memory levels. Once you complete your logging efforts, restore the subsystems | |
177 | to their default level or to a level suitable for normal operations. | |
178 | ||
179 | ||
180 | +--------------------+-----------+--------------+ | |
181 | | Subsystem | Log Level | Memory Level | | |
182 | +====================+===========+==============+ | |
183 | | ``default`` | 0 | 5 | | |
184 | +--------------------+-----------+--------------+ | |
185 | | ``lockdep`` | 0 | 5 | | |
186 | +--------------------+-----------+--------------+ | |
187 | | ``context`` | 0 | 5 | | |
188 | +--------------------+-----------+--------------+ | |
189 | | ``crush`` | 1 | 5 | | |
190 | +--------------------+-----------+--------------+ | |
191 | | ``mds`` | 1 | 5 | | |
192 | +--------------------+-----------+--------------+ | |
193 | | ``mds balancer`` | 1 | 5 | | |
194 | +--------------------+-----------+--------------+ | |
195 | | ``mds locker`` | 1 | 5 | | |
196 | +--------------------+-----------+--------------+ | |
197 | | ``mds log`` | 1 | 5 | | |
198 | +--------------------+-----------+--------------+ | |
199 | | ``mds log expire`` | 1 | 5 | | |
200 | +--------------------+-----------+--------------+ | |
201 | | ``mds migrator`` | 1 | 5 | | |
202 | +--------------------+-----------+--------------+ | |
203 | | ``buffer`` | 0 | 0 | | |
204 | +--------------------+-----------+--------------+ | |
205 | | ``timer`` | 0 | 5 | | |
206 | +--------------------+-----------+--------------+ | |
207 | | ``filer`` | 0 | 5 | | |
208 | +--------------------+-----------+--------------+ | |
209 | | ``objecter`` | 0 | 0 | | |
210 | +--------------------+-----------+--------------+ | |
211 | | ``rados`` | 0 | 5 | | |
212 | +--------------------+-----------+--------------+ | |
213 | | ``rbd`` | 0 | 5 | | |
214 | +--------------------+-----------+--------------+ | |
215 | | ``journaler`` | 0 | 5 | | |
216 | +--------------------+-----------+--------------+ | |
217 | | ``objectcacher`` | 0 | 5 | | |
218 | +--------------------+-----------+--------------+ | |
219 | | ``client`` | 0 | 5 | | |
220 | +--------------------+-----------+--------------+ | |
221 | | ``osd`` | 0 | 5 | | |
222 | +--------------------+-----------+--------------+ | |
223 | | ``optracker`` | 0 | 5 | | |
224 | +--------------------+-----------+--------------+ | |
225 | | ``objclass`` | 0 | 5 | | |
226 | +--------------------+-----------+--------------+ | |
227 | | ``filestore`` | 1 | 5 | | |
228 | +--------------------+-----------+--------------+ | |
229 | | ``journal`` | 1 | 5 | | |
230 | +--------------------+-----------+--------------+ | |
231 | | ``ms`` | 0 | 5 | | |
232 | +--------------------+-----------+--------------+ | |
233 | | ``mon`` | 1 | 5 | | |
234 | +--------------------+-----------+--------------+ | |
235 | | ``monc`` | 0 | 5 | | |
236 | +--------------------+-----------+--------------+ | |
237 | | ``paxos`` | 0 | 5 | | |
238 | +--------------------+-----------+--------------+ | |
239 | | ``tp`` | 0 | 5 | | |
240 | +--------------------+-----------+--------------+ | |
241 | | ``auth`` | 1 | 5 | | |
242 | +--------------------+-----------+--------------+ | |
243 | | ``finisher`` | 1 | 5 | | |
244 | +--------------------+-----------+--------------+ | |
245 | | ``heartbeatmap`` | 1 | 5 | | |
246 | +--------------------+-----------+--------------+ | |
247 | | ``perfcounter`` | 1 | 5 | | |
248 | +--------------------+-----------+--------------+ | |
249 | | ``rgw`` | 1 | 5 | | |
250 | +--------------------+-----------+--------------+ | |
251 | | ``javaclient`` | 1 | 5 | | |
252 | +--------------------+-----------+--------------+ | |
253 | | ``asok`` | 1 | 5 | | |
254 | +--------------------+-----------+--------------+ | |
255 | | ``throttle`` | 1 | 5 | | |
256 | +--------------------+-----------+--------------+ | |
257 | ||
258 | ||
259 | Logging Settings | |
260 | ---------------- | |
261 | ||
262 | Logging and debugging settings are not required in a Ceph configuration file, | |
263 | but you may override default settings as needed. Ceph supports the following | |
264 | settings: | |
265 | ||
266 | ||
267 | ``log file`` | |
268 | ||
269 | :Description: The location of the logging file for your cluster. | |
270 | :Type: String | |
271 | :Required: No | |
272 | :Default: ``/var/log/ceph/$cluster-$name.log`` | |
273 | ||
274 | ||
275 | ``log max new`` | |
276 | ||
277 | :Description: The maximum number of new log files. | |
278 | :Type: Integer | |
279 | :Required: No | |
280 | :Default: ``1000`` | |
281 | ||
282 | ||
283 | ``log max recent`` | |
284 | ||
285 | :Description: The maximum number of recent events to include in a log file. | |
286 | :Type: Integer | |
287 | :Required: No | |
288 | :Default: ``1000000`` | |
289 | ||
290 | ||
291 | ``log to stderr`` | |
292 | ||
293 | :Description: Determines if logging messages should appear in ``stderr``. | |
294 | :Type: Boolean | |
295 | :Required: No | |
296 | :Default: ``true`` | |
297 | ||
298 | ||
299 | ``err to stderr`` | |
300 | ||
301 | :Description: Determines if error messages should appear in ``stderr``. | |
302 | :Type: Boolean | |
303 | :Required: No | |
304 | :Default: ``true`` | |
305 | ||
306 | ||
307 | ``log to syslog`` | |
308 | ||
309 | :Description: Determines if logging messages should appear in ``syslog``. | |
310 | :Type: Boolean | |
311 | :Required: No | |
312 | :Default: ``false`` | |
313 | ||
314 | ||
315 | ``err to syslog`` | |
316 | ||
317 | :Description: Determines if error messages should appear in ``syslog``. | |
318 | :Type: Boolean | |
319 | :Required: No | |
320 | :Default: ``false`` | |
321 | ||
322 | ||
323 | ``log flush on exit`` | |
324 | ||
325 | :Description: Determines if Ceph should flush the log files after exit. | |
326 | :Type: Boolean | |
327 | :Required: No | |
328 | :Default: ``true`` | |
329 | ||
330 | ||
331 | ``clog to monitors`` | |
332 | ||
333 | :Description: Determines if ``clog`` messages should be sent to monitors. | |
334 | :Type: Boolean | |
335 | :Required: No | |
336 | :Default: ``true`` | |
337 | ||
338 | ||
339 | ``clog to syslog`` | |
340 | ||
341 | :Description: Determines if ``clog`` messages should be sent to syslog. | |
342 | :Type: Boolean | |
343 | :Required: No | |
344 | :Default: ``false`` | |
345 | ||
346 | ||
347 | ``mon cluster log to syslog`` | |
348 | ||
349 | :Description: Determines if the cluster log should be output to the syslog. | |
350 | :Type: Boolean | |
351 | :Required: No | |
352 | :Default: ``false`` | |
353 | ||
354 | ||
355 | ``mon cluster log file`` | |
356 | ||
357 | :Description: The location of the cluster's log file. | |
358 | :Type: String | |
359 | :Required: No | |
360 | :Default: ``/var/log/ceph/$cluster.log`` | |
361 | ||
362 | ||
363 | ||
364 | OSD | |
365 | --- | |
366 | ||
367 | ||
368 | ``osd debug drop ping probability`` | |
369 | ||
370 | :Description: ? | |
371 | :Type: Double | |
372 | :Required: No | |
373 | :Default: 0 | |
374 | ||
375 | ||
376 | ``osd debug drop ping duration`` | |
377 | ||
378 | :Description: | |
379 | :Type: Integer | |
380 | :Required: No | |
381 | :Default: 0 | |
382 | ||
383 | ``osd debug drop pg create probability`` | |
384 | ||
385 | :Description: | |
386 | :Type: Integer | |
387 | :Required: No | |
388 | :Default: 0 | |
389 | ||
390 | ``osd debug drop pg create duration`` | |
391 | ||
392 | :Description: ? | |
393 | :Type: Double | |
394 | :Required: No | |
395 | :Default: 1 | |
396 | ||
7c673cae FG |
397 | |
398 | ``osd tmapput sets uses tmap`` | |
399 | ||
400 | :Description: Uses ``tmap``. For debug only. | |
401 | :Type: Boolean | |
402 | :Required: No | |
403 | :Default: ``false`` | |
404 | ||
405 | ||
406 | ``osd min pg log entries`` | |
407 | ||
408 | :Description: The minimum number of log entries for placement groups. | |
409 | :Type: 32-bit Unsigned Integer | |
410 | :Required: No | |
411 | :Default: 1000 | |
412 | ||
413 | ||
414 | ``osd op log threshold`` | |
415 | ||
416 | :Description: How many op log messages to show up in one pass. | |
417 | :Type: Integer | |
418 | :Required: No | |
419 | :Default: 5 | |
420 | ||
421 | ||
422 | ||
423 | Filestore | |
424 | --------- | |
425 | ||
426 | ``filestore debug omap check`` | |
427 | ||
428 | :Description: Debugging check on synchronization. This is an expensive operation. | |
429 | :Type: Boolean | |
430 | :Required: No | |
431 | :Default: 0 | |
432 | ||
433 | ||
434 | MDS | |
435 | --- | |
436 | ||
437 | ||
438 | ``mds debug scatterstat`` | |
439 | ||
440 | :Description: Ceph will assert that various recursive stat invariants are true | |
441 | (for developers only). | |
442 | ||
443 | :Type: Boolean | |
444 | :Required: No | |
445 | :Default: ``false`` | |
446 | ||
447 | ||
448 | ``mds debug frag`` | |
449 | ||
450 | :Description: Ceph will verify directory fragmentation invariants when | |
451 | convenient (developers only). | |
452 | ||
453 | :Type: Boolean | |
454 | :Required: No | |
455 | :Default: ``false`` | |
456 | ||
457 | ||
458 | ``mds debug auth pins`` | |
459 | ||
460 | :Description: The debug auth pin invariants (for developers only). | |
461 | :Type: Boolean | |
462 | :Required: No | |
463 | :Default: ``false`` | |
464 | ||
465 | ||
466 | ``mds debug subtrees`` | |
467 | ||
468 | :Description: The debug subtree invariants (for developers only). | |
469 | :Type: Boolean | |
470 | :Required: No | |
471 | :Default: ``false`` | |
472 | ||
473 | ||
474 | ||
475 | RADOS Gateway | |
476 | ------------- | |
477 | ||
478 | ||
479 | ``rgw log nonexistent bucket`` | |
480 | ||
481 | :Description: Should we log a non-existent buckets? | |
482 | :Type: Boolean | |
483 | :Required: No | |
484 | :Default: ``false`` | |
485 | ||
486 | ||
487 | ``rgw log object name`` | |
488 | ||
489 | :Description: Should an object's name be logged. // man date to see codes (a subset are supported) | |
490 | :Type: String | |
491 | :Required: No | |
492 | :Default: ``%Y-%m-%d-%H-%i-%n`` | |
493 | ||
494 | ||
495 | ``rgw log object name utc`` | |
496 | ||
497 | :Description: Object log name contains UTC? | |
498 | :Type: Boolean | |
499 | :Required: No | |
500 | :Default: ``false`` | |
501 | ||
502 | ||
503 | ``rgw enable ops log`` | |
504 | ||
505 | :Description: Enables logging of every RGW operation. | |
506 | :Type: Boolean | |
507 | :Required: No | |
508 | :Default: ``true`` | |
509 | ||
510 | ||
511 | ``rgw enable usage log`` | |
512 | ||
513 | :Description: Enable logging of RGW's bandwidth usage. | |
514 | :Type: Boolean | |
515 | :Required: No | |
516 | :Default: ``true`` | |
517 | ||
518 | ||
519 | ``rgw usage log flush threshold`` | |
520 | ||
521 | :Description: Threshold to flush pending log data. | |
522 | :Type: Integer | |
523 | :Required: No | |
524 | :Default: ``1024`` | |
525 | ||
526 | ||
527 | ``rgw usage log tick interval`` | |
528 | ||
529 | :Description: Flush pending log data every ``s`` seconds. | |
530 | :Type: Integer | |
531 | :Required: No | |
532 | :Default: 30 | |
533 | ||
534 | ||
535 | ``rgw intent log object name`` | |
536 | ||
537 | :Description: | |
538 | :Type: String | |
539 | :Required: No | |
540 | :Default: ``%Y-%m-%d-%i-%n`` | |
541 | ||
542 | ||
543 | ``rgw intent log object name utc`` | |
544 | ||
545 | :Description: Include a UTC timestamp in the intent log object name. | |
546 | :Type: Boolean | |
547 | :Required: No | |
548 | :Default: ``false`` | |
549 | ||
550 | .. [#] there are levels >20 in some rare cases and that they are extremely verbose. |