]>
Commit | Line | Data |
---|---|---|
31f18b77 FG |
1 | ================= |
2 | Perf histograms | |
3 | ================= | |
4 | ||
5 | The perf histograms build on perf counters infrastructure. Histograms are built for a number of counters and simplify gathering data on which groups of counter values occur most often over time. | |
6 | Perf histograms are currently unsigned 64-bit integer counters, so they're mostly useful for time and sizes. Data dumped by perf histogram can then be feed into other analysis tools/scripts. | |
7 | ||
8 | Access | |
9 | ------ | |
10 | ||
11 | The perf histogram data are accessed via the admin socket. For example:: | |
12 | ||
13 | ceph daemon osd.0 perf histogram schema | |
14 | ceph daemon osd.0 perf histogram dump | |
15 | ||
16 | ||
17 | Collections | |
18 | ----------- | |
19 | ||
20 | The histograms are grouped into named collections, normally representing a subsystem or an instance of a subsystem. For example, the internal ``throttle`` mechanism reports statistics on how it is throttling, and each instance is named something like:: | |
21 | ||
22 | ||
23 | op_r_latency_out_bytes_histogram | |
24 | op_rw_latency_in_bytes_histogram | |
25 | op_rw_latency_out_bytes_histogram | |
26 | ... | |
27 | ||
28 | ||
29 | Schema | |
30 | ------ | |
31 | ||
32 | The ``perf histogram schema`` command dumps a json description of which values are available, and what their type is. Each named value as a ``type`` bitfield, with the 5-th bit always set and following bits defined. | |
33 | ||
34 | +------+-------------------------------------+ | |
35 | | bit | meaning | | |
36 | +======+=====================================+ | |
37 | | 1 | floating point value | | |
38 | +------+-------------------------------------+ | |
39 | | 2 | unsigned 64-bit integer value | | |
40 | +------+-------------------------------------+ | |
41 | | 4 | average (sum + count pair) | | |
42 | +------+-------------------------------------+ | |
43 | | 8 | counter (vs gauge) | | |
44 | +------+-------------------------------------+ | |
45 | ||
46 | In other words, histogram of type "18" is a histogram of unsigned 64-bit integer values (16 + 2). | |
47 | ||
48 | Here is an example of the schema output:: | |
49 | ||
224ce89b WB |
50 | { |
51 | "AsyncMessenger::Worker-0": {}, | |
52 | "AsyncMessenger::Worker-1": {}, | |
53 | "AsyncMessenger::Worker-2": {}, | |
54 | "mutex-WBThrottle::lock": {}, | |
55 | "objecter": {}, | |
56 | "osd": { | |
57 | "op_r_latency_out_bytes_histogram": { | |
58 | "type": 18, | |
59 | "description": "Histogram of operation latency (including queue time) + da ta read", | |
60 | "nick": "" | |
61 | }, | |
62 | "op_w_latency_in_bytes_histogram": { | |
63 | "type": 18, | |
64 | "description": "Histogram of operation latency (including queue time) + da ta written", | |
65 | "nick": "" | |
66 | }, | |
67 | "op_rw_latency_in_bytes_histogram": { | |
68 | "type": 18, | |
69 | "description": "Histogram of rw operation latency (including queue time) + data written", | |
70 | "nick": "" | |
71 | }, | |
72 | "op_rw_latency_out_bytes_histogram": { | |
73 | "type": 18, | |
74 | "description": "Histogram of rw operation latency (including queue time) + data read", | |
75 | "nick": "" | |
76 | } | |
31f18b77 FG |
77 | } |
78 | } | |
31f18b77 FG |
79 | |
80 | ||
81 | Dump | |
82 | ---- | |
83 | ||
84 | The actual dump is similar to the schema, except that there are actual value groups. For example:: | |
85 | ||
86 | "osd": { | |
87 | "op_r_latency_out_bytes_histogram": { | |
88 | "axes": [ | |
89 | { | |
90 | "name": "Latency (usec)", | |
91 | "min": 0, | |
92 | "quant_size": 100000, | |
93 | "buckets": 32, | |
94 | "scale_type": "log2", | |
95 | "ranges": [ | |
96 | { | |
97 | "max": -1 | |
98 | }, | |
99 | { | |
100 | "min": 0, | |
101 | "max": 99999 | |
102 | }, | |
103 | { | |
104 | "min": 100000, | |
105 | "max": 199999 | |
106 | }, | |
107 | { | |
108 | "min": 200000, | |
109 | "max": 399999 | |
110 | }, | |
111 | { | |
112 | "min": 400000, | |
113 | "max": 799999 | |
114 | }, | |
115 | { | |
116 | "min": 800000, | |
117 | "max": 1599999 | |
118 | }, | |
119 | { | |
120 | "min": 1600000, | |
121 | "max": 3199999 | |
122 | }, | |
123 | { | |
124 | "min": 3200000, | |
125 | "max": 6399999 | |
126 | }, | |
127 | { | |
128 | "min": 6400000, | |
129 | "max": 12799999 | |
130 | }, | |
131 | { | |
132 | "min": 12800000, | |
133 | "max": 25599999 | |
134 | }, | |
135 | { | |
136 | "min": 25600000, | |
137 | "max": 51199999 | |
138 | }, | |
139 | { | |
140 | "min": 51200000, | |
141 | "max": 102399999 | |
142 | }, | |
143 | { | |
144 | "min": 102400000, | |
145 | "max": 204799999 | |
146 | }, | |
147 | { | |
148 | "min": 204800000, | |
149 | "max": 409599999 | |
150 | }, | |
151 | { | |
152 | "min": 409600000, | |
153 | "max": 819199999 | |
154 | }, | |
155 | { | |
156 | "min": 819200000, | |
157 | "max": 1638399999 | |
158 | }, | |
159 | { | |
160 | "min": 1638400000, | |
161 | "max": 3276799999 | |
162 | }, | |
163 | { | |
164 | "min": 3276800000, | |
165 | "max": 6553599999 | |
166 | }, | |
167 | { | |
168 | "min": 6553600000, | |
169 | "max": 13107199999 | |
170 | }, | |
171 | { | |
172 | "min": 13107200000, | |
173 | "max": 26214399999 | |
174 | }, | |
175 | { | |
176 | "min": 26214400000, | |
177 | "max": 52428799999 | |
178 | }, | |
179 | { | |
180 | "min": 52428800000, | |
181 | "max": 104857599999 | |
182 | }, | |
183 | { | |
184 | "min": 104857600000, | |
185 | "max": 209715199999 | |
186 | }, | |
187 | { | |
188 | "min": 209715200000, | |
189 | "max": 419430399999 | |
190 | }, | |
191 | { | |
192 | "min": 419430400000, | |
193 | "max": 838860799999 | |
194 | }, | |
195 | { | |
196 | "min": 838860800000, | |
197 | "max": 1677721599999 | |
198 | }, | |
199 | { | |
200 | "min": 1677721600000, | |
201 | "max": 3355443199999 | |
202 | }, | |
203 | { | |
204 | "min": 3355443200000, | |
205 | "max": 6710886399999 | |
206 | }, | |
207 | { | |
208 | "min": 6710886400000, | |
209 | "max": 13421772799999 | |
210 | }, | |
211 | { | |
212 | "min": 13421772800000, | |
213 | "max": 26843545599999 | |
214 | }, | |
215 | { | |
216 | "min": 26843545600000, | |
217 | "max": 53687091199999 | |
218 | }, | |
219 | }, | |
220 | { | |
221 | "min": 53687091200000 | |
222 | } | |
223 | ] | |
224 | }, | |
225 | { | |
226 | "name": "Request size (bytes)", | |
227 | "min": 0, | |
228 | "quant_size": 512, | |
229 | "buckets": 32, | |
230 | "scale_type": "log2", | |
231 | "ranges": [ | |
232 | { | |
233 | "max": -1 | |
234 | }, | |
235 | { | |
236 | "min": 0, | |
237 | "max": 511 | |
238 | }, | |
239 | { | |
240 | "min": 512, | |
241 | "max": 1023 | |
242 | }, | |
243 | { | |
244 | "min": 1024, | |
245 | "max": 2047 | |
246 | }, | |
247 | { | |
248 | "min": 2048, | |
249 | "max": 4095 | |
250 | }, | |
251 | { | |
252 | "min": 4096, | |
253 | "max": 8191 | |
254 | }, | |
255 | { | |
256 | "min": 8192, | |
257 | "max": 16383 | |
258 | }, | |
259 | { | |
260 | "min": 16384, | |
261 | "max": 32767 | |
262 | }, | |
263 | { | |
264 | "min": 32768, | |
265 | "max": 65535 | |
266 | }, | |
267 | { | |
268 | "min": 65536, | |
269 | "max": 131071 | |
270 | }, | |
271 | { | |
272 | "min": 131072, | |
273 | "max": 262143 | |
274 | }, | |
275 | { | |
276 | "min": 262144, | |
277 | "max": 524287 | |
278 | }, | |
279 | { | |
280 | "min": 524288, | |
281 | "max": 1048575 | |
282 | }, | |
283 | { | |
284 | "min": 1048576, | |
285 | "max": 2097151 | |
286 | }, | |
287 | { | |
288 | "min": 2097152, | |
289 | "max": 4194303 | |
290 | }, | |
291 | { | |
292 | "min": 4194304, | |
293 | "max": 8388607 | |
294 | }, | |
295 | { | |
296 | "min": 8388608, | |
297 | "max": 16777215 | |
298 | }, | |
299 | { | |
300 | "min": 16777216, | |
301 | "max": 33554431 | |
302 | }, | |
303 | { | |
304 | "min": 33554432, | |
305 | "max": 67108863 | |
306 | }, | |
307 | { | |
308 | "min": 67108864, | |
309 | "max": 134217727 | |
310 | }, | |
311 | { | |
312 | "min": 134217728, | |
313 | "max": 268435455 | |
314 | }, | |
315 | { | |
316 | "min": 268435456, | |
317 | "max": 536870911 | |
318 | }, | |
319 | { | |
320 | "min": 536870912, | |
321 | "max": 1073741823 | |
322 | }, | |
323 | { | |
324 | "min": 1073741824, | |
325 | "max": 2147483647 | |
326 | }, | |
327 | { | |
328 | "min": 2147483648, | |
329 | "max": 4294967295 | |
330 | }, | |
331 | { | |
332 | "min": 4294967296, | |
333 | "max": 8589934591 | |
334 | }, | |
335 | { | |
336 | "min": 8589934592, | |
337 | "max": 17179869183 | |
338 | }, | |
339 | { | |
340 | "min": 17179869184, | |
341 | "max": 34359738367 | |
342 | }, | |
343 | { | |
344 | "min": 34359738368, | |
345 | "max": 68719476735 | |
346 | }, | |
347 | { | |
348 | "min": 68719476736, | |
349 | "max": 137438953471 | |
350 | }, | |
351 | { | |
352 | "min": 137438953472, | |
353 | "max": 274877906943 | |
354 | }, | |
355 | { | |
356 | "min": 274877906944 | |
357 | } | |
358 | ] | |
359 | } | |
360 | ], | |
361 | "values": [ | |
362 | [ | |
363 | 0, | |
364 | 0, | |
365 | 0, | |
366 | 0, | |
367 | 0, | |
368 | 0, | |
369 | 0, | |
370 | 0, | |
371 | 0, | |
372 | 0, | |
373 | 0, | |
374 | 0, | |
375 | 0, | |
376 | 0, | |
377 | 0, | |
378 | 0, | |
379 | 0, | |
380 | 0, | |
381 | 0, | |
382 | 0, | |
383 | 0, | |
384 | 0, | |
385 | 0, | |
386 | 0, | |
387 | 0, | |
388 | 0, | |
389 | 0, | |
390 | 0, | |
391 | 0, | |
392 | 0, | |
393 | 0, | |
394 | 0 | |
395 | ], | |
396 | [ | |
397 | 0, | |
398 | 0, | |
399 | 0, | |
400 | 0, | |
401 | 0, | |
402 | 0, | |
403 | 0, | |
404 | 0, | |
405 | 0, | |
406 | 0, | |
407 | 0, | |
408 | 0, | |
409 | 0, | |
410 | 0, | |
411 | 0, | |
412 | 0, | |
413 | 0, | |
414 | 0, | |
415 | 0, | |
416 | 0, | |
417 | 0, | |
418 | 0, | |
419 | 0, | |
420 | 0, | |
421 | 0, | |
422 | 0, | |
423 | 0, | |
424 | 0, | |
425 | 0, | |
426 | 0, | |
427 | 0, | |
428 | 0 | |
429 | ], | |
430 | [ | |
431 | 0, | |
432 | 0, | |
433 | 0, | |
434 | 0, | |
435 | 0, | |
436 | 0, | |
437 | 0, | |
438 | 0, | |
439 | 0, | |
440 | 0, | |
441 | 0, | |
442 | 0, | |
443 | 0, | |
444 | 0, | |
445 | 0, | |
446 | 0, | |
447 | 0, | |
448 | 0, | |
449 | 0, | |
450 | 0, | |
451 | 0, | |
452 | 0, | |
453 | 0, | |
454 | 0, | |
455 | 0, | |
456 | 0, | |
457 | 0, | |
458 | 0, | |
459 | 0, | |
460 | 0, | |
461 | 0, | |
462 | 0 | |
463 | ], | |
464 | [ | |
465 | 0, | |
466 | 0, | |
467 | 0, | |
468 | 0, | |
469 | 0, | |
470 | 0, | |
471 | 0, | |
472 | 0, | |
473 | 0, | |
474 | 0, | |
475 | 0, | |
476 | 0, | |
477 | 0, | |
478 | 0, | |
479 | 0, | |
480 | 0, | |
481 | 0, | |
482 | 0, | |
483 | 0, | |
484 | 0, | |
485 | 0, | |
486 | 0, | |
487 | 0, | |
488 | 0, | |
489 | 0, | |
490 | 0, | |
491 | 0, | |
492 | 0, | |
493 | 0, | |
494 | 0, | |
495 | 0, | |
496 | 0 | |
497 | ], | |
498 | [ | |
499 | 0, | |
500 | 0, | |
501 | 0, | |
502 | 0, | |
503 | 0, | |
504 | 0, | |
505 | 0, | |
506 | 0, | |
507 | 0, | |
508 | 0, | |
509 | 0, | |
510 | 0, | |
511 | 0, | |
512 | 0, | |
513 | 0, | |
514 | 0, | |
515 | 0, | |
516 | 0, | |
517 | 0, | |
518 | 0, | |
519 | 0, | |
520 | 0, | |
521 | 0, | |
522 | 0, | |
523 | 0, | |
524 | 0, | |
525 | 0, | |
526 | 0, | |
527 | 0, | |
528 | 0, | |
529 | 0, | |
530 | 0 | |
531 | ], | |
532 | [ | |
533 | 0, | |
534 | 0, | |
535 | 0, | |
536 | 0, | |
537 | 0, | |
538 | 0, | |
539 | 0, | |
540 | 0, | |
541 | 0, | |
542 | 0, | |
543 | 0, | |
544 | 0, | |
545 | 0, | |
546 | 0, | |
547 | 0, | |
548 | 0, | |
549 | 0, | |
550 | 0, | |
551 | 0, | |
552 | 0, | |
553 | 0, | |
554 | 0, | |
555 | 0, | |
556 | 0, | |
557 | 0, | |
558 | 0, | |
559 | 0, | |
560 | 0, | |
561 | 0, | |
562 | 0, | |
563 | 0, | |
564 | 0 | |
565 | ], | |
566 | [ | |
567 | 0, | |
568 | 0, | |
569 | 0, | |
570 | 0, | |
571 | 0, | |
572 | 0, | |
573 | 0, | |
574 | 0, | |
575 | 0, | |
576 | 0, | |
577 | 0, | |
578 | 0, | |
579 | 0, | |
580 | 0, | |
581 | 0, | |
582 | 0, | |
583 | 0, | |
584 | 0, | |
585 | 0, | |
586 | 0, | |
587 | 0, | |
588 | 0, | |
589 | 0, | |
590 | 0, | |
591 | 0, | |
592 | 0, | |
593 | 0, | |
594 | 0, | |
595 | 0, | |
596 | 0, | |
597 | 0, | |
598 | 0 | |
599 | ], | |
600 | [ | |
601 | 0, | |
602 | 0, | |
603 | 0, | |
604 | 0, | |
605 | 0, | |
606 | 0, | |
607 | 0, | |
608 | 0, | |
609 | 0, | |
610 | 0, | |
611 | 0, | |
612 | 0, | |
613 | 0, | |
614 | 0, | |
615 | 0, | |
616 | 0, | |
617 | 0, | |
618 | 0, | |
619 | 0, | |
620 | 0, | |
621 | 0, | |
622 | 0, | |
623 | 0, | |
624 | 0, | |
625 | 0, | |
626 | 0, | |
627 | 0, | |
628 | 0, | |
629 | 0, | |
630 | 0, | |
631 | 0, | |
632 | 0 | |
633 | ], | |
634 | [ | |
635 | 0, | |
636 | 0, | |
637 | 0, | |
638 | 0, | |
639 | 0, | |
640 | 0, | |
641 | 0, | |
642 | 0, | |
643 | 0, | |
644 | 0, | |
645 | 0, | |
646 | 0, | |
647 | 0, | |
648 | 0, | |
649 | 0, | |
650 | 0, | |
651 | 0, | |
652 | 0, | |
653 | 0, | |
654 | 0, | |
655 | 0, | |
656 | 0, | |
657 | 0, | |
658 | 0, | |
659 | 0, | |
660 | 0, | |
661 | 0, | |
662 | 0, | |
663 | 0, | |
664 | 0, | |
665 | 0, | |
666 | 0 | |
667 | ] | |
668 | ] | |
669 | } | |
670 | }, | |
671 | ||
672 | This represents the 2d histogram, consisting of 9 history entrires and 32 value groups per each history entry. | |
673 | "Ranges" element denote value bounds for each of value groups. "Buckets" denote amount of value groups ("buckets"), | |
674 | "Min" is a minimum accepted valaue, "quant_size" is quantization unit and "scale_type" is either "log2" (logarhitmic | |
675 | scale) or "linear" (linear scale). | |
676 | You can use histogram_dump.py tool (see src/tools/histogram_dump.py) for quick visualisation of existing histogram | |
677 | data. |