]>
Commit | Line | Data |
---|---|---|
1 | ================== | |
2 | Memory Profiling | |
3 | ================== | |
4 | ||
5 | Ceph MON, OSD and MDS can generate heap profiles using | |
6 | ``tcmalloc``. To generate heap profiles, ensure you have | |
7 | ``google-perftools`` installed:: | |
8 | ||
9 | sudo apt-get install google-perftools | |
10 | ||
11 | The profiler dumps output to your ``log file`` directory (i.e., | |
12 | ``/var/log/ceph``). See `Logging and Debugging`_ for details. | |
13 | To view the profiler logs with Google's performance tools, execute the | |
14 | following:: | |
15 | ||
16 | google-pprof --text {path-to-daemon} {log-path/filename} | |
17 | ||
18 | For example:: | |
19 | ||
20 | $ ceph tell osd.0 heap start_profiler | |
21 | $ ceph tell osd.0 heap dump | |
22 | osd.0 tcmalloc heap stats:------------------------------------------------ | |
23 | MALLOC: 2632288 ( 2.5 MiB) Bytes in use by application | |
24 | MALLOC: + 499712 ( 0.5 MiB) Bytes in page heap freelist | |
25 | MALLOC: + 543800 ( 0.5 MiB) Bytes in central cache freelist | |
26 | MALLOC: + 327680 ( 0.3 MiB) Bytes in transfer cache freelist | |
27 | MALLOC: + 1239400 ( 1.2 MiB) Bytes in thread cache freelists | |
28 | MALLOC: + 1142936 ( 1.1 MiB) Bytes in malloc metadata | |
29 | MALLOC: ------------ | |
30 | MALLOC: = 6385816 ( 6.1 MiB) Actual memory used (physical + swap) | |
31 | MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped) | |
32 | MALLOC: ------------ | |
33 | MALLOC: = 6385816 ( 6.1 MiB) Virtual address space used | |
34 | MALLOC: | |
35 | MALLOC: 231 Spans in use | |
36 | MALLOC: 56 Thread heaps in use | |
37 | MALLOC: 8192 Tcmalloc page size | |
38 | ------------------------------------------------ | |
39 | Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). | |
40 | Bytes released to the OS take up virtual address space but no physical memory. | |
41 | $ google-pprof --text \ | |
42 | /usr/bin/ceph-osd \ | |
43 | /var/log/ceph/ceph-osd.0.profile.0001.heap | |
44 | Total: 3.7 MB | |
45 | 1.9 51.1% 51.1% 1.9 51.1% ceph::log::Log::create_entry | |
46 | 1.8 47.3% 98.4% 1.8 47.3% std::string::_Rep::_S_create | |
47 | 0.0 0.4% 98.9% 0.0 0.6% SimpleMessenger::add_accept_pipe | |
48 | 0.0 0.4% 99.2% 0.0 0.6% decode_message | |
49 | ... | |
50 | ||
51 | Another heap dump on the same daemon will add another file. It is | |
52 | convenient to compare to a previous heap dump to show what has grown | |
53 | in the interval. For instance:: | |
54 | ||
55 | $ google-pprof --text --base out/osd.0.profile.0001.heap \ | |
56 | ceph-osd out/osd.0.profile.0003.heap | |
57 | Total: 0.2 MB | |
58 | 0.1 50.3% 50.3% 0.1 50.3% ceph::log::Log::create_entry | |
59 | 0.1 46.6% 96.8% 0.1 46.6% std::string::_Rep::_S_create | |
60 | 0.0 0.9% 97.7% 0.0 26.1% ReplicatedPG::do_op | |
61 | 0.0 0.8% 98.5% 0.0 0.8% __gnu_cxx::new_allocator::allocate | |
62 | ||
63 | Refer to `Google Heap Profiler`_ for additional details. | |
64 | ||
65 | Once you have the heap profiler installed, start your cluster and | |
66 | begin using the heap profiler. You may enable or disable the heap | |
67 | profiler at runtime, or ensure that it runs continuously. For the | |
68 | following commandline usage, replace ``{daemon-type}`` with ``mon``, | |
69 | ``osd`` or ``mds``, and replace ``{daemon-id}`` with the OSD number or | |
70 | the MON or MDS id. | |
71 | ||
72 | ||
73 | Starting the Profiler | |
74 | --------------------- | |
75 | ||
76 | To start the heap profiler, execute the following:: | |
77 | ||
78 | ceph tell {daemon-type}.{daemon-id} heap start_profiler | |
79 | ||
80 | For example:: | |
81 | ||
82 | ceph tell osd.1 heap start_profiler | |
83 | ||
84 | Alternatively the profile can be started when the daemon starts | |
85 | running if the ``CEPH_HEAP_PROFILER_INIT=true`` variable is found in | |
86 | the environment. | |
87 | ||
88 | Printing Stats | |
89 | -------------- | |
90 | ||
91 | To print out statistics, execute the following:: | |
92 | ||
93 | ceph tell {daemon-type}.{daemon-id} heap stats | |
94 | ||
95 | For example:: | |
96 | ||
97 | ceph tell osd.0 heap stats | |
98 | ||
99 | .. note:: Printing stats does not require the profiler to be running and does | |
100 | not dump the heap allocation information to a file. | |
101 | ||
102 | ||
103 | Dumping Heap Information | |
104 | ------------------------ | |
105 | ||
106 | To dump heap information, execute the following:: | |
107 | ||
108 | ceph tell {daemon-type}.{daemon-id} heap dump | |
109 | ||
110 | For example:: | |
111 | ||
112 | ceph tell mds.a heap dump | |
113 | ||
114 | .. note:: Dumping heap information only works when the profiler is running. | |
115 | ||
116 | ||
117 | Releasing Memory | |
118 | ---------------- | |
119 | ||
120 | To release memory that ``tcmalloc`` has allocated but which is not being used by | |
121 | the Ceph daemon itself, execute the following:: | |
122 | ||
123 | ceph tell {daemon-type}{daemon-id} heap release | |
124 | ||
125 | For example:: | |
126 | ||
127 | ceph tell osd.2 heap release | |
128 | ||
129 | ||
130 | Stopping the Profiler | |
131 | --------------------- | |
132 | ||
133 | To stop the heap profiler, execute the following:: | |
134 | ||
135 | ceph tell {daemon-type}.{daemon-id} heap stop_profiler | |
136 | ||
137 | For example:: | |
138 | ||
139 | ceph tell osd.0 heap stop_profiler | |
140 | ||
141 | .. _Logging and Debugging: ../log-and-debug | |
142 | .. _Google Heap Profiler: http://goog-perftools.sourceforge.net/doc/heap_profiler.html |