ceph/doc/rados/troubleshooting/memory-profiling.rst

   1 ==================
   2  Memory Profiling
   3 ==================
   4
   5 Ceph Monitor, OSD, and MDS can report ``TCMalloc`` heap profiles. Install
   6 ``google-perftools`` if you want to generate these. Your OS distribution might
   7 package this under a different name (for example, ``gperftools``), and your OS
   8 distribution might use a different package manager. Run a command similar to
   9 this one to install ``google-perftools``:
  10
  11 .. prompt:: bash
  12
  13     sudo apt-get install google-perftools
  14
  15 The profiler dumps output to your ``log file`` directory (``/var/log/ceph``).
  16 See `Logging and Debugging`_ for details.
  17
  18 To view the profiler logs with Google's performance tools, run the following
  19 command:
  20
  21 .. prompt:: bash
  22
  23     google-pprof --text {path-to-daemon}  {log-path/filename}
  24
  25 For example::
  26
  27     $ ceph tell osd.0 heap start_profiler
  28     $ ceph tell osd.0 heap dump
  29     osd.0 tcmalloc heap stats:------------------------------------------------
  30     MALLOC:        2632288 (    2.5 MiB) Bytes in use by application
  31     MALLOC: +       499712 (    0.5 MiB) Bytes in page heap freelist
  32     MALLOC: +       543800 (    0.5 MiB) Bytes in central cache freelist
  33     MALLOC: +       327680 (    0.3 MiB) Bytes in transfer cache freelist
  34     MALLOC: +      1239400 (    1.2 MiB) Bytes in thread cache freelists
  35     MALLOC: +      1142936 (    1.1 MiB) Bytes in malloc metadata
  36     MALLOC:   ------------
  37     MALLOC: =      6385816 (    6.1 MiB) Actual memory used (physical + swap)
  38     MALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)
  39     MALLOC:   ------------
  40     MALLOC: =      6385816 (    6.1 MiB) Virtual address space used
  41     MALLOC:
  42     MALLOC:            231              Spans in use
  43     MALLOC:             56              Thread heaps in use
  44     MALLOC:           8192              Tcmalloc page size
  45     ------------------------------------------------
  46     Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
  47     Bytes released to the OS take up virtual address space but no physical memory.
  48     $ google-pprof --text \
  49                    /usr/bin/ceph-osd  \
  50                    /var/log/ceph/ceph-osd.0.profile.0001.heap
  51      Total: 3.7 MB
  52      1.9  51.1%  51.1%      1.9  51.1% ceph::log::Log::create_entry
  53      1.8  47.3%  98.4%      1.8  47.3% std::string::_Rep::_S_create
  54      0.0   0.4%  98.9%      0.0   0.6% SimpleMessenger::add_accept_pipe
  55      0.0   0.4%  99.2%      0.0   0.6% decode_message
  56      ...
  57
  58 Performing another heap dump on the same daemon creates another file. It is
  59 convenient to compare the new file to a file created by a previous heap dump to
  60 show what has grown in the interval. For example::
  61
  62     $ google-pprof --text --base out/osd.0.profile.0001.heap \
  63           ceph-osd out/osd.0.profile.0003.heap
  64      Total: 0.2 MB
  65      0.1  50.3%  50.3%      0.1  50.3% ceph::log::Log::create_entry
  66      0.1  46.6%  96.8%      0.1  46.6% std::string::_Rep::_S_create
  67      0.0   0.9%  97.7%      0.0  26.1% ReplicatedPG::do_op
  68      0.0   0.8%  98.5%      0.0   0.8% __gnu_cxx::new_allocator::allocate
  69
  70 See `Google Heap Profiler`_ for additional details.
  71
  72 After you have installed the heap profiler, start your cluster and begin using
  73 the heap profiler. You can enable or disable the heap profiler at runtime, or
  74 ensure that it runs continuously. When running commands based on the examples
  75 that follow, do the following:
  76
  77 #. replace ``{daemon-type}`` with ``mon``, ``osd`` or ``mds``
  78 #. replace ``{daemon-id}`` with the OSD number or the MON ID or the MDS ID
  79
  80
  81 Starting the Profiler
  82 ---------------------
  83
  84 To start the heap profiler, run a command of the following form:
  85
  86 .. prompt:: bash
  87
  88    ceph tell {daemon-type}.{daemon-id} heap start_profiler
  89
  90 For example:
  91
  92 .. prompt:: bash
  93
  94    ceph tell osd.1 heap start_profiler
  95
  96 Alternatively, if the ``CEPH_HEAP_PROFILER_INIT=true`` variable is found in the
  97 environment, the profile will be started when the daemon starts running.
  98
  99 Printing Stats
 100 --------------
 101
 102 To print out statistics, run a command of the following form:
 103
 104 .. prompt:: bash
 105
 106    ceph  tell {daemon-type}.{daemon-id} heap stats
 107
 108 For example:
 109
 110 .. prompt:: bash
 111
 112    ceph tell osd.0 heap stats
 113
 114 .. note:: The reporting of stats with this command does not require the
 115    profiler to be running and does not dump the heap allocation information to
 116    a file.
 117
 118
 119 Dumping Heap Information
 120 ------------------------
 121
 122 To dump heap information, run a command of the following form:
 123
 124 .. prompt:: bash
 125
 126    ceph tell {daemon-type}.{daemon-id} heap dump
 127
 128 For example:
 129
 130 .. prompt:: bash
 131
 132    ceph tell mds.a heap dump
 133
 134 .. note:: Dumping heap information works only when the profiler is running.
 135
 136
 137 Releasing Memory
 138 ----------------
 139
 140 To release memory that ``tcmalloc`` has allocated but which is not being used
 141 by the Ceph daemon itself, run a command of the following form:
 142
 143 .. prompt:: bash
 144
 145    ceph tell {daemon-type}{daemon-id} heap release
 146
 147 For example:
 148
 149 .. prompt:: bash
 150
 151     ceph tell osd.2 heap release
 152
 153
 154 Stopping the Profiler
 155 ---------------------
 156
 157 To stop the heap profiler, run a command of the following form:
 158
 159 .. prompt:: bash
 160
 161    ceph tell {daemon-type}.{daemon-id} heap stop_profiler
 162
 163 For example:
 164
 165 .. prompt:: bash
 166
 167    ceph tell osd.0 heap stop_profiler
 168
 169 .. _Logging and Debugging: ../log-and-debug
 170 .. _Google Heap Profiler: http://goog-perftools.sourceforge.net/doc/heap_profiler.html
 171
 172 Alternative Methods of  Memory Profiling
 173 ----------------------------------------
 174
 175 Running Massif heap profiler with Valgrind
 176 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 177
 178 The Massif heap profiler tool can be used with Valgrind to measure how much
 179 heap memory is used. This method is well-suited to troubleshooting RadosGW.
 180
 181 See the `Massif documentation
 182 <https://valgrind.org/docs/manual/ms-manual.html>`_ for more information.
 183
 184 Install Valgrind from the package manager for your distribution then start the
 185 Ceph daemon you want to troubleshoot:
 186
 187 .. prompt:: bash
 188
 189    sudo -u ceph valgrind --max-threads=1024 --tool=massif /usr/bin/radosgw -f --cluster ceph --name NAME --setuser ceph --setgroup ceph
 190
 191 When this command has completed its run, a file with a name of the form
 192 ``massif.out.<pid>`` will be saved in your current working directory. To run
 193 the command above, the user who runs it must have write permissions in the
 194 current directory.
 195
 196 Run the ``ms_print`` command to get a graph and statistics from the collected
 197 data in the ``massif.out.<pid>`` file:
 198
 199 .. prompt:: bash
 200
 201    ms_print massif.out.12345
 202
 203 The output of this command is helpful when submitting a bug report.