ceph/doc/dev/crimson.rst

   1 =======
   2 crimson
   3 =======
   4
   5 Crimson is the code name of crimson-osd, which is the next generation ceph-osd.
   6 It targets fast networking devices, fast storage devices by leveraging state of
   7 the art technologies like DPDK and SPDK, for better performance. And it will
   8 keep the support of HDDs and low-end SSDs via BlueStore. Crismon will try to
   9 be backward compatible with classic OSD.
  10
  11
  12 Running Crimson
  13 ===============
  14
  15 As you might expect, crimson is not featurewise on par with its predecessor yet.
  16
  17 daemonize
  18 ---------
  19
  20 Unlike ``ceph-osd``, ``crimson-osd`` does daemonize itself even if the
  21 ``daemonize`` option is enabled. Because, to read this option, ``crimson-osd``
  22 needs to ready its config sharded service, but this sharded service lives
  23 in the seastar reactor. If we fork a child process and exit the parent after
  24 starting the Seastar engine, that will leave us with a single thread which is
  25 the replica of the thread calls `fork()`_. This would unnecessarily complicate
  26 the code, if we would have tackled this problem in crimson.
  27
  28 Since a lot of GNU/Linux distros are using systemd nowadays, which is able to
  29 daemonize the application, there is no need to daemonize by ourselves. For
  30 those who are using sysvinit, they can use ``start-stop-daemon`` for daemonizing
  31 ``crimson-osd``. If this is not acceptable, we can whip up a helper utility
  32 to do the trick.
  33
  34
  35 .. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
  36
  37
  38 logging
  39 -------
  40
  41 Currently, ``crimson-osd`` uses the logging utility offered by Seastar. see
  42 ``src/common/dout.h`` for the mapping between different logging levels to
  43 the severity levels in Seastar. For instance, the messages sent to ``derr``
  44 will be printed using ``logger::error()``, and the messages with debug level
  45 over ``20`` will be printed using ``logger::trace()``.
  46
  47 +---------+---------+
  48 | ceph    | seastar |
  49 +---------+---------+
  50 | < 0     | error   |
  51 +---------+---------+
  52 |   0     | warn    |
  53 +---------+---------+
  54 | [1, 5)  | info    |
  55 +---------+---------+
  56 | [5, 20] | debug   |
  57 +---------+---------+
  58 | >  20   | trace   |
  59 +---------+---------+
  60
  61 Please note, ``crimson-osd``
  62 does not send the logging message to specified ``log_file``. It writes
  63 the logging messages to stdout and/or syslog. Again, this behavior can be
  64 changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line
  65 options. By default, ``log-to-stdout`` is enabled, and the latter disabled.
  66
  67
  68 vstart.sh
  69 ---------
  70
  71 To facilitate the development of crimson, following options would be handy when
  72 using ``vstart.sh``,
  73
  74 ``--crimson``
  75     start ``crimson-osd`` instead of ``ceph-osd``
  76
  77 ``--nodaemon``
  78     do not daemonize the service
  79
  80 ``--redirect-output``
  81     redirect the stdout and stderr of service to ``out/$type.$num.stdout``.
  82
  83 ``--osd-args``
  84     pass extra command line options to crimson-osd or ceph-osd. It's quite
  85     useful for passing Seastar options to crimson-osd.
  86
  87 So, a typical command to start a single-crimson-node cluster is::
  88
  89   MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x --without-dashboard --memstore \
  90     --crimson --nodaemon --redirect-output \
  91     --osd-args "--memory 4G --smp 1 --cpuset 0"
  92
  93 Where we assign 4 GiB memory, a single thread running on core-0 to crimson-osd.
  94 Please refer ``crimson-osd --help-seastar`` for more Seastar specific command
  95 line options.
  96
  97 You could stop the vstart cluster using::
  98
  99   ../src/stop.sh --crimson
 100
 101
 102 CBT Based Testing
 103 =================
 104
 105 We can use `cbt`_ for performing perf tests::
 106
 107   $ git checkout master
 108   $ make crimson-osd
 109   $ ../src/script/run-cbt.sh --cbt ~/dev/cbt -a /tmp/baseline ../src/test/crimson/cbt/radosbench_4K_read.yaml
 110   $ git checkout yet-another-pr
 111   $ make crimson-osd
 112   $ ../src/script/run-cbt.sh --cbt ~/dev/cbt -a /tmp/yap ../src/test/crimson/cbt/radosbench_4K_read.yaml
 113   $ ~/dev/cbt/compare.py -b /tmp/baseline -a /tmp/yap -v
 114   19:48:23 - INFO     - cbt      - prefill/gen8/0: bandwidth: (or (greater) (near 0.05)):: 0.183165/0.186155  => accepted
 115   19:48:23 - INFO     - cbt      - prefill/gen8/0: iops_avg: (or (greater) (near 0.05)):: 46.0/47.0  => accepted
 116   19:48:23 - WARNING  - cbt      - prefill/gen8/0: iops_stddev: (or (less) (near 0.05)):: 10.4403/6.65833  => rejected
 117   19:48:23 - INFO     - cbt      - prefill/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.340868/0.333712  => accepted
 118   19:48:23 - INFO     - cbt      - prefill/gen8/1: bandwidth: (or (greater) (near 0.05)):: 0.190447/0.177619  => accepted
 119   19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_avg: (or (greater) (near 0.05)):: 48.0/45.0  => accepted
 120   19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_stddev: (or (less) (near 0.05)):: 6.1101/9.81495  => accepted
 121   19:48:23 - INFO     - cbt      - prefill/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.325163/0.350251  => accepted
 122   19:48:23 - INFO     - cbt      - seq/gen8/0: bandwidth: (or (greater) (near 0.05)):: 1.24654/1.22336  => accepted
 123   19:48:23 - INFO     - cbt      - seq/gen8/0: iops_avg: (or (greater) (near 0.05)):: 319.0/313.0  => accepted
 124   19:48:23 - INFO     - cbt      - seq/gen8/0: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  => accepted
 125   19:48:23 - INFO     - cbt      - seq/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.0497733/0.0509029  => accepted
 126   19:48:23 - INFO     - cbt      - seq/gen8/1: bandwidth: (or (greater) (near 0.05)):: 1.22717/1.11372  => accepted
 127   19:48:23 - INFO     - cbt      - seq/gen8/1: iops_avg: (or (greater) (near 0.05)):: 314.0/285.0  => accepted
 128   19:48:23 - INFO     - cbt      - seq/gen8/1: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  => accepted
 129   19:48:23 - INFO     - cbt      - seq/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.0508262/0.0557337  => accepted
 130   19:48:23 - WARNING  - cbt      - 1 tests failed out of 16
 131
 132 Where we compile and run the same test against two branches. One is ``master``, another is ``yet-another-pr`` branch.
 133 And then we compare the test results. Along with every test case, a set of rules is defined to check if we have
 134 performance regressions when comparing two set of test results. If a possible regression is found, the rule and
 135 corresponding test results are highlighted.
 136
 137 .. _cbt: https://github.com/ceph/cbt
 138
 139
 140 Debugging Crimson
 141 =================
 142
 143
 144 debugging tips
 145 --------------
 146
 147 When a seastar application crashes, it leaves us a serial of addresses, like::
 148
 149   Segmentation fault.
 150   Backtrace:
 151   0x00000000108254aa
 152   0x00000000107f74b9
 153   0x00000000105366cc
 154   0x000000001053682c
 155   0x00000000105d2c2e
 156   0x0000000010629b96
 157   0x0000000010629c31
 158   0x00002a02ebd8272f
 159   0x00000000105d93ee
 160   0x00000000103eff59
 161   0x000000000d9c1d0a
 162   /lib/x86_64-linux-gnu/libc.so.6+0x000000000002409a
 163   0x000000000d833ac9
 164   Segmentation fault
 165
 166 ``seastar-addr2line`` offered by Seastar can be used to decipher these
 167 addresses. After running the script, it will be waiting for input from stdin,
 168 so we need to copy and paste the above addresses, then send the EOF by inputting
 169 ``control-D`` in the terminal::
 170
 171   $ ../src/seastar/scripts/seastar-addr2line -e bin/crimson-osd
 172
 173   0x00000000108254aa
 174   0x00000000107f74b9
 175   0x00000000105366cc
 176   0x000000001053682c
 177   0x00000000105d2c2e
 178   0x0000000010629b96
 179   0x0000000010629c31
 180   0x00002a02ebd8272f
 181   0x00000000105d93ee
 182   0x00000000103eff59
 183   0x000000000d9c1d0a
 184   0x00000000108254aa
 185   [Backtrace #0]
 186   seastar::backtrace_buffer::append_backtrace() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1136
 187   seastar::print_with_backtrace(seastar::backtrace_buffer&) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1157
 188   seastar::print_with_backtrace(char const*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1164
 189   seastar::sigsegv_action() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5119
 190   seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5105
 191   seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5101
 192   ?? ??:0
 193   seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5418
 194   seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/kefu/dev/ceph/build/../src/seastar/src/core/app-template.cc:173 (discriminator 5)
 195   main at /home/kefu/dev/ceph/build/../src/crimson/osd/main.cc:131 (discriminator 1)