ceph/doc/rados/configuration/msgr2.rst

   1 .. _msgr2:
   2
   3 Messenger v2
   4 ============
   5
   6 What is it
   7 ----------
   8
   9 The messenger v2 protocol, or msgr2, is the second major revision on
  10 Ceph's on-wire protocol.  It brings with it several key features:
  11
  12 * A *secure* mode that encrypts all data passing over the network
  13 * Improved encapsulation of authentication payloads, enabling future
  14   integration of new authentication modes like Kerberos
  15 * Improved earlier feature advertisement and negotiation, enabling
  16   future protocol revisions
  17
  18 Ceph daemons can now bind to multiple ports, allowing both legacy Ceph
  19 clients and new v2-capable clients to connect to the same cluster.
  20
  21 By default, monitors now bind to the new IANA-assigned port ``3300``
  22 (ce4h or 0xce4) for the new v2 protocol, while also binding to the
  23 old default port ``6789`` for the legacy v1 protocol.
  24
  25 Address formats
  26 ---------------
  27
  28 Prior to nautilus, all network addresses were rendered like
  29 ``1.2.3.4:567/89012`` where there was an IP address, a port, and a
  30 nonce to uniquely identify a client or daemon on the network.
  31 Starting with nautilus, we now have three different address types:
  32
  33 * **v2**: ``v2:1.2.3.4:578/89012`` identifies a daemon binding to a
  34   port speaking the new v2 protocol
  35 * **v1**: ``v1:1.2.3.4:578/89012`` identifies a daemon binding to a
  36   port speaking the legacy v1 protocol.  Any address that was
  37   previously shown with any prefix is now shown as a ``v1:`` address.
  38 * **TYPE_ANY** addresses identify a client that can speak either
  39   version of the protocol.  Prior to nautilus, clients would appear as
  40   ``1.2.3.4:0/123456``, where the port of 0 indicates they are clients
  41   and do not accept incoming connections.  Starting with Nautilus,
  42   these clients are now internally represented by a **TYPE_ANY**
  43   address, and still shown with no prefix, because they may
  44   connect to daemons using the v2 or v1 protocol, depending on what
  45   protocol(s) the daemons are using.
  46
  47 Because daemons now bind to multiple ports, they are now described by
  48 a vector of addresses instead of a single address.  For example,
  49 dumping the monitor map on a Nautilus cluster now includes lines
  50 like::
  51
  52   epoch 1
  53   fsid 50fcf227-be32-4bcb-8b41-34ca8370bd16
  54   last_changed 2019-02-25 11:10:46.700821
  55   created 2019-02-25 11:10:46.700821
  56   min_mon_release 14 (nautilus)
  57   0: [v2:10.0.0.10:3300/0,v1:10.0.0.10:6789/0] mon.foo
  58   1: [v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0] mon.bar
  59   2: [v2:10.0.0.12:3300/0,v1:10.0.0.12:6789/0] mon.baz
  60
  61 The bracketed list or vector of addresses means that the same daemon can be
  62 reached on multiple ports (and protocols).  Any client or other daemon
  63 connecting to that daemon will use the v2 protocol (listed first) if
  64 possible; otherwise it will back to the legacy v1 protocol.  Legacy
  65 clients will only see the v1 addresses and will continue to connect as
  66 they did before, with the v1 protocol.
  67
  68 Starting in Nautilus, the ``mon_host`` configuration option and ``-m
  69 <mon-host>`` command line options support the same bracketed address
  70 vector syntax.
  71
  72
  73 Bind configuration options
  74 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  75
  76 Two new configuration options control whether the v1 and/or v2
  77 protocol is used:
  78
  79   * ``ms_bind_msgr1`` [default: true] controls whether a daemon binds
  80     to a port speaking the v1 protocol
  81   * ``ms_bind_msgr2`` [default: true] controls whether a daemon binds
  82     to a port speaking the v2 protocol
  83
  84 Similarly, two options control whether IPv4 and IPv6 addresses are used:
  85
  86   * ``ms_bind_ipv4`` [default: true] controls whether a daemon binds
  87     to an IPv4 address
  88   * ``ms_bind_ipv6`` [default: false] controls whether a daemon binds
  89     to an IPv6 address
  90
  91 .. note: The ability to bind to multiple ports has paved the way for
  92    dual-stack IPv4 and IPv6 support.  That said, dual-stack support is
  93    not yet tested as of Nautilus v14.2.0 and likely needs some
  94    additional code changes to work correctly.
  95
  96 Connection modes
  97 ----------------
  98
  99 The v2 protocol supports two connection modes:
 100
 101 * *crc* mode provides:
 102
 103   - a strong initial authentication when the connection is established
 104     (with cephx, mutual authentication of both parties with protection
 105     from a man-in-the-middle or eavesdropper), and
 106   - a crc32c integrity check to protect against bit flips due to flaky
 107     hardware or cosmic rays
 108
 109   *crc* mode does *not* provide:
 110
 111   - secrecy (an eavesdropper on the network can see all
 112     post-authentication traffic as it goes by) or
 113   - protection from a malicious man-in-the-middle (who can deliberate
 114     modify traffic as it goes by, as long as they are careful to
 115     adjust the crc32c values to match)
 116
 117 * *secure* mode provides:
 118
 119   - a strong initial authentication when the connection is established
 120     (with cephx, mutual authentication of both parties with protection
 121     from a man-in-the-middle or eavesdropper), and
 122   - full encryption of all post-authentication traffic, including a
 123     cryptographic integrity check.
 124
 125   In Nautilus, secure mode uses the `AES-GCM
 126   <https://en.wikipedia.org/wiki/Galois/Counter_Mode>`_ stream cipher,
 127   which is generally very fast on modern processors (e.g., faster than
 128   a SHA-256 cryptographic hash).
 129
 130 Connection mode configuration options
 131 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 132
 133 For most connections, there are options that control which modes are used:
 134
 135 * ``ms_cluster_mode`` is the connection mode (or permitted modes) used
 136   for intra-cluster communication between Ceph daemons.  If multiple
 137   modes are listed, the modes listed first are preferred.
 138 * ``ms_service_mode`` is a list of permitted modes for clients to use
 139   when connecting to the cluster.
 140 * ``ms_client_mode`` is a list of connection modes, in order of
 141   preference, for clients to use (or allow) when talking to a Ceph
 142   cluster.
 143
 144 There are a parallel set of options that apply specifically to
 145 monitors, allowing administrators to set different (usually more
 146 secure) requirements on communication with the monitors.
 147
 148 * ``ms_mon_cluster_mode`` is the connection mode (or permitted modes)
 149   to use between monitors.
 150 * ``ms_mon_service_mode`` is a list of permitted modes for clients or
 151   other Ceph daemons to use when connecting to monitors.
 152 * ``ms_mon_client_mode`` is a list of connection modes, in order of
 153   preference, for clients or non-monitor daemons to use when
 154   connecting to monitors.
 155
 156
 157 Transitioning from v1-only to v2-plus-v1
 158 ----------------------------------------
 159
 160 By default, ``ms_bind_msgr2`` is true starting with Nautilus 14.2.z.
 161 However, until the monitors start using v2, only limited services will
 162 start advertising v2 addresses.
 163
 164 For most users, the monitors are binding to the default legacy port ``6789`` for the v1 protocol.  When this is the case, enabling v2 is as simple as::
 165
 166   ceph mon enable-msgr2
 167
 168 If the monitors are bound to non-standard ports, you will need to
 169 specify an additional port for v2 explicitly.  For example, if your
 170 monitor ``mon.a`` binds to ``1.2.3.4:1111``, and you want to add v2 on
 171 port ``1112``,::
 172
 173   ceph mon set-addrs a [v2:1.2.3.4:1112,v1:1.2.3.4:1111]
 174
 175 Once the monitors bind to v2, each daemon will start advertising a v2
 176 address when it is next restarted.
 177
 178
 179 .. _msgr2_ceph_conf:
 180
 181 Updating ceph.conf and mon_host
 182 -------------------------------
 183
 184 Prior to Nautilus, a CLI user or daemon will normally discover the
 185 monitors via the ``mon_host`` option in ``/etc/ceph/ceph.conf``.  The
 186 syntax for this option has expanded starting with Nautilus to allow
 187 support the new bracketed list format.  For example, an old line
 188 like::
 189
 190   mon_host = 10.0.0.1:6789,10.0.0.2:6789,10.0.0.3:6789
 191
 192 Can be changed to::
 193
 194   mon_host = [v2:10.0.0.1:3300/0,v1:10.0.0.1:6789/0],[v2:10.0.0.2:3300/0,v1:10.0.0.2:6789/0],[v2:10.0.0.3:3300/0,v1:10.0.0.3:6789/0]
 195
 196 However, when default ports are used (``3300`` and ``6789``), they can
 197 be omitted::
 198
 199   mon_host = 10.0.0.1,10.0.0.2,10.0.0.3
 200
 201 Once v2 has been enabled on the monitors, ``ceph.conf`` may need to be
 202 updated to either specify no ports (this is usually simplest), or
 203 explicitly specify both the v2 and v1 addresses.  Note, however, that
 204 the new bracketed syntax is only understood by Nautilus and later, so
 205 do not make that change on hosts that have not yet had their ceph
 206 packages upgraded.
 207
 208 When you are updating ``ceph.conf``, note the new ``ceph config
 209 generate-minimal-conf`` command (which generates a barebones config
 210 file with just enough information to reach the monitors) and the
 211 ``ceph config assimilate-conf`` (which moves config file options into
 212 the monitors' configuration database) may be helpful.  For example,::
 213
 214   # ceph config assimilate-conf < /etc/ceph/ceph.conf
 215   # ceph config generate-minimal-config > /etc/ceph/ceph.conf.new
 216   # cat /etc/ceph/ceph.conf.new
 217   # minimal ceph.conf for 0e5a806b-0ce5-4bc6-b949-aa6f68f5c2a3
 218   [global]
 219           fsid = 0e5a806b-0ce5-4bc6-b949-aa6f68f5c2a3
 220           mon_host = [v2:10.0.0.1:3300/0,v1:10.0.0.1:6789/0]
 221   # mv /etc/ceph/ceph.conf.new /etc/ceph/ceph.conf
 222
 223 Protocol
 224 --------
 225
 226 For a detailed description of the v2 wire protocol, see :ref:`msgr2-protocol`.