9 The messenger v2 protocol, or msgr2, is the second major revision on
10 Ceph's on-wire protocol. It brings with it several key features:
12 * A *secure* mode that encrypts all data passing over the network
13 * Improved encapsulation of authentication payloads, enabling future
14 integration of new authentication modes like Kerberos
15 * Improved earlier feature advertisement and negotiation, enabling
16 future protocol revisions
18 Ceph daemons can now bind to multiple ports, allowing both legacy Ceph
19 clients and new v2-capable clients to connect to the same cluster.
21 By default, monitors now bind to the new IANA-assigned port ``3300``
22 (ce4h or 0xce4) for the new v2 protocol, while also binding to the
23 old default port ``6789`` for the legacy v1 protocol.
30 Prior to Nautilus, all network addresses were rendered like
31 ``1.2.3.4:567/89012`` where there was an IP address, a port, and a
32 nonce to uniquely identify a client or daemon on the network.
33 Starting with Nautilus, we now have three different address types:
35 * **v2**: ``v2:1.2.3.4:578/89012`` identifies a daemon binding to a
36 port speaking the new v2 protocol
37 * **v1**: ``v1:1.2.3.4:578/89012`` identifies a daemon binding to a
38 port speaking the legacy v1 protocol. Any address that was
39 previously shown with any prefix is now shown as a ``v1:`` address.
40 * **TYPE_ANY** ``any:1.2.3.4:578/89012`` identifies a client that can
41 speak either version of the protocol. Prior to nautilus, clients would appear as
42 ``1.2.3.4:0/123456``, where the port of 0 indicates they are clients
43 and do not accept incoming connections. Starting with Nautilus,
44 these clients are now internally represented by a **TYPE_ANY**
45 address, and still shown with no prefix, because they may
46 connect to daemons using the v2 or v1 protocol, depending on what
47 protocol(s) the daemons are using.
49 Because daemons now bind to multiple ports, they are now described by
50 a vector of addresses instead of a single address. For example,
51 dumping the monitor map on a Nautilus cluster now includes lines
55 fsid 50fcf227-be32-4bcb-8b41-34ca8370bd16
56 last_changed 2019-02-25 11:10:46.700821
57 created 2019-02-25 11:10:46.700821
58 min_mon_release 14 (nautilus)
59 0: [v2:10.0.0.10:3300/0,v1:10.0.0.10:6789/0] mon.foo
60 1: [v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0] mon.bar
61 2: [v2:10.0.0.12:3300/0,v1:10.0.0.12:6789/0] mon.baz
63 The bracketed list or vector of addresses means that the same daemon can be
64 reached on multiple ports (and protocols). Any client or other daemon
65 connecting to that daemon will use the v2 protocol (listed first) if
66 possible; otherwise it will back to the legacy v1 protocol. Legacy
67 clients will only see the v1 addresses and will continue to connect as
68 they did before, with the v1 protocol.
70 Starting in Nautilus, the ``mon_host`` configuration option and ``-m
71 <mon-host>`` command line options support the same bracketed address
75 Bind configuration options
76 ^^^^^^^^^^^^^^^^^^^^^^^^^^
78 Two new configuration options control whether the v1 and/or v2
81 * :confval:`ms_bind_msgr1` [default: true] controls whether a daemon binds
82 to a port speaking the v1 protocol
83 * :confval:`ms_bind_msgr2` [default: true] controls whether a daemon binds
84 to a port speaking the v2 protocol
86 Similarly, two options control whether IPv4 and IPv6 addresses are used:
88 * :confval:`ms_bind_ipv4` [default: true] controls whether a daemon binds
90 * :confval:`ms_bind_ipv6` [default: false] controls whether a daemon binds
93 .. note:: The ability to bind to multiple ports has paved the way for
94 dual-stack IPv4 and IPv6 support. That said, dual-stack support is
95 not yet tested as of Nautilus v14.2.0 and likely needs some
96 additional code changes to work correctly.
101 The v2 protocol supports two connection modes:
103 * *crc* mode provides:
105 - a strong initial authentication when the connection is established
106 (with cephx, mutual authentication of both parties with protection
107 from a man-in-the-middle or eavesdropper), and
108 - a crc32c integrity check to protect against bit flips due to flaky
109 hardware or cosmic rays
111 *crc* mode does *not* provide:
113 - secrecy (an eavesdropper on the network can see all
114 post-authentication traffic as it goes by) or
115 - protection from a malicious man-in-the-middle (who can deliberate
116 modify traffic as it goes by, as long as they are careful to
117 adjust the crc32c values to match)
119 * *secure* mode provides:
121 - a strong initial authentication when the connection is established
122 (with cephx, mutual authentication of both parties with protection
123 from a man-in-the-middle or eavesdropper), and
124 - full encryption of all post-authentication traffic, including a
125 cryptographic integrity check.
127 In Nautilus, secure mode uses the `AES-GCM
128 <https://en.wikipedia.org/wiki/Galois/Counter_Mode>`_ stream cipher,
129 which is generally very fast on modern processors (e.g., faster than
130 a SHA-256 cryptographic hash).
132 Connection mode configuration options
133 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
135 For most connections, there are options that control which modes are used:
137 .. confval:: ms_cluster_mode
138 .. confval:: ms_service_mode
139 .. confval:: ms_client_mode
141 There are a parallel set of options that apply specifically to
142 monitors, allowing administrators to set different (usually more
143 secure) requirements on communication with the monitors.
145 .. confval:: ms_mon_cluster_mode
146 .. confval:: ms_mon_service_mode
147 .. confval:: ms_mon_client_mode
153 The v2 protocol supports two compression modes:
155 * *force* mode provides:
157 - In multi-availability zones deployment, compressing replication messages between OSDs saves latency.
158 - In the public cloud, inter-AZ communications are expensive. Thus, minimizing message
159 size reduces network costs to cloud provider.
160 - When using instance storage on AWS (probably other public clouds as well) the instances with NVMe
161 provide low network bandwidth relative to the device bandwidth.
162 In this case, NW compression can improve the overall performance since this is clearly
165 * *none* mode provides:
167 - messages are transmitted without compression.
170 Compression mode configuration options
171 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
173 For all connections, there is an option that controls compression usage in secure mode
175 .. confval:: ms_compress_secure
177 There is a parallel set of options that apply specifically to OSDs,
178 allowing administrators to set different requirements on communication between OSDs.
180 .. confval:: ms_osd_compress_mode
181 .. confval:: ms_osd_compress_min_size
182 .. confval:: ms_osd_compression_algorithm
184 Transitioning from v1-only to v2-plus-v1
185 ----------------------------------------
187 By default, ``ms_bind_msgr2`` is true starting with Nautilus 14.2.z.
188 However, until the monitors start using v2, only limited services will
189 start advertising v2 addresses.
191 For most users, the monitors are binding to the default legacy port ``6789`` for the v1 protocol. When this is the case, enabling v2 is as simple as::
193 ceph mon enable-msgr2
195 If the monitors are bound to non-standard ports, you will need to
196 specify an additional port for v2 explicitly. For example, if your
197 monitor ``mon.a`` binds to ``1.2.3.4:1111``, and you want to add v2 on
200 ceph mon set-addrs a [v2:1.2.3.4:1112,v1:1.2.3.4:1111]
202 Once the monitors bind to v2, each daemon will start advertising a v2
203 address when it is next restarted.
208 Updating ceph.conf and mon_host
209 -------------------------------
211 Prior to Nautilus, a CLI user or daemon will normally discover the
212 monitors via the ``mon_host`` option in ``/etc/ceph/ceph.conf``. The
213 syntax for this option has expanded starting with Nautilus to allow
214 support the new bracketed list format. For example, an old line
217 mon_host = 10.0.0.1:6789,10.0.0.2:6789,10.0.0.3:6789
221 mon_host = [v2:10.0.0.1:3300/0,v1:10.0.0.1:6789/0],[v2:10.0.0.2:3300/0,v1:10.0.0.2:6789/0],[v2:10.0.0.3:3300/0,v1:10.0.0.3:6789/0]
223 However, when default ports are used (``3300`` and ``6789``), they can
226 mon_host = 10.0.0.1,10.0.0.2,10.0.0.3
228 Once v2 has been enabled on the monitors, ``ceph.conf`` may need to be
229 updated to either specify no ports (this is usually simplest), or
230 explicitly specify both the v2 and v1 addresses. Note, however, that
231 the new bracketed syntax is only understood by Nautilus and later, so
232 do not make that change on hosts that have not yet had their ceph
235 When you are updating ``ceph.conf``, note the new ``ceph config
236 generate-minimal-conf`` command (which generates a barebones config
237 file with just enough information to reach the monitors) and the
238 ``ceph config assimilate-conf`` (which moves config file options into
239 the monitors' configuration database) may be helpful. For example,::
241 # ceph config assimilate-conf < /etc/ceph/ceph.conf
242 # ceph config generate-minimal-config > /etc/ceph/ceph.conf.new
243 # cat /etc/ceph/ceph.conf.new
244 # minimal ceph.conf for 0e5a806b-0ce5-4bc6-b949-aa6f68f5c2a3
246 fsid = 0e5a806b-0ce5-4bc6-b949-aa6f68f5c2a3
247 mon_host = [v2:10.0.0.1:3300/0,v1:10.0.0.1:6789/0]
248 # mv /etc/ceph/ceph.conf.new /etc/ceph/ceph.conf
253 For a detailed description of the v2 wire protocol, see :ref:`msgr2-protocol`.