6 This is a revision of the legacy Ceph on-wire protocol that was
7 implemented by the SimpleMessenger. It addresses performance and
13 This protocol revision has several goals relative to the original protocol:
15 * *Flexible handshaking*. The original protocol did not have a
16 sufficiently flexible protocol negotiation that allows for features
17 that were not required.
18 * *Encryption*. We will incorporate encryption over the wire.
19 * *Performance*. We would like to provide for protocol features
20 (e.g., padding) that keep computation and memory copies out of the
21 fast path where possible.
22 * *Signing*. We will allow for traffic to be signed (but not
23 necessarily encrypted). This may not be implemented in the initial version.
28 * *client* (C): the party initiating a (TCP) connection
29 * *server* (S): the party accepting a (TCP) connection
30 * *connection*: an instance of a (TCP) connection between two processes.
31 * *entity*: a ceph entity instantiation, e.g. 'osd.0'. each entity
32 has one or more unique entity_addr_t's by virtue of the 'nonce'
33 field, which is typically a pid or random value.
34 * *session*: a stateful session between two entities in which message
35 exchange is ordered and lossless. A session might span multiple
36 connections if there is an interruption (TCP connection disconnect).
37 * *frame*: a discrete message sent between the peers. Each frame
38 consists of a tag (type code), payload, and (if signing
39 or encryption is enabled) some other fields. See below for the
41 * *tag*: a type code associated with a frame. The tag
42 determines the structure of the payload.
47 A connection has four distinct phases:
50 #. authentication frame exchange
51 #. message flow handshake frame exchange
52 #. message frame exchange
57 Both the client and server, upon connecting, send a banner::
59 "ceph %x %x\n", protocol_features_suppored, protocol_features_required
61 The protocol features are a new, distinct namespace. Initially no
62 features are defined or required, so this will be "ceph 0 0\n".
64 If the remote party advertises required features we don't support, we
68 .. ditaa:: +---------+ +--------+
70 +---------+ +--------+
82 All further data sent or received is contained by a frame. Each frame has
87 frame_header_checksum (le32)
89 [payload padding -- only present after stream auth phase]
90 [signature -- only present after stream auth phase]
93 * The frame_header_checksum is over just the frame_len and tag values (8 bytes).
95 * frame_len includes everything after the frame_len le32 up to the end of the
96 frame (all payloads, signatures, and padding).
98 * The payload format and length is determined by the tag.
100 * The signature portion is only present if the authentication phase
101 has completed (TAG_AUTH_DONE has been sent) and signatures are
107 * TAG_HELLO: client->server and server->client::
110 entity_addr_t peer_socket_address
112 - We immediately share our entity type and the address of the peer (which can be useful
113 for detecting our effective IP address, especially in the presence of NAT).
119 * TAG_AUTH_REQUEST: client->server::
121 __le32 method; // CEPH_AUTH_{NONE, CEPHX, ...}
122 __le32 num_preferred_modes;
123 list<__le32> mode // CEPH_CON_MODE_*
124 method specific payload
126 * TAG_AUTH_BAD_METHOD server -> client: reject client-selected auth method::
129 __le32 negative error result code
131 list<__le32> allowed_methods // CEPH_AUTH_{NONE, CEPHX, ...}
133 list<__le32> allowed_modes // CEPH_CON_MODE_*
135 - Returns the attempted auth method, and error code (-EOPNOTSUPP if
136 the method is unsupported), and the list of allowed authentication
139 * TAG_AUTH_REPLY_MORE: server->client::
142 method specific payload
144 * TAG_AUTH_REQUEST_MORE: client->server::
147 method specific payload
149 * TAG_AUTH_DONE: (server->client)::
152 __le32 connection mode // CEPH_CON_MODE_*
153 method specific payload
155 - The server is the one to decide authentication has completed and what
156 the final connection mode will be.
159 Example of authentication phase interaction when the client uses an
160 allowed authentication method:
162 .. ditaa:: +---------+ +--------+
163 | Client | | Server |
164 +---------+ +--------+
176 Example of authentication phase interaction when the client uses a forbidden
177 authentication method as the first attempt:
179 .. ditaa:: +---------+ +--------+
180 | Client | | Server |
181 +---------+ +--------+
198 Post-auth frame format
199 ----------------------
201 The frame format is fixed (see above), but can take three different
202 forms, depending on the AUTH_DONE flags:
204 * If neither FLAG_SIGNED or FLAG_ENCRYPTED is specified, things are simple::
209 payload_padding (out to auth block_size)
211 - The padding is some number of bytes < the auth block_size that
212 brings the total length of the payload + payload_padding to a
213 multiple of block_size. It does not include the frame_len or tag. Padding
214 content can be zeros or (better) random bytes.
216 * If FLAG_SIGNED has been specified::
221 payload_padding (out to auth block_size)
222 signature (sig_size bytes)
224 Here the padding just makes life easier for the signature. It can be
225 random data to add additional confounder. Note also that the
226 signature input must include some state from the session key and the
229 * If FLAG_ENCRYPTED has been specified::
235 payload_padding (out to auth block_size)
238 Note that the padding ensures that the total frame is a multiple of
239 the auth method's block_size so that the message can be sent out over
240 the wire without waiting for the next frame in the stream.
243 Message flow handshake
244 ----------------------
246 In this phase the peers identify each other and (if desired) reconnect to
247 an established session.
249 * TAG_CLIENT_IDENT (client->server): identify ourselves::
252 entity_addrvec_t*num_addrs entity addrs
253 entity_addr_t target entity addr
254 __le64 gid (numeric part of osd.0, client.123456, ...)
256 __le64 features supported (CEPH_FEATURE_* bitmask)
257 __le64 features required (CEPH_FEATURE_* bitmask)
258 __le64 flags (CEPH_MSG_CONNECT_* bitmask)
261 - client will send first, server will reply with same. if this is a
262 new session, the client and server can proceed to the message exchange.
263 - the target addr is who the client is trying to connect *to*, so
264 that the server side can close the connection if the client is
265 talking to the wrong daemon.
266 - type.gid (entity_name_t) is set here, by combinging the type shared in the hello
267 frame with the gid here. this means we don't need it
268 in the header of every message. it also means that we can't send
269 messages "from" other entity_name_t's. the current
270 implementations set this at the top of _send_message etc so this
271 shouldn't break any existing functionality. implementation will
272 likely want to mask this against what the authenticated credential
274 - cookie is the client coookie used to identify a session, and can be used
275 to reconnect to an existing session.
276 - we've dropped the 'protocol_version' field from msgr1
278 * TAG_IDENT_MISSING_FEATURES (server->client): complain about a TAG_IDENT
279 with too few features::
281 __le64 features we require that the peer didn't advertise
283 * TAG_SERVER_IDENT (server->client): accept client ident and identify server::
286 entity_addrvec_t*num_addrs entity addrs
287 __le64 gid (numeric part of osd.0, client.123456, ...)
289 __le64 features supported (CEPH_FEATURE_* bitmask)
290 __le64 features required (CEPH_FEATURE_* bitmask)
291 __le64 flags (CEPH_MSG_CONNECT_* bitmask)
294 - The server cookie can be used by the client if it is later disconnected
295 and wants to reconnect and resume the session.
297 * TAG_RECONNECT (client->server): reconnect to an established session::
300 entity_addr_t * num_addrs
305 __le64 msg_seq (the last msg seq received)
307 * TAG_RECONNECT_OK (server->client): acknowledge a reconnect attempt::
309 __le64 msg_seq (last msg seq received)
311 - once the client receives this, the client can proceed to message exchange.
312 - once the server sends this, the server can proceed to message exchange.
314 * TAG_RECONNECT_RETRY_SESSION (server only): fail reconnect due to stale connect_seq
316 * TAG_RECONNECT_RETRY_GLOBAL (server only): fail reconnect due to stale global_seq
318 * TAG_RECONNECT_WAIT (server only): fail reconnect due to connect race.
320 - Indicates that the server is already connecting to the client, and
321 that direction should win the race. The client should wait for that
322 connection to complete.
324 * TAG_RESET_SESSION (server only): ask client to reset session::
328 - full flag indicates whether peer should do a full reset, i.e., drop
332 Example of failure scenarios:
334 * First client's client_ident message is lost, and then client reconnects.
336 .. ditaa:: +---------+ +--------+
337 | Client | | Server |
338 +---------+ +--------+
340 c_cookie(a) | client_ident(a) |
344 |-------------------->|
345 |<--------------------|
346 | server_ident(b) | s_cookie(b)
348 | session established |
352 * Server's server_ident message is lost, and then client reconnects.
354 .. ditaa:: +---------+ +--------+
355 | Client | | Server |
356 +---------+ +--------+
358 c_cookie(a) | client_ident(a) |
359 |-------------------->|
361 | server_ident(b) | s_cookie(b)
365 |-------------------->|
366 |<--------------------|
367 | server_ident(c) | s_cookie(c)
369 | session established |
373 * Server's server_ident message is lost, and then server reconnects.
375 .. ditaa:: +---------+ +--------+
376 | Client | | Server |
377 +---------+ +--------+
379 c_cookie(a) | client_ident(a) |
380 |-------------------->|
382 | server_ident(b) | s_cookie(b)
386 |<--------------------|
387 |-------------------->|
390 | client_ident(a) | c_cookie(a)
391 |<--------------------|
392 |-------------------->|
393 s_cookie(c) | server_ident(c) |
397 * Connection failure after session is established, and then client reconnects.
399 .. ditaa:: +---------+ +--------+
400 | Client | | Server |
401 +---------+ +--------+
403 c_cookie(a) | session established | s_cookie(b)
404 |<------------------->|
408 |-------------------->|
409 |<--------------------|
414 * Connection failure after session is established because server reset,
415 and then client reconnects.
417 .. ditaa:: +---------+ +--------+
418 | Client | | Server |
419 +---------+ +--------+
421 c_cookie(a) | session established | s_cookie(b)
422 |<------------------->|
423 | X------------| reset
426 |-------------------->|
427 |<--------------------|
428 | reset_session(RC*) |
430 c_cookie(c) | client_ident(c) |
431 |-------------------->|
432 |<--------------------|
433 | server_ident(d) | s_cookie(d)
436 RC* means that the reset session full flag depends on the policy.resetcheck
440 * Connection failure after session is established because client reset,
441 and then client reconnects.
443 .. ditaa:: +---------+ +--------+
444 | Client | | Server |
445 +---------+ +--------+
447 c_cookie(a) | session established | s_cookie(b)
448 |<------------------->|
449 reset | X------------|
451 c_cookie(c) | client_ident(c) |
452 |-------------------->|
453 |<--------------------| reset if policy.resetcheck
454 | server_ident(d) | s_cookie(d)
461 Once a session is established, we can exchange messages.
463 * TAG_MSG: a message::
471 - The ceph_msg_header2 is modified from ceph_msg_header:
472 * include an ack_seq. This avoids the need for a TAG_ACK
473 message most of the time.
474 * remove the src field, which we now get from the message flow
475 handshake (TAG_IDENT).
476 * specifies the data_pre_padding length, which can be used to
477 adjust the alignment of the data payload. (NOTE: is this is
480 * TAG_ACK: acknowledge receipt of message(s)::
484 - This is only used for stateful sessions.
486 * TAG_KEEPALIVE2: check for connection liveness::
490 - Time stamp is local to sender.
492 * TAG_KEEPALIVE2_ACK: reply to a keepalive2::
496 - Time stamp is from the TAG_KEEPALIVE2 we are responding to.
498 * TAG_CLOSE: terminate a connection
500 Indicates that a connection should be terminated. This is equivalent
501 to a hangup or reset (i.e., should trigger ms_handle_reset). It
502 isn't strictly necessary or useful as we could just disconnect the
506 Example of protocol interaction (WIP)
507 _____________________________________
510 .. ditaa:: +---------+ +--------+
511 | Client | | Server |
512 +---------+ +--------+
521 |------------------>|
523 |------------------>|
524 |<------------------|
528 |------------------>|
529 |<------------------|
533 |------------------>|
534 |<------------------|