docs/specs/vhost-user.txt

   1 Vhost-user Protocol
   2 ===================
   3
   4 Copyright (c) 2014 Virtual Open Systems Sarl.
   5
   6 This work is licensed under the terms of the GNU GPL, version 2 or later.
   7 See the COPYING file in the top-level directory.
   8 ===================
   9
  10 This protocol is aiming to complement the ioctl interface used to control the
  11 vhost implementation in the Linux kernel. It implements the control plane needed
  12 to establish virtqueue sharing with a user space process on the same host. It
  13 uses communication over a Unix domain socket to share file descriptors in the
  14 ancillary data of the message.
  15
  16 The protocol defines 2 sides of the communication, master and slave. Master is
  17 the application that shares its virtqueues, in our case QEMU. Slave is the
  18 consumer of the virtqueues.
  19
  20 In the current implementation QEMU is the Master, and the Slave is intended to
  21 be a software Ethernet switch running in user space, such as Snabbswitch.
  22
  23 Master and slave can be either a client (i.e. connecting) or server (listening)
  24 in the socket communication.
  25
  26 Message Specification
  27 ---------------------
  28
  29 Note that all numbers are in the machine native byte order. A vhost-user message
  30 consists of 3 header fields and a payload:
  31
  32 ------------------------------------
  33 | request | flags | size | payload |
  34 ------------------------------------
  35
  36  * Request: 32-bit type of the request
  37  * Flags: 32-bit bit field:
  38    - Lower 2 bits are the version (currently 0x01)
  39    - Bit 2 is the reply flag - needs to be sent on each reply from the slave
  40  * Size - 32-bit size of the payload
  41
  42
  43 Depending on the request type, payload can be:
  44
  45  * A single 64-bit integer
  46    -------
  47    | u64 |
  48    -------
  49
  50    u64: a 64-bit unsigned integer
  51
  52  * A vring state description
  53    ---------------
  54   | index | num |
  55   ---------------
  56
  57    Index: a 32-bit index
  58    Num: a 32-bit number
  59
  60  * A vring address description
  61    --------------------------------------------------------------
  62    | index | flags | size | descriptor | used | available | log |
  63    --------------------------------------------------------------
  64
  65    Index: a 32-bit vring index
  66    Flags: a 32-bit vring flags
  67    Descriptor: a 64-bit user address of the vring descriptor table
  68    Used: a 64-bit user address of the vring used ring
  69    Available: a 64-bit user address of the vring available ring
  70    Log: a 64-bit guest address for logging
  71
  72  * Memory regions description
  73    ---------------------------------------------------
  74    | num regions | padding | region0 | ... | region7 |
  75    ---------------------------------------------------
  76
  77    Num regions: a 32-bit number of regions
  78    Padding: 32-bit
  79
  80    A region is:
  81    -----------------------------------------------------
  82    | guest address | size | user address | mmap offset |
  83    -----------------------------------------------------
  84
  85    Guest address: a 64-bit guest address of the region
  86    Size: a 64-bit size
  87    User address: a 64-bit user address
  88    mmap offset: 64-bit offset where region starts in the mapped memory
  89
  90 In QEMU the vhost-user message is implemented with the following struct:
  91
  92 typedef struct VhostUserMsg {
  93     VhostUserRequest request;
  94     uint32_t flags;
  95     uint32_t size;
  96     union {
  97         uint64_t u64;
  98         struct vhost_vring_state state;
  99         struct vhost_vring_addr addr;
 100         VhostUserMemory memory;
 101     };
 102 } QEMU_PACKED VhostUserMsg;
 103
 104 Communication
 105 -------------
 106
 107 The protocol for vhost-user is based on the existing implementation of vhost
 108 for the Linux Kernel. Most messages that can be sent via the Unix domain socket
 109 implementing vhost-user have an equivalent ioctl to the kernel implementation.
 110
 111 The communication consists of master sending message requests and slave sending
 112 message replies. Most of the requests don't require replies. Here is a list of
 113 the ones that do:
 114
 115  * VHOST_GET_FEATURES
 116  * VHOST_GET_VRING_BASE
 117
 118 There are several messages that the master sends with file descriptors passed
 119 in the ancillary data:
 120
 121  * VHOST_SET_MEM_TABLE
 122  * VHOST_SET_LOG_FD
 123  * VHOST_SET_VRING_KICK
 124  * VHOST_SET_VRING_CALL
 125  * VHOST_SET_VRING_ERR
 126
 127 If Master is unable to send the full message or receives a wrong reply it will
 128 close the connection. An optional reconnection mechanism can be implemented.
 129
 130 Message types
 131 -------------
 132
 133  * VHOST_USER_GET_FEATURES
 134
 135       Id: 1
 136       Equivalent ioctl: VHOST_GET_FEATURES
 137       Master payload: N/A
 138       Slave payload: u64
 139
 140       Get from the underlying vhost implementation the features bitmask.
 141
 142  * VHOST_USER_SET_FEATURES
 143
 144       Id: 2
 145       Ioctl: VHOST_SET_FEATURES
 146       Master payload: u64
 147
 148       Enable features in the underlying vhost implementation using a bitmask.
 149
 150  * VHOST_USER_SET_OWNER
 151
 152       Id: 3
 153       Equivalent ioctl: VHOST_SET_OWNER
 154       Master payload: N/A
 155
 156       Issued when a new connection is established. It sets the current Master
 157       as an owner of the session. This can be used on the Slave as a
 158       "session start" flag.
 159
 160  * VHOST_USER_RESET_OWNER
 161
 162       Id: 4
 163       Equivalent ioctl: VHOST_RESET_OWNER
 164       Master payload: N/A
 165
 166       Issued when a new connection is about to be closed. The Master will no
 167       longer own this connection (and will usually close it).
 168
 169  * VHOST_USER_SET_MEM_TABLE
 170
 171       Id: 5
 172       Equivalent ioctl: VHOST_SET_MEM_TABLE
 173       Master payload: memory regions description
 174
 175       Sets the memory map regions on the slave so it can translate the vring
 176       addresses. In the ancillary data there is an array of file descriptors
 177       for each memory mapped region. The size and ordering of the fds matches
 178       the number and ordering of memory regions.
 179
 180  * VHOST_USER_SET_LOG_BASE
 181
 182       Id: 6
 183       Equivalent ioctl: VHOST_SET_LOG_BASE
 184       Master payload: u64
 185
 186       Sets the logging base address.
 187
 188  * VHOST_USER_SET_LOG_FD
 189
 190       Id: 7
 191       Equivalent ioctl: VHOST_SET_LOG_FD
 192       Master payload: N/A
 193
 194       Sets the logging file descriptor, which is passed as ancillary data.
 195
 196  * VHOST_USER_SET_VRING_NUM
 197
 198       Id: 8
 199       Equivalent ioctl: VHOST_SET_VRING_NUM
 200       Master payload: vring state description
 201
 202       Sets the number of vrings for this owner.
 203
 204  * VHOST_USER_SET_VRING_ADDR
 205
 206       Id: 9
 207       Equivalent ioctl: VHOST_SET_VRING_ADDR
 208       Master payload: vring address description
 209       Slave payload: N/A
 210
 211       Sets the addresses of the different aspects of the vring.
 212
 213  * VHOST_USER_SET_VRING_BASE
 214
 215       Id: 10
 216       Equivalent ioctl: VHOST_SET_VRING_BASE
 217       Master payload: vring state description
 218
 219       Sets the base offset in the available vring.
 220
 221  * VHOST_USER_GET_VRING_BASE
 222
 223       Id: 11
 224       Equivalent ioctl: VHOST_USER_GET_VRING_BASE
 225       Master payload: vring state description
 226       Slave payload: vring state description
 227
 228       Get the available vring base offset.
 229
 230  * VHOST_USER_SET_VRING_KICK
 231
 232       Id: 12
 233       Equivalent ioctl: VHOST_SET_VRING_KICK
 234       Master payload: u64
 235
 236       Set the event file descriptor for adding buffers to the vring. It
 237       is passed in the ancillary data.
 238       Bits (0-7) of the payload contain the vring index. Bit 8 is the
 239       invalid FD flag. This flag is set when there is no file descriptor
 240       in the ancillary data. This signals that polling should be used
 241       instead of waiting for a kick.
 242
 243  * VHOST_USER_SET_VRING_CALL
 244
 245       Id: 13
 246       Equivalent ioctl: VHOST_SET_VRING_CALL
 247       Master payload: u64
 248
 249       Set the event file descriptor to signal when buffers are used. It
 250       is passed in the ancillary data.
 251       Bits (0-7) of the payload contain the vring index. Bit 8 is the
 252       invalid FD flag. This flag is set when there is no file descriptor
 253       in the ancillary data. This signals that polling will be used
 254       instead of waiting for the call.
 255
 256  * VHOST_USER_SET_VRING_ERR
 257
 258       Id: 14
 259       Equivalent ioctl: VHOST_SET_VRING_ERR
 260       Master payload: u64
 261
 262       Set the event file descriptor to signal when error occurs. It
 263       is passed in the ancillary data.
 264       Bits (0-7) of the payload contain the vring index. Bit 8 is the
 265       invalid FD flag. This flag is set when there is no file descriptor
 266       in the ancillary data.