docs/specs/vhost-user.txt

   1 Vhost-user Protocol
   2 ===================
   3
   4 Copyright (c) 2014 Virtual Open Systems Sarl.
   5
   6 This work is licensed under the terms of the GNU GPL, version 2 or later.
   7 See the COPYING file in the top-level directory.
   8 ===================
   9
  10 This protocol is aiming to complement the ioctl interface used to control the
  11 vhost implementation in the Linux kernel. It implements the control plane needed
  12 to establish virtqueue sharing with a user space process on the same host. It
  13 uses communication over a Unix domain socket to share file descriptors in the
  14 ancillary data of the message.
  15
  16 The protocol defines 2 sides of the communication, master and slave. Master is
  17 the application that shares its virtqueues, in our case QEMU. Slave is the
  18 consumer of the virtqueues.
  19
  20 In the current implementation QEMU is the Master, and the Slave is intended to
  21 be a software Ethernet switch running in user space, such as Snabbswitch.
  22
  23 Master and slave can be either a client (i.e. connecting) or server (listening)
  24 in the socket communication.
  25
  26 Message Specification
  27 ---------------------
  28
  29 Note that all numbers are in the machine native byte order. A vhost-user message
  30 consists of 3 header fields and a payload:
  31
  32 ------------------------------------
  33 | request | flags | size | payload |
  34 ------------------------------------
  35
  36  * Request: 32-bit type of the request
  37  * Flags: 32-bit bit field:
  38    - Lower 2 bits are the version (currently 0x01)
  39    - Bit 2 is the reply flag - needs to be sent on each reply from the slave
  40  * Size - 32-bit size of the payload
  41
  42
  43 Depending on the request type, payload can be:
  44
  45  * A single 64-bit integer
  46    -------
  47    | u64 |
  48    -------
  49
  50    u64: a 64-bit unsigned integer
  51
  52  * A vring state description
  53    ---------------
  54   | index | num |
  55   ---------------
  56
  57    Index: a 32-bit index
  58    Num: a 32-bit number
  59
  60  * A vring address description
  61    --------------------------------------------------------------
  62    | index | flags | size | descriptor | used | available | log |
  63    --------------------------------------------------------------
  64
  65    Index: a 32-bit vring index
  66    Flags: a 32-bit vring flags
  67    Descriptor: a 64-bit user address of the vring descriptor table
  68    Used: a 64-bit user address of the vring used ring
  69    Available: a 64-bit user address of the vring available ring
  70    Log: a 64-bit guest address for logging
  71
  72  * Memory regions description
  73    ---------------------------------------------------
  74    | num regions | padding | region0 | ... | region7 |
  75    ---------------------------------------------------
  76
  77    Num regions: a 32-bit number of regions
  78    Padding: 32-bit
  79
  80    A region is:
  81    -----------------------------------------------------
  82    | guest address | size | user address | mmap offset |
  83    -----------------------------------------------------
  84
  85    Guest address: a 64-bit guest address of the region
  86    Size: a 64-bit size
  87    User address: a 64-bit user address
  88    mmmap offset: 64-bit offset where region starts in the mapped memory
  89
  90
  91 In QEMU the vhost-user message is implemented with the following struct:
  92
  93 typedef struct VhostUserMsg {
  94     VhostUserRequest request;
  95     uint32_t flags;
  96     uint32_t size;
  97     union {
  98         uint64_t u64;
  99         struct vhost_vring_state state;
 100         struct vhost_vring_addr addr;
 101         VhostUserMemory memory;
 102     };
 103 } QEMU_PACKED VhostUserMsg;
 104
 105 Communication
 106 -------------
 107
 108 The protocol for vhost-user is based on the existing implementation of vhost
 109 for the Linux Kernel. Most messages that can be sent via the Unix domain socket
 110 implementing vhost-user have an equivalent ioctl to the kernel implementation.
 111
 112 The communication consists of master sending message requests and slave sending
 113 message replies. Most of the requests don't require replies. Here is a list of
 114 the ones that do:
 115
 116  * VHOST_GET_FEATURES
 117  * VHOST_GET_VRING_BASE
 118
 119 There are several messages that the master sends with file descriptors passed
 120 in the ancillary data:
 121
 122  * VHOST_SET_MEM_TABLE
 123  * VHOST_SET_LOG_FD
 124  * VHOST_SET_VRING_KICK
 125  * VHOST_SET_VRING_CALL
 126  * VHOST_SET_VRING_ERR
 127
 128 If Master is unable to send the full message or receives a wrong reply it will
 129 close the connection. An optional reconnection mechanism can be implemented.
 130
 131 Message types
 132 -------------
 133
 134  * VHOST_USER_GET_FEATURES
 135
 136       Id: 1
 137       Equivalent ioctl: VHOST_GET_FEATURES
 138       Master payload: N/A
 139       Slave payload: u64
 140
 141       Get from the underlying vhost implementation the features bitmask.
 142
 143  * VHOST_USER_SET_FEATURES
 144
 145       Id: 2
 146       Ioctl: VHOST_SET_FEATURES
 147       Master payload: u64
 148
 149       Enable features in the underlying vhost implementation using a bitmask.
 150
 151  * VHOST_USER_SET_OWNER
 152
 153       Id: 3
 154       Equivalent ioctl: VHOST_SET_OWNER
 155       Master payload: N/A
 156
 157       Issued when a new connection is established. It sets the current Master
 158       as an owner of the session. This can be used on the Slave as a
 159       "session start" flag.
 160
 161  * VHOST_USER_RESET_OWNER
 162
 163       Id: 4
 164       Equivalent ioctl: VHOST_RESET_OWNER
 165       Master payload: N/A
 166
 167       Issued when a new connection is about to be closed. The Master will no
 168       longer own this connection (and will usually close it).
 169
 170  * VHOST_USER_SET_MEM_TABLE
 171
 172       Id: 5
 173       Equivalent ioctl: VHOST_SET_MEM_TABLE
 174       Master payload: memory regions description
 175
 176       Sets the memory map regions on the slave so it can translate the vring
 177       addresses. In the ancillary data there is an array of file descriptors
 178       for each memory mapped region. The size and ordering of the fds matches
 179       the number and ordering of memory regions.
 180
 181  * VHOST_USER_SET_LOG_BASE
 182
 183       Id: 6
 184       Equivalent ioctl: VHOST_SET_LOG_BASE
 185       Master payload: u64
 186
 187       Sets the logging base address.
 188
 189  * VHOST_USER_SET_LOG_FD
 190
 191       Id: 7
 192       Equivalent ioctl: VHOST_SET_LOG_FD
 193       Master payload: N/A
 194
 195       Sets the logging file descriptor, which is passed as ancillary data.
 196
 197  * VHOST_USER_SET_VRING_NUM
 198
 199       Id: 8
 200       Equivalent ioctl: VHOST_SET_VRING_NUM
 201       Master payload: vring state description
 202
 203       Sets the number of vrings for this owner.
 204
 205  * VHOST_USER_SET_VRING_ADDR
 206
 207       Id: 9
 208       Equivalent ioctl: VHOST_SET_VRING_ADDR
 209       Master payload: vring address description
 210       Slave payload: N/A
 211
 212       Sets the addresses of the different aspects of the vring.
 213
 214  * VHOST_USER_SET_VRING_BASE
 215
 216       Id: 10
 217       Equivalent ioctl: VHOST_SET_VRING_BASE
 218       Master payload: vring state description
 219
 220       Sets the base offset in the available vring.
 221
 222  * VHOST_USER_GET_VRING_BASE
 223
 224       Id: 11
 225       Equivalent ioctl: VHOST_USER_GET_VRING_BASE
 226       Master payload: vring state description
 227       Slave payload: vring state description
 228
 229       Get the available vring base offset.
 230
 231  * VHOST_USER_SET_VRING_KICK
 232
 233       Id: 12
 234       Equivalent ioctl: VHOST_SET_VRING_KICK
 235       Master payload: u64
 236
 237       Set the event file descriptor for adding buffers to the vring. It
 238       is passed in the ancillary data.
 239       Bits (0-7) of the payload contain the vring index. Bit 8 is the
 240       invalid FD flag. This flag is set when there is no file descriptor
 241       in the ancillary data. This signals that polling should be used
 242       instead of waiting for a kick.
 243
 244  * VHOST_USER_SET_VRING_CALL
 245
 246       Id: 13
 247       Equivalent ioctl: VHOST_SET_VRING_CALL
 248       Master payload: u64
 249
 250       Set the event file descriptor to signal when buffers are used. It
 251       is passed in the ancillary data.
 252       Bits (0-7) of the payload contain the vring index. Bit 8 is the
 253       invalid FD flag. This flag is set when there is no file descriptor
 254       in the ancillary data. This signals that polling will be used
 255       instead of waiting for the call.
 256
 257  * VHOST_USER_SET_VRING_ERR
 258
 259       Id: 14
 260       Equivalent ioctl: VHOST_SET_VRING_ERR
 261       Master payload: u64
 262
 263       Set the event file descriptor to signal when error occurs. It
 264       is passed in the ancillary data.
 265       Bits (0-7) of the payload contain the vring index. Bit 8 is the
 266       invalid FD flag. This flag is set when there is no file descriptor
 267       in the ancillary data.