]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | # vhost Getting Started Guide {#vhost_getting_started} |
2 | ||
3 | The Storage Performance Development Kit vhost application is named "vhost". | |
4 | This application extends SPDK to present virtio-scsi controllers to QEMU-based | |
5 | VMs and process I/O submitted to devices attached to those controllers. | |
6 | ||
7 | # Prerequisites {#vhost_prereqs} | |
8 | ||
9 | The base SPDK build instructions are located README.md in SPDK main directory. | |
10 | This guide assumes familiarity with building SPDK using the default options. | |
11 | ||
12 | ## Supported Guest Operating Systems | |
13 | The guest OS must contain virtio drivers. The SPDK vhost target has been tested | |
14 | with Ubuntu 16.04, Fedora 25, Windows 2012 R2. | |
15 | ||
16 | # Building | |
17 | ||
18 | ## SPDK | |
19 | The vhost target is built by default. To enable/disable building the vhost | |
20 | target, either modify the following line in the CONFIG file in the root directory: | |
21 | ||
22 | ~~~ | |
23 | CONFIG_VHOST?=y | |
24 | ~~~ | |
25 | ||
26 | Or specify on the command line: | |
27 | ||
28 | ~~~ | |
29 | make CONFIG_VHOST=y | |
30 | ~~~ | |
31 | ||
32 | Once built, the binary will be at `app/vhost/vhost`. | |
33 | ||
34 | ## QEMU | |
35 | ||
36 | Vhost functionality is dependent on QEMU patches to enable vhost-scsi in | |
37 | userspace - those patches are currently working their way through the QEMU | |
38 | mailing list, but temporary patches to enable this functionality are available | |
39 | in the spdk branch at https://github.com/spdk/qemu. | |
40 | ||
41 | # Configuration {#vhost_config} | |
42 | ||
43 | ## SPDK | |
44 | A `vhost` specific configuration file is used to configure the SPDK vhost | |
45 | target. A fully documented example configuration file is located at | |
46 | `etc/spdk/vhost.conf.in`. This file defines the following: | |
47 | ||
48 | ### Storage Backends | |
49 | Storage backends are block devices which will be exposed as SCSI LUNs on | |
50 | devices attached to the vhost-scsi controller. SPDK supports several different | |
51 | types of storage backends, including NVMe, Linux AIO, malloc ramdisk and Ceph | |
52 | RBD. Refer to @ref bdev_getting_started for additional information on | |
53 | specifying storage backends in the configuration file. | |
54 | ||
55 | ### Mappings Between SCSI Controllers and Storage Backends | |
56 | The vhost target is exposing SCSI controllers to the virtual machines. | |
57 | Each device in the vhost controller is associated with an SPDK block device and | |
58 | configuration file defines those associations. The block device to Dev mappings | |
59 | are specified in the configuration file as: | |
60 | ||
61 | ~~~ | |
62 | [VhostScsiX] | |
63 | Name vhost.X # Name of vhost socket | |
64 | Dev 0 BackendX # "BackendX" is block device name from previous | |
65 | # sections in config file | |
66 | Dev 1 BackendY | |
67 | ... | |
68 | Dev n BackendN | |
69 | #Cpumask 0x1 # Optional parameter defining which core controller uses | |
70 | ||
71 | ~~~ | |
72 | ||
73 | ### Vhost Sockets | |
74 | Userspace vhost uses UNIX domain sockets for communication between QEMU | |
75 | and the vhost target. Each vhost controller is associated with a UNIX domain | |
76 | socket file with filename equal to the Name argument in configuration file. | |
77 | Sockets are created at current directory when starting the SPDK vhost target. | |
78 | ||
79 | ### Core Affinity Configuration | |
80 | Vhost target can be restricted to run on certain cores by specifying a ReactorMask. | |
81 | Default is to allow vhost target work on core 0. For NUMA systems it is essential | |
82 | to run vhost with cores on each socket to achieve optimal performance. | |
83 | ||
84 | To specify which core each controller should use, it can be defined by optional | |
85 | Cpumask parameter in configuration file. For NUMA systems the Cpumask should | |
86 | specify cores on the same CPU socket as its associated VM. | |
87 | ||
88 | ## QEMU | |
89 | ||
90 | Userspace vhost-scsi adds the following command line option for QEMU: | |
91 | ~~~ | |
92 | -device vhost-user-scsi-pci,id=scsi0,chardev=char0 | |
93 | ~~~ | |
94 | ||
95 | In order to start qemu with vhost you need to specify following options: | |
96 | ||
97 | - Socket, which QEMU will use for vhost communication with SPDK: | |
98 | ~~~ | |
99 | -chardev socket,id=char0,path=/path/to/vhost/socket | |
100 | ~~~ | |
101 | ||
102 | - Hugepages to share memory between vm and vhost target | |
103 | ~~~ | |
104 | -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on | |
105 | ~~~ | |
106 | ||
107 | # Running Vhost Target | |
108 | To get started, the following example is usually sufficient: | |
109 | ~~~ | |
110 | app/vhost/vhost -c /path/to/vhost.conf | |
111 | ~~~ | |
112 | ||
113 | A full list of command line arguments to vhost can be obtained by: | |
114 | ~~~ | |
115 | app/vhost/vhost -h | |
116 | ~~~ | |
117 | ||
118 | ||
119 | ## Example | |
120 | Assume that qemu and spdk are in respectively `qemu` and `spdk` directories. | |
121 | ~~~ | |
122 | ./qemu/build/x86_64-softmmu/qemu-system-x86_64 \ | |
123 | -m 1024 \ | |
124 | -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on \ | |
125 | -numa node,memdev=mem \ | |
126 | -drive file=$PROJECTS/os.qcow2,if=none,id=disk \ | |
127 | -device ide-hd,drive=disk,bootindex=0 \ | |
128 | -chardev socket,id=char0,path=./spdk/vhost.0 \ | |
129 | -device vhost-user-scsi-pci,id=scsi0,chardev=char0 \ | |
130 | --enable-kvm | |
131 | ~~~ | |
132 | ||
133 | # Experimental features {#vhost_experimental} | |
134 | ||
135 | ## Multi-Queue Block Layer (blk_mq) | |
136 | It is possible to use multiqueue feature in vhost. | |
137 | To enable it on linux it is required to modify kernel options inside | |
138 | virtual machine. | |
139 | ||
140 | Instructions below for Ubuntu OS: | |
141 | 1. `vi /etc/default/grub` | |
142 | 2. Make sure mq is enabled: | |
143 | GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1" | |
144 | 3. `sudo update-grub` | |
145 | 4. Reboot virtual machine | |
146 | ||
147 | To achieve better performance make sure to increase number of cores | |
148 | assigned to vm. | |
149 | ||
150 | # Known bugs and limitations {#vhost_bugs} | |
151 | ||
152 | ## VFIO driver is not supported | |
153 | Currently, VFIO driver is not supported by vhost library. Please use UIO | |
154 | driver if running SPDK vhost app. Otherwise any IO to physical device | |
155 | (eg. IOAT or NVMe) will fail. Support for vhost with VFIO is in progress. | |
156 | ||
157 | ## Hot plug is not supported | |
158 | Hot plug is not supported in vhost yet. Event queue path doesn't handle that | |
159 | case. While hot plug will be just ignored, hot removal might cause segmentation | |
160 | fault. |