]>
Commit | Line | Data |
---|---|---|
11fdf7f2 TL |
1 | .. SPDX-License-Identifier: BSD-3-Clause |
2 | Copyright(c) 2016 Intel Corporation. | |
7c673cae FG |
3 | |
4 | Live Migration of VM with Virtio on host running vhost_user | |
5 | =========================================================== | |
6 | ||
7 | Overview | |
8 | -------- | |
9 | ||
10 | Live Migration of a VM with DPDK Virtio PMD on a host which is | |
11 | running the Vhost sample application (vhost-switch) and using the DPDK PMD (ixgbe or i40e). | |
12 | ||
13 | The Vhost sample application uses VMDQ so SRIOV must be disabled on the NIC's. | |
14 | ||
15 | The following sections show an example of how to do this migration. | |
16 | ||
17 | Test Setup | |
18 | ---------- | |
19 | ||
20 | To test the Live Migration two servers with identical operating systems installed are used. | |
21 | KVM and QEMU is also required on the servers. | |
22 | ||
23 | QEMU 2.5 is required for Live Migration of a VM with vhost_user running on the hosts. | |
24 | ||
25 | In this example, the servers have Niantic and or Fortville NIC's installed. | |
26 | The NIC's on both servers are connected to a switch | |
27 | which is also connected to the traffic generator. | |
28 | ||
29 | The switch is configured to broadcast traffic on all the NIC ports. | |
30 | ||
31 | The ip address of host_server_1 is 10.237.212.46 | |
32 | ||
33 | The ip address of host_server_2 is 10.237.212.131 | |
34 | ||
35 | .. _figure_lm_vhost_user: | |
36 | ||
37 | .. figure:: img/lm_vhost_user.* | |
38 | ||
39 | Live Migration steps | |
40 | -------------------- | |
41 | ||
42 | The sample scripts mentioned in the steps below can be found in the | |
43 | :ref:`Sample host scripts <lm_virtio_vhost_user_host_scripts>` and | |
44 | :ref:`Sample VM scripts <lm_virtio_vhost_user_vm_scripts>` sections. | |
45 | ||
46 | On host_server_1: Terminal 1 | |
47 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
48 | ||
49 | Setup DPDK on host_server_1 | |
50 | ||
51 | .. code-block:: console | |
52 | ||
53 | cd /root/dpdk/host_scripts | |
54 | ./setup_dpdk_on_host.sh | |
55 | ||
56 | On host_server_1: Terminal 2 | |
57 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
58 | ||
59 | Bind the Niantic or Fortville NIC to igb_uio on host_server_1. | |
60 | ||
61 | For Fortville NIC. | |
62 | ||
63 | .. code-block:: console | |
64 | ||
11fdf7f2 | 65 | cd /root/dpdk/usertools |
7c673cae FG |
66 | ./dpdk-devbind.py -b igb_uio 0000:02:00.0 |
67 | ||
68 | For Niantic NIC. | |
69 | ||
70 | .. code-block:: console | |
71 | ||
11fdf7f2 | 72 | cd /root/dpdk/usertools |
7c673cae FG |
73 | ./dpdk-devbind.py -b igb_uio 0000:09:00.0 |
74 | ||
75 | On host_server_1: Terminal 3 | |
76 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
77 | ||
78 | For Fortville and Niantic NIC's reset SRIOV and run the | |
79 | vhost_user sample application (vhost-switch) on host_server_1. | |
80 | ||
81 | .. code-block:: console | |
82 | ||
83 | cd /root/dpdk/host_scripts | |
84 | ./reset_vf_on_212_46.sh | |
85 | ./run_vhost_switch_on_host.sh | |
86 | ||
87 | On host_server_1: Terminal 1 | |
88 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
89 | ||
90 | Start the VM on host_server_1 | |
91 | ||
92 | .. code-block:: console | |
93 | ||
94 | ./vm_virtio_vhost_user.sh | |
95 | ||
96 | On host_server_1: Terminal 4 | |
97 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
98 | ||
99 | Connect to the QEMU monitor on host_server_1. | |
100 | ||
101 | .. code-block:: console | |
102 | ||
103 | cd /root/dpdk/host_scripts | |
104 | ./connect_to_qemu_mon_on_host.sh | |
105 | (qemu) | |
106 | ||
107 | On host_server_1: Terminal 1 | |
108 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
109 | ||
110 | **In VM on host_server_1:** | |
111 | ||
112 | Setup DPDK in the VM and run testpmd in the VM. | |
113 | ||
114 | .. code-block:: console | |
115 | ||
116 | cd /root/dpdk/vm_scripts | |
117 | ./setup_dpdk_in_vm.sh | |
118 | ./run_testpmd_in_vm.sh | |
119 | ||
120 | testpmd> show port info all | |
121 | testpmd> set fwd mac retry | |
122 | testpmd> start tx_first | |
123 | testpmd> show port stats all | |
124 | ||
125 | Virtio traffic is seen at P1 and P2. | |
126 | ||
127 | On host_server_2: Terminal 1 | |
128 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
129 | ||
130 | Set up DPDK on the host_server_2. | |
131 | ||
132 | .. code-block:: console | |
133 | ||
134 | cd /root/dpdk/host_scripts | |
135 | ./setup_dpdk_on_host.sh | |
136 | ||
137 | On host_server_2: Terminal 2 | |
138 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
139 | ||
140 | Bind the Niantic or Fortville NIC to igb_uio on host_server_2. | |
141 | ||
142 | For Fortville NIC. | |
143 | ||
144 | .. code-block:: console | |
145 | ||
11fdf7f2 | 146 | cd /root/dpdk/usertools |
7c673cae FG |
147 | ./dpdk-devbind.py -b igb_uio 0000:03:00.0 |
148 | ||
149 | For Niantic NIC. | |
150 | ||
151 | .. code-block:: console | |
152 | ||
11fdf7f2 | 153 | cd /root/dpdk/usertools |
7c673cae FG |
154 | ./dpdk-devbind.py -b igb_uio 0000:06:00.0 |
155 | ||
156 | On host_server_2: Terminal 3 | |
157 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
158 | ||
159 | For Fortville and Niantic NIC's reset SRIOV, and run | |
160 | the vhost_user sample application on host_server_2. | |
161 | ||
162 | .. code-block:: console | |
163 | ||
164 | cd /root/dpdk/host_scripts | |
165 | ./reset_vf_on_212_131.sh | |
166 | ./run_vhost_switch_on_host.sh | |
167 | ||
168 | On host_server_2: Terminal 1 | |
169 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
170 | ||
171 | Start the VM on host_server_2. | |
172 | ||
173 | .. code-block:: console | |
174 | ||
175 | ./vm_virtio_vhost_user_migrate.sh | |
176 | ||
177 | On host_server_2: Terminal 4 | |
178 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
179 | ||
180 | Connect to the QEMU monitor on host_server_2. | |
181 | ||
182 | .. code-block:: console | |
183 | ||
184 | cd /root/dpdk/host_scripts | |
185 | ./connect_to_qemu_mon_on_host.sh | |
186 | (qemu) info status | |
187 | VM status: paused (inmigrate) | |
188 | (qemu) | |
189 | ||
190 | On host_server_1: Terminal 4 | |
191 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
192 | ||
193 | Check that switch is up before migrating the VM. | |
194 | ||
195 | .. code-block:: console | |
196 | ||
197 | (qemu) migrate tcp:10.237.212.131:5555 | |
198 | (qemu) info status | |
199 | VM status: paused (postmigrate) | |
200 | ||
201 | (qemu) info migrate | |
202 | capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off | |
203 | Migration status: completed | |
204 | total time: 11619 milliseconds | |
205 | downtime: 5 milliseconds | |
206 | setup: 7 milliseconds | |
207 | transferred ram: 379699 kbytes | |
208 | throughput: 267.82 mbps | |
209 | remaining ram: 0 kbytes | |
210 | total ram: 1590088 kbytes | |
211 | duplicate: 303985 pages | |
212 | skipped: 0 pages | |
213 | normal: 94073 pages | |
214 | normal bytes: 376292 kbytes | |
215 | dirty sync count: 2 | |
216 | (qemu) quit | |
217 | ||
218 | On host_server_2: Terminal 1 | |
219 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
220 | ||
221 | **In VM on host_server_2:** | |
222 | ||
223 | Hit Enter key. This brings the user to the testpmd prompt. | |
224 | ||
225 | .. code-block:: console | |
226 | ||
227 | testpmd> | |
228 | ||
229 | On host_server_2: Terminal 4 | |
230 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
231 | ||
232 | **In QEMU monitor on host_server_2** | |
233 | ||
234 | .. code-block:: console | |
235 | ||
236 | (qemu) info status | |
237 | VM status: running | |
238 | ||
239 | On host_server_2: Terminal 1 | |
240 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
241 | ||
242 | **In VM on host_server_2:** | |
243 | ||
244 | .. code-block:: console | |
245 | ||
9f95a23c | 246 | testpmd> show port info all |
7c673cae FG |
247 | testpmd> show port stats all |
248 | ||
249 | Virtio traffic is seen at P0 and P1. | |
250 | ||
251 | ||
252 | .. _lm_virtio_vhost_user_host_scripts: | |
253 | ||
254 | Sample host scripts | |
255 | ------------------- | |
256 | ||
257 | reset_vf_on_212_46.sh | |
258 | ~~~~~~~~~~~~~~~~~~~~~ | |
259 | ||
260 | .. code-block:: sh | |
261 | ||
262 | #!/bin/sh | |
263 | # This script is run on the host 10.237.212.46 to reset SRIOV | |
264 | ||
265 | # BDF for Fortville NIC is 0000:02:00.0 | |
266 | cat /sys/bus/pci/devices/0000\:02\:00.0/max_vfs | |
267 | echo 0 > /sys/bus/pci/devices/0000\:02\:00.0/max_vfs | |
268 | cat /sys/bus/pci/devices/0000\:02\:00.0/max_vfs | |
269 | ||
270 | # BDF for Niantic NIC is 0000:09:00.0 | |
271 | cat /sys/bus/pci/devices/0000\:09\:00.0/max_vfs | |
272 | echo 0 > /sys/bus/pci/devices/0000\:09\:00.0/max_vfs | |
273 | cat /sys/bus/pci/devices/0000\:09\:00.0/max_vfs | |
274 | ||
275 | vm_virtio_vhost_user.sh | |
276 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
277 | ||
278 | .. code-block:: sh | |
279 | ||
280 | #/bin/sh | |
281 | # Script for use with vhost_user sample application | |
282 | # The host system has 8 cpu's (0-7) | |
283 | ||
284 | # Path to KVM tool | |
285 | KVM_PATH="/usr/bin/qemu-system-x86_64" | |
286 | ||
287 | # Guest Disk image | |
288 | DISK_IMG="/home/user/disk_image/virt1_sml.disk" | |
289 | ||
290 | # Number of guest cpus | |
291 | VCPUS_NR="6" | |
292 | ||
293 | # Memory | |
294 | MEM=1024 | |
295 | ||
296 | VIRTIO_OPTIONS="csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off" | |
297 | ||
298 | # Socket Path | |
299 | SOCKET_PATH="/root/dpdk/host_scripts/usvhost" | |
300 | ||
301 | taskset -c 2-7 $KVM_PATH \ | |
302 | -enable-kvm \ | |
303 | -m $MEM \ | |
304 | -smp $VCPUS_NR \ | |
305 | -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on \ | |
306 | -numa node,memdev=mem,nodeid=0 \ | |
307 | -cpu host \ | |
308 | -name VM1 \ | |
309 | -no-reboot \ | |
310 | -net none \ | |
311 | -vnc none \ | |
312 | -nographic \ | |
313 | -hda $DISK_IMG \ | |
314 | -chardev socket,id=chr0,path=$SOCKET_PATH \ | |
315 | -netdev type=vhost-user,id=net1,chardev=chr0,vhostforce \ | |
316 | -device virtio-net-pci,netdev=net1,mac=CC:BB:BB:BB:BB:BB,$VIRTIO_OPTIONS \ | |
317 | -chardev socket,id=chr1,path=$SOCKET_PATH \ | |
318 | -netdev type=vhost-user,id=net2,chardev=chr1,vhostforce \ | |
319 | -device virtio-net-pci,netdev=net2,mac=DD:BB:BB:BB:BB:BB,$VIRTIO_OPTIONS \ | |
320 | -monitor telnet::3333,server,nowait | |
321 | ||
322 | connect_to_qemu_mon_on_host.sh | |
323 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
324 | ||
325 | .. code-block:: sh | |
326 | ||
327 | #!/bin/sh | |
328 | # This script is run on both hosts when the VM is up, | |
329 | # to connect to the Qemu Monitor. | |
330 | ||
331 | telnet 0 3333 | |
332 | ||
333 | reset_vf_on_212_131.sh | |
334 | ~~~~~~~~~~~~~~~~~~~~~~ | |
335 | ||
336 | .. code-block:: sh | |
337 | ||
338 | #!/bin/sh | |
339 | # This script is run on the host 10.237.212.131 to reset SRIOV | |
340 | ||
9f95a23c | 341 | # BDF for Niantic NIC is 0000:06:00.0 |
7c673cae FG |
342 | cat /sys/bus/pci/devices/0000\:06\:00.0/max_vfs |
343 | echo 0 > /sys/bus/pci/devices/0000\:06\:00.0/max_vfs | |
344 | cat /sys/bus/pci/devices/0000\:06\:00.0/max_vfs | |
345 | ||
346 | # BDF for Fortville NIC is 0000:03:00.0 | |
347 | cat /sys/bus/pci/devices/0000\:03\:00.0/max_vfs | |
348 | echo 0 > /sys/bus/pci/devices/0000\:03\:00.0/max_vfs | |
349 | cat /sys/bus/pci/devices/0000\:03\:00.0/max_vfs | |
350 | ||
351 | vm_virtio_vhost_user_migrate.sh | |
352 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
353 | ||
354 | .. code-block:: sh | |
355 | ||
356 | #/bin/sh | |
357 | # Script for use with vhost user sample application | |
358 | # The host system has 8 cpu's (0-7) | |
359 | ||
360 | # Path to KVM tool | |
361 | KVM_PATH="/usr/bin/qemu-system-x86_64" | |
362 | ||
363 | # Guest Disk image | |
364 | DISK_IMG="/home/user/disk_image/virt1_sml.disk" | |
365 | ||
366 | # Number of guest cpus | |
367 | VCPUS_NR="6" | |
368 | ||
369 | # Memory | |
370 | MEM=1024 | |
371 | ||
372 | VIRTIO_OPTIONS="csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off" | |
373 | ||
374 | # Socket Path | |
375 | SOCKET_PATH="/root/dpdk/host_scripts/usvhost" | |
376 | ||
377 | taskset -c 2-7 $KVM_PATH \ | |
378 | -enable-kvm \ | |
379 | -m $MEM \ | |
380 | -smp $VCPUS_NR \ | |
381 | -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on \ | |
382 | -numa node,memdev=mem,nodeid=0 \ | |
383 | -cpu host \ | |
384 | -name VM1 \ | |
385 | -no-reboot \ | |
386 | -net none \ | |
387 | -vnc none \ | |
388 | -nographic \ | |
389 | -hda $DISK_IMG \ | |
390 | -chardev socket,id=chr0,path=$SOCKET_PATH \ | |
391 | -netdev type=vhost-user,id=net1,chardev=chr0,vhostforce \ | |
392 | -device virtio-net-pci,netdev=net1,mac=CC:BB:BB:BB:BB:BB,$VIRTIO_OPTIONS \ | |
393 | -chardev socket,id=chr1,path=$SOCKET_PATH \ | |
394 | -netdev type=vhost-user,id=net2,chardev=chr1,vhostforce \ | |
395 | -device virtio-net-pci,netdev=net2,mac=DD:BB:BB:BB:BB:BB,$VIRTIO_OPTIONS \ | |
396 | -incoming tcp:0:5555 \ | |
397 | -monitor telnet::3333,server,nowait | |
398 | ||
399 | .. _lm_virtio_vhost_user_vm_scripts: | |
400 | ||
401 | Sample VM scripts | |
402 | ----------------- | |
403 | ||
404 | setup_dpdk_virtio_in_vm.sh | |
405 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
406 | ||
407 | .. code-block:: sh | |
408 | ||
409 | #!/bin/sh | |
410 | # this script matches the vm_virtio_vhost_user script | |
411 | # virtio port is 03 | |
412 | # virtio port is 04 | |
413 | ||
414 | cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
415 | echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
416 | cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
417 | ||
418 | ifconfig -a | |
11fdf7f2 | 419 | /root/dpdk/usertools/dpdk-devbind.py --status |
7c673cae FG |
420 | |
421 | rmmod virtio-pci | |
422 | ||
423 | modprobe uio | |
9f95a23c | 424 | insmod /root/dpdk/x86_64-default-linux-gcc/kmod/igb_uio.ko |
7c673cae | 425 | |
11fdf7f2 TL |
426 | /root/dpdk/usertools/dpdk-devbind.py -b igb_uio 0000:00:03.0 |
427 | /root/dpdk/usertools/dpdk-devbind.py -b igb_uio 0000:00:04.0 | |
7c673cae | 428 | |
11fdf7f2 | 429 | /root/dpdk/usertools/dpdk-devbind.py --status |
7c673cae FG |
430 | |
431 | run_testpmd_in_vm.sh | |
432 | ~~~~~~~~~~~~~~~~~~~~ | |
433 | ||
434 | .. code-block:: sh | |
435 | ||
436 | #!/bin/sh | |
437 | # Run testpmd for use with vhost_user sample app. | |
438 | # test system has 8 cpus (0-7), use cpus 2-7 for VM | |
439 | ||
9f95a23c | 440 | /root/dpdk/x86_64-default-linux-gcc/app/testpmd \ |
11fdf7f2 | 441 | -l 0-5 -n 4 --socket-mem 350 -- --burst=64 --i |