INSTALL.DPDK.md

   1 Using Open vSwitch with DPDK
   2 ============================
   3
   4 Open vSwitch can use Intel(R) DPDK lib to operate entirely in
   5 userspace. This file explains how to install and use Open vSwitch in
   6 such a mode.
   7
   8 The DPDK support of Open vSwitch is considered experimental.
   9 It has not been thoroughly tested.
  10
  11 This version of Open vSwitch should be built manually with `configure`
  12 and `make`.
  13
  14 OVS needs a system with 1GB hugepages support.
  15
  16 Building and Installing:
  17 ------------------------
  18
  19 Required DPDK 1.8.0, `fuse`, `fuse-devel` (`libfuse-dev` on Debian/Ubuntu)
  20
  21 1. Configure build & install DPDK:
  22   1. Set `$DPDK_DIR`
  23
  24      ```
  25      export DPDK_DIR=/usr/src/dpdk-1.8.0
  26      cd $DPDK_DIR
  27      ```
  28
  29   2. Update `config/common_linuxapp` so that DPDK generate single lib file.
  30      (modification also required for IVSHMEM build)
  31
  32      `CONFIG_RTE_BUILD_COMBINE_LIBS=y`
  33
  34      Update `config/common_linuxapp` so that DPDK is built with vhost
  35      libraries:
  36
  37      `CONFIG_RTE_LIBRTE_VHOST=y`
  38
  39      Then run `make install` to build and install the library.
  40      For default install without IVSHMEM:
  41
  42      `make install T=x86_64-native-linuxapp-gcc`
  43
  44      To include IVSHMEM (shared memory):
  45
  46      `make install T=x86_64-ivshmem-linuxapp-gcc`
  47
  48      For further details refer to http://dpdk.org/
  49
  50 2. Configure & build the Linux kernel:
  51
  52    Refer to intel-dpdk-getting-started-guide.pdf for understanding
  53    DPDK kernel requirement.
  54
  55 3. Configure & build OVS:
  56
  57    * Non IVSHMEM:
  58
  59      `export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/`
  60
  61    * IVSHMEM:
  62
  63      `export DPDK_BUILD=$DPDK_DIR/x86_64-ivshmem-linuxapp-gcc/`
  64
  65    ```
  66    cd $(OVS_DIR)/openvswitch
  67    ./boot.sh
  68    ./configure --with-dpdk=$DPDK_BUILD
  69    make
  70    ```
  71
  72 To have better performance one can enable aggressive compiler optimizations and
  73 use the special instructions(popcnt, crc32) that may not be available on all
  74 machines. Instead of typing `make`, type:
  75
  76 `make CFLAGS='-O3 -march=native'`
  77
  78 Refer to [INSTALL.userspace.md] for general requirements of building userspace OVS.
  79
  80 Using the DPDK with ovs-vswitchd:
  81 ---------------------------------
  82
  83 1. Setup system boot
  84    Add the following options to the kernel bootline:
  85
  86    `default_hugepagesz=1GB hugepagesz=1G hugepages=1`
  87
  88 2. Setup DPDK devices:
  89
  90    DPDK devices can be setup using either the VFIO (for DPDK 1.7+) or UIO
  91    modules. UIO requires inserting an out of tree driver igb_uio.ko that is
  92    available in DPDK. Setup for both methods are described below.
  93
  94    * UIO:
  95      1. insert uio.ko: `modprobe uio`
  96      2. insert igb_uio.ko: `insmod $DPDK_BUILD/kmod/igb_uio.ko`
  97      3. Bind network device to igb_uio:
  98             `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio eth1`
  99
 100    * VFIO:
 101
 102      VFIO needs to be supported in the kernel and the BIOS. More information
 103      can be found in the [DPDK Linux GSG].
 104
 105      1. Insert vfio-pci.ko: `modprobe vfio-pci`
 106      2. Set correct permissions on vfio device: `sudo /usr/bin/chmod a+x /dev/vfio`
 107         and: `sudo /usr/bin/chmod 0666 /dev/vfio/*`
 108      3. Bind network device to vfio-pci:
 109             `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=vfio-pci eth1`
 110
 111 3. Mount the hugetable filsystem
 112
 113    `mount -t hugetlbfs -o pagesize=1G none /dev/hugepages`
 114
 115    Ref to http://www.dpdk.org/doc/quick-start for verifying DPDK setup.
 116
 117 4. Follow the instructions in [INSTALL.md] to install only the
 118    userspace daemons and utilities (via 'make install').
 119    1. First time only db creation (or clearing):
 120
 121       ```
 122       mkdir -p /usr/local/etc/openvswitch
 123       mkdir -p /usr/local/var/run/openvswitch
 124       rm /usr/local/etc/openvswitch/conf.db
 125       ovsdb-tool create /usr/local/etc/openvswitch/conf.db  \
 126              /usr/local/share/openvswitch/vswitch.ovsschema
 127       ```
 128
 129    2. Start ovsdb-server
 130
 131       ```
 132       ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
 133           --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
 134           --private-key=db:Open_vSwitch,SSL,private_key \
 135           --certificate=Open_vSwitch,SSL,certificate \
 136           --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach
 137       ```
 138
 139     3. First time after db creation, initialize:
 140
 141        ```
 142        ovs-vsctl --no-wait init
 143        ```
 144
 145 5. Start vswitchd:
 146
 147    DPDK configuration arguments can be passed to vswitchd via `--dpdk`
 148    argument. This needs to be first argument passed to vswitchd process.
 149    dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter
 150    for dpdk initialization.
 151
 152    ```
 153    export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
 154    ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile --detach
 155    ```
 156
 157    If allocated more than one GB hugepage (as for IVSHMEM), set amount and
 158    use NUMA node 0 memory:
 159
 160    ```
 161    ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024,0 \
 162    -- unix:$DB_SOCK --pidfile --detach
 163    ```
 164
 165 6. Add bridge & ports
 166
 167    To use ovs-vswitchd with DPDK, create a bridge with datapath_type
 168    "netdev" in the configuration database.  For example:
 169
 170    `ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev`
 171
 172    Now you can add dpdk devices. OVS expect DPDK device name start with dpdk
 173    and end with portid. vswitchd should print (in the log file) the number
 174    of dpdk devices found.
 175
 176    ```
 177    ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
 178    ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
 179    ```
 180
 181    Once first DPDK port is added to vswitchd, it creates a Polling thread and
 182    polls dpdk device in continuous loop. Therefore CPU utilization
 183    for that thread is always 100%.
 184
 185 7. Add test flows
 186
 187    Test flow script across NICs (assuming ovs in /usr/src/ovs):
 188    Execute script:
 189
 190    ```
 191    #! /bin/sh
 192    # Move to command directory
 193    cd /usr/src/ovs/utilities/
 194
 195    # Clear current flows
 196    ./ovs-ofctl del-flows br0
 197
 198    # Add flows between port 1 (dpdk0) to port 2 (dpdk1)
 199    ./ovs-ofctl add-flow br0 in_port=1,action=output:2
 200    ./ovs-ofctl add-flow br0 in_port=2,action=output:1
 201    ```
 202
 203 8. Performance tuning
 204
 205    With pmd multi-threading support, OVS creates one pmd thread for each
 206    numa node as default.  The pmd thread handles the I/O of all DPDK
 207    interfaces on the same numa node.  The following two commands can be used
 208    to configure the multi-threading behavior.
 209
 210    `ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=<hex string>`
 211
 212    The command above asks for a CPU mask for setting the affinity of pmd
 213    threads.  A set bit in the mask means a pmd thread is created and pinned
 214    to the corresponding CPU core.  For more information, please refer to
 215    `man ovs-vswitchd.conf.db`
 216
 217    `ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer>`
 218
 219    The command above sets the number of rx queues of each DPDK interface. The
 220    rx queues are assigned to pmd threads on the same numa node in round-robin
 221    fashion.  For more information, please refer to `man ovs-vswitchd.conf.db`
 222
 223    Ideally for maximum throughput, the pmd thread should not be scheduled out
 224    which temporarily halts its execution. The following affinitization methods
 225    can help.
 226
 227    Lets pick core 4,6,8,10 for pmd threads to run on.  Also assume a dual 8 core
 228    sandy bridge system with hyperthreading enabled where CPU1 has cores 0,...,7
 229    and 16,...,23 & CPU2 cores 8,...,15 & 24,...,31.  (A different cpu
 230    configuration could have different core mask requirements).
 231
 232    To kernel bootline add core isolation list for cores and associated hype cores
 233    (e.g.  isolcpus=4,20,6,22,8,24,10,26,).  Reboot system for isolation to take
 234    effect, restart everything.
 235
 236    Configure pmd threads on core 4,6,8,10 using 'pmd-cpu-mask':
 237
 238    `ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=00000550`
 239
 240    You should be able to check that pmd threads are pinned to the correct cores
 241    via:
 242
 243    ```
 244    top -p `pidof ovs-vswitchd` -H -d1
 245    ```
 246
 247    Note, the pmd threads on a numa node are only created if there is at least
 248    one DPDK interface from the numa node that has been added to OVS.
 249
 250    Note, core 0 is always reserved from non-pmd threads and should never be set
 251    in the cpu mask.
 252
 253 DPDK Rings :
 254 ------------
 255
 256 Following the steps above to create a bridge, you can now add dpdk rings
 257 as a port to the vswitch.  OVS will expect the DPDK ring device name to
 258 start with dpdkr and end with a portid.
 259
 260 `ovs-vsctl add-port br0 dpdkr0 -- set Interface dpdkr0 type=dpdkr`
 261
 262 DPDK rings client test application
 263
 264 Included in the test directory is a sample DPDK application for testing
 265 the rings.  This is from the base dpdk directory and modified to work
 266 with the ring naming used within ovs.
 267
 268 location tests/ovs_client
 269
 270 To run the client :
 271
 272 ```
 273 cd /usr/src/ovs/tests/
 274 ovsclient -c 1 -n 4 --proc-type=secondary -- -n "port id you gave dpdkr"
 275 ```
 276
 277 In the case of the dpdkr example above the "port id you gave dpdkr" is 0.
 278
 279 It is essential to have --proc-type=secondary
 280
 281 The application simply receives an mbuf on the receive queue of the
 282 ethernet ring and then places that same mbuf on the transmit ring of
 283 the ethernet ring.  It is a trivial loopback application.
 284
 285 DPDK rings in VM (IVSHMEM shared memory communications)
 286 -------------------------------------------------------
 287
 288 In addition to executing the client in the host, you can execute it within
 289 a guest VM. To do so you will need a patched qemu.  You can download the
 290 patch and getting started guide at :
 291
 292 https://01.org/packet-processing/downloads
 293
 294 A general rule of thumb for better performance is that the client
 295 application should not be assigned the same dpdk core mask "-c" as
 296 the vswitchd.
 297
 298 DPDK vhost:
 299 -----------
 300
 301 vhost-cuse is only supported at present i.e. not using the standard QEMU
 302 vhost-user interface. It is intended that vhost-user support will be added
 303 in future releases when supported in DPDK and that vhost-cuse will eventually
 304 be deprecated. See [DPDK Docs] for more info on vhost.
 305
 306 Prerequisites:
 307 1.  Insert the Cuse module:
 308
 309       `modprobe cuse`
 310
 311 2.  Build and insert the `eventfd_link` module:
 312
 313      `cd $DPDK_DIR/lib/librte_vhost/eventfd_link/`
 314      `make`
 315      `insmod $DPDK_DIR/lib/librte_vhost/eventfd_link.ko`
 316
 317 Following the steps above to create a bridge, you can now add DPDK vhost
 318 as a port to the vswitch.
 319
 320 `ovs-vsctl add-port br0 dpdkvhost0 -- set Interface dpdkvhost0 type=dpdkvhost`
 321
 322 Unlike DPDK ring ports, DPDK vhost ports can have arbitrary names:
 323
 324 `ovs-vsctl add-port br0 port123ABC -- set Interface port123ABC type=dpdkvhost`
 325
 326 However, please note that when attaching userspace devices to QEMU, the
 327 name provided during the add-port operation must match the ifname parameter
 328 on the QEMU command line.
 329
 330
 331 DPDK vhost VM configuration:
 332 ----------------------------
 333
 334    vhost ports use a Linux* character device to communicate with QEMU.
 335    By default it is set to `/dev/vhost-net`. It is possible to reuse this
 336    standard device for DPDK vhost, which makes setup a little simpler but it
 337    is better practice to specify an alternative character device in order to
 338    avoid any conflicts if kernel vhost is to be used in parallel.
 339
 340 1. This step is only needed if using an alternative character device.
 341
 342    The new character device filename must be specified on the vswitchd
 343    commandline:
 344
 345         `./vswitchd/ovs-vswitchd --dpdk --cuse_dev_name my-vhost-net -c 0x1 ...`
 346
 347    Note that the `--cuse_dev_name` argument and associated string must be the first
 348    arguments after `--dpdk` and come before the EAL arguments. In the example
 349    above, the character device to be used will be `/dev/my-vhost-net`.
 350
 351 2. This step is only needed if reusing the standard character device. It will
 352    conflict with the kernel vhost character device so the user must first
 353    remove it.
 354
 355        `rm -rf /dev/vhost-net`
 356
 357 3a. Configure virtio-net adaptors:
 358    The following parameters must be passed to the QEMU binary:
 359
 360      ```
 361      -netdev tap,id=<id>,script=no,downscript=no,ifname=<name>,vhost=on
 362      -device virtio-net-pci,netdev=net1,mac=<mac>
 363      ```
 364
 365      Repeat the above parameters for multiple devices.
 366
 367      The DPDK vhost library will negiotiate its own features, so they
 368      need not be passed in as command line params. Note that as offloads are
 369      disabled this is the equivalent of setting:
 370
 371      `csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off`
 372
 373 3b. If using an alternative character device. It must be also explicitly
 374     passed to QEMU using the `vhostfd` argument:
 375
 376      ```
 377      -netdev tap,id=<id>,script=no,downscript=no,ifname=<name>,vhost=on,
 378      vhostfd=<open_fd>
 379      -device virtio-net-pci,netdev=net1,mac=<mac>
 380      ```
 381
 382      The open file descriptor must be passed to QEMU running as a child
 383      process. This could be done with a simple python script.
 384
 385        ```
 386        #!/usr/bin/python
 387        fd = os.open("/dev/usvhost", os.O_RDWR)
 388        subprocess.call("qemu-system-x86_64 .... -netdev tap,id=vhostnet0,\
 389                         vhost=on,vhostfd=" + fd +"...", shell=True)
 390
 391    Alternatively the the `qemu-wrap.py` script can be used to automate the
 392    requirements specified above and can be used in conjunction with libvirt if
 393    desired. See the "DPDK vhost VM configuration with QEMU wrapper" section
 394    below.
 395
 396 4. Configure huge pages:
 397    QEMU must allocate the VM's memory on hugetlbfs. Vhost ports access a
 398    virtio-net device's virtual rings and packet buffers mapping the VM's
 399    physical memory on hugetlbfs. To enable vhost-ports to map the VM's
 400    memory into their process address space, pass the following paramters
 401    to QEMU:
 402
 403      `-object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,
 404       share=on -numa node,memdev=mem -mem-prealloc`
 405
 406
 407 DPDK vhost VM configuration with QEMU wrapper:
 408 ----------------------------------------------
 409
 410 The QEMU wrapper script automatically detects and calls QEMU with the
 411 necessary parameters. It performs the following actions:
 412
 413   * Automatically detects the location of the hugetlbfs and inserts this
 414     into the command line parameters.
 415   * Automatically open file descriptors for each virtio-net device and
 416     inserts this into the command line parameters.
 417   * Calls QEMU passing both the command line parameters passed to the
 418     script itself and those it has auto-detected.
 419
 420 Before use, you **must** edit the configuration parameters section of the
 421 script to point to the correct emulator location and set additional
 422 settings. Of these settings, `emul_path` and `us_vhost_path` **must** be
 423 set. All other settings are optional.
 424
 425 To use directly from the command line simply pass the wrapper some of the
 426 QEMU parameters: it will configure the rest. For example:
 427
 428 ```
 429 qemu-wrap.py -cpu host -boot c -hda <disk image> -m 4096 -smp 4
 430   --enable-kvm -nographic -vnc none -net none -netdev tap,id=net1,
 431   script=no,downscript=no,ifname=if1,vhost=on -device virtio-net-pci,
 432   netdev=net1,mac=00:00:00:00:00:01
 433 ```
 434
 435 DPDK vhost VM configuration with libvirt:
 436 -----------------------------------------
 437
 438 If you are using libvirt, you must enable libvirt to access the character
 439 device by adding it to controllers cgroup for libvirtd using the following
 440 steps.
 441
 442      1. In `/etc/libvirt/qemu.conf` add/edit the following lines:
 443
 444         ```
 445         1) clear_emulator_capabilities = 0
 446         2) user = "root"
 447         3) group = "root"
 448         4) cgroup_device_acl = [
 449                "/dev/null", "/dev/full", "/dev/zero",
 450                "/dev/random", "/dev/urandom",
 451                "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
 452                "/dev/rtc", "/dev/hpet", "/dev/net/tun",
 453                "/dev/<my-vhost-device>",
 454                "/dev/hugepages"]
 455         ```
 456
 457         <my-vhost-device> refers to "vhost-net" if using the `/dev/vhost-net`
 458         device. If you have specificed a different name on the ovs-vswitchd
 459         commandline using the "--cuse_dev_name" parameter, please specify that
 460         filename instead.
 461
 462      2. Disable SELinux or set to permissive mode
 463
 464      3. Restart the libvirtd process
 465         For example, on Fedora:
 466
 467           `systemctl restart libvirtd.service`
 468
 469 After successfully editing the configuration, you may launch your
 470 vhost-enabled VM. The XML describing the VM can be configured like so
 471 within the <qemu:commandline> section:
 472
 473      1. Set up shared hugepages:
 474
 475      ```
 476      <qemu:arg value='-object'/>
 477      <qemu:arg value='memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on'/>
 478      <qemu:arg value='-numa'/>
 479      <qemu:arg value='node,memdev=mem'/>
 480      <qemu:arg value='-mem-prealloc'/>
 481      ```
 482
 483      2. Set up your tap devices:
 484
 485      ```
 486      <qemu:arg value='-netdev'/>
 487      <qemu:arg value='type=tap,id=net1,script=no,downscript=no,ifname=vhost0,vhost=on'/>
 488      <qemu:arg value='-device'/>
 489      <qemu:arg value='virtio-net-pci,netdev=net1,mac=00:00:00:00:00:01'/>
 490      ```
 491
 492      Repeat for as many devices as are desired, modifying the id, ifname
 493      and mac as necessary.
 494
 495      Again, if you are using an alternative character device (other than
 496      `/dev/vhost-net`), please specify the file descriptor like so:
 497
 498      `<qemu:arg value='type=tap,id=net3,script=no,downscript=no,ifname=vhost0,vhost=on,vhostfd=<open_fd>'/>`
 499
 500      Where <open_fd> refers to the open file descriptor of the character device.
 501      Instructions of how to retrieve the file descriptor can be found in the
 502      "DPDK vhost VM configuration" section.
 503      Alternatively, the process is automated with the qemu-wrap.py script,
 504      detailed in the next section.
 505
 506 Now you may launch your VM using virt-manager, or like so:
 507
 508     `virsh create my_vhost_vm.xml`
 509
 510 DPDK vhost VM configuration with libvirt and QEMU wrapper:
 511 ----------------------------------------------------------
 512
 513 To use the qemu-wrapper script in conjuntion with libvirt, follow the
 514 steps in the previous section before proceeding with the following steps:
 515
 516   1. Place `qemu-wrap.py` in libvirtd's binary search PATH ($PATH)
 517      Ideally in the same directory that the QEMU binary is located.
 518
 519   2. Ensure that the script has the same owner/group and file permissions
 520      as the QEMU binary.
 521
 522   3. Update the VM xml file using "virsh edit VM.xml"
 523
 524        1. Set the VM to use the launch script.
 525           Set the emulator path contained in the `<emulator><emulator/>` tags.
 526           For example, replace:
 527
 528             `<emulator>/usr/bin/qemu-kvm<emulator/>`
 529
 530             with:
 531
 532             `<emulator>/usr/bin/qemu-wrap.py<emulator/>`
 533
 534   4. Edit the Configuration Parameters section of the script to point to
 535   the correct emulator location and set any additional options. If you are
 536   using a alternative character device name, please set "us_vhost_path" to the
 537   location of that device. The script will automatically detect and insert
 538   the correct "vhostfd" value in the QEMU command line arguements.
 539
 540   5. Use virt-manager to launch the VM
 541
 542 Restrictions:
 543 -------------
 544
 545   - Work with 1500 MTU, needs few changes in DPDK lib to fix this issue.
 546   - Currently DPDK port does not make use any offload functionality.
 547   - DPDK-vHost support works with 1G huge pages.
 548
 549   ivshmem:
 550   - The shared memory is currently restricted to the use of a 1GB
 551     huge pages.
 552   - All huge pages are shared amongst the host, clients, virtual
 553     machines etc.
 554
 555 Bug Reporting:
 556 --------------
 557
 558 Please report problems to bugs@openvswitch.org.
 559
 560 [INSTALL.userspace.md]:INSTALL.userspace.md
 561 [INSTALL.md]:INSTALL.md
 562 [DPDK Linux GSG]: http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules
 563 [DPDK Docs]: http://dpdk.org/doc