ceph/src/seastar/dpdk/doc/guides/nics/ena.rst

   1 .. BSD LICENSE
   2
   3     Copyright (c) 2015-2018 Amazon.com, Inc. or its affiliates.
   4     All rights reserved.
   5
   6     Redistribution and use in source and binary forms, with or without
   7     modification, are permitted provided that the following conditions
   8     are met:
   9
  10     * Redistributions of source code must retain the above copyright
  11     notice, this list of conditions and the following disclaimer.
  12     * Redistributions in binary form must reproduce the above copyright
  13     notice, this list of conditions and the following disclaimer in
  14     the documentation and/or other materials provided with the
  15     distribution.
  16     * Neither the name of Amazon.com, Inc. nor the names of its
  17     contributors may be used to endorse or promote products derived
  18     from this software without specific prior written permission.
  19
  20     THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
  21     "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
  22     LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
  23     A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
  24     OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  25     SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
  26     LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
  27     DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
  28     THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  29     (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
  30     OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  31
  32 ENA Poll Mode Driver
  33 ====================
  34
  35 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
  36 Network Adapter (ENA) family.
  37
  38 Overview
  39 --------
  40
  41 The ENA driver exposes a lightweight management interface with a
  42 minimal set of memory mapped registers and an extendable command set
  43 through an Admin Queue.
  44
  45 The driver supports a wide range of ENA adapters, is link-speed
  46 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
  47 etc.), and it negotiates and supports an extendable feature set.
  48
  49 ENA adapters allow high speed and low overhead Ethernet traffic
  50 processing by providing a dedicated Tx/Rx queue pair per CPU core.
  51
  52 The ENA driver supports industry standard TCP/IP offload features such
  53 as checksum offload and TCP transmit segmentation offload (TSO).
  54
  55 Receive-side scaling (RSS) is supported for multi-core scaling.
  56
  57 Some of the ENA devices support a working mode called Low-latency
  58 Queue (LLQ), which saves several more microseconds.
  59
  60 Management Interface
  61 --------------------
  62
  63 ENA management interface is exposed by means of:
  64
  65 * Device Registers
  66 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
  67
  68 ENA device memory-mapped PCIe space for registers (MMIO registers)
  69 are accessed only during driver initialization and are not involved
  70 in further normal device operation.
  71
  72 AQ is used for submitting management commands, and the
  73 results/responses are reported asynchronously through ACQ.
  74
  75 ENA introduces a very small set of management commands with room for
  76 vendor-specific extensions. Most of the management operations are
  77 framed in a generic Get/Set feature command.
  78
  79 The following admin queue commands are supported:
  80
  81 * Create I/O submission queue
  82 * Create I/O completion queue
  83 * Destroy I/O submission queue
  84 * Destroy I/O completion queue
  85 * Get feature
  86 * Set feature
  87 * Get statistics
  88
  89 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
  90 properties.
  91
  92 Data Path Interface
  93 -------------------
  94
  95 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
  96 SQ correspondingly). Each SQ has a completion queue (CQ) associated
  97 with it.
  98
  99 The SQs and CQs are implemented as descriptor rings in contiguous
 100 physical memory.
 101
 102 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
 103
 104 The driver supports multi-queue for both Tx and Rx.
 105
 106 Configuration information
 107 -------------------------
 108
 109 **DPDK Configuration Parameters**
 110
 111   The following configuration options are available for the ENA PMD:
 112
 113    * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
 114      of the ENA PMD driver in the DPDK compilation.
 115
 116    * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
 117      logging of RX logic within the ENA PMD driver.
 118
 119    * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
 120      logging of TX logic within the ENA PMD driver.
 121
 122    * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
 123      logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
 124
 125 **ENA Configuration Parameters**
 126
 127    * **Number of Queues**
 128
 129      This is the requested number of queues upon initialization, however, the actual
 130      number of receive and transmit queues to be created will be the minimum between
 131      the maximal number supported by the device and number of queues requested.
 132
 133    * **Size of Queues**
 134
 135      This is the requested size of receive/transmit queues, while the actual size
 136      will be the minimum between the requested size and the maximal receive/transmit
 137      supported by the device.
 138
 139 Building DPDK
 140 -------------
 141
 142 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
 143 instructions on how to build DPDK.
 144
 145 By default the ENA PMD library will be built into the DPDK library.
 146
 147 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
 148 documentation that comes with DPDK suite <linux_gsg>`.
 149
 150 Supported ENA adapters
 151 ----------------------
 152
 153 Current ENA PMD supports the following ENA adapters including:
 154
 155 * ``1d0f:ec20`` - ENA VF
 156 * ``1d0f:ec21`` - ENA VF with LLQ support
 157
 158 Supported Operating Systems
 159 ---------------------------
 160
 161 Any Linux distribution fulfilling the conditions described in ``System Requirements``
 162 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
 163
 164 Supported features
 165 ------------------
 166
 167 * MTU configuration
 168 * Jumbo frames up to 9K
 169 * IPv4/TCP/UDP checksum offload
 170 * TSO offload
 171 * Multiple receive and transmit queues
 172 * RSS hash
 173 * RSS indirection table configuration
 174 * Low Latency Queue for Tx
 175 * Basic and extended statistics
 176 * LSC event notification
 177 * Watchdog (requires handling of timers in the application)
 178 * Device reset upon failure
 179
 180 Prerequisites
 181 -------------
 182
 183 #. Prepare the system as recommended by DPDK suite.  This includes environment
 184    variables, hugepages configuration, tool-chains and configuration.
 185
 186 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
 187
 188    (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
 189    reduces the latency of the packets by pushing the header directly through
 190    the PCI to the device, before the DMA is even triggered. For proper work
 191    kernel PCI driver must support write combining (WC). In mainline version of
 192    ``igb_uio`` (in DPDK repo) it must be enabled by loading module with
 193    ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
 194    driver in kernel doesn't have WC support yet (planed to be added).
 195    If vfio-pci used user should be either turn off ENAv2 (to avoid performance
 196    impact) or recompile vfio-pci driver with patch provided in
 197    `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
 198
 199 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
 200    ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
 201    respectively.
 202
 203 #. For ``vfio-pci`` users only:
 204    Please make sure that ``IOMMU`` is enabled in your system,
 205    or use ``vfio`` driver in ``noiommu`` mode::
 206
 207      echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 208
 209 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
 210
 211
 212 At this point the system should be ready to run DPDK applications. Once the
 213 application runs to completion, the ENA can be detached from igb_uio if necessary.
 214
 215 Usage example
 216 -------------
 217
 218 Follow instructions available in the document
 219 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
 220 **testpmd** with Amazon ENA devices managed by librte_pmd_ena.
 221
 222 Example output:
 223
 224 .. code-block:: console
 225
 226    [...]
 227    EAL: PCI device 0000:00:06.0 on NUMA socket -1
 228    EAL:   Invalid NUMA socket, default to 0
 229    EAL:   probe driver: 1d0f:ec20 net_ena
 230
 231    Interactive-mode selected
 232    testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
 233    testpmd: preferred mempool ops selected: ring_mp_mc
 234    Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
 235    Configuring Port 0 (socket 0)
 236    Port 0: 00:00:00:11:00:01
 237    Checking link statuses...
 238
 239    Done
 240    testpmd>