[ceph.git] / ceph / src / dpdk / doc / guides / nics / ena.rst

.. BSD LICENSE

    Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates.
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:

    * Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in
    the documentation and/or other materials provided with the
    distribution.
    * Neither the name of Amazon.com, Inc. nor the names of its
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ENA Poll Mode Driver
====================

The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
Network Adapter (ENA) family.

Overview
--------

The ENA driver exposes a lightweight management interface with a
minimal set of memory mapped registers and an extendable command set
through an Admin Queue.

The driver supports a wide range of ENA adapters, is link-speed
independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
etc.), and it negotiates and supports an extendable feature set.

ENA adapters allow high speed and low overhead Ethernet traffic
processing by providing a dedicated Tx/Rx queue pair per CPU core.

The ENA driver supports industry standard TCP/IP offload features such
as checksum offload and TCP transmit segmentation offload (TSO).

Receive-side scaling (RSS) is supported for multi-core scaling.

Some of the ENA devices support a working mode called Low-latency
Queue (LLQ), which saves several more microseconds.

Management Interface
--------------------

ENA management interface is exposed by means of:

* Device Registers
* Admin Queue (AQ) and Admin Completion Queue (ACQ)

ENA device memory-mapped PCIe space for registers (MMIO registers)
are accessed only during driver initialization and are not involved
in further normal device operation.

AQ is used for submitting management commands, and the
results/responses are reported asynchronously through ACQ.

ENA introduces a very small set of management commands with room for
vendor-specific extensions. Most of the management operations are
framed in a generic Get/Set feature command.

The following admin queue commands are supported:

* Create I/O submission queue
* Create I/O completion queue
* Destroy I/O submission queue
* Destroy I/O completion queue
* Get feature
* Set feature
* Get statistics

Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
properties.

Data Path Interface
-------------------

I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
SQ correspondingly). Each SQ has a completion queue (CQ) associated
with it.

The SQs and CQs are implemented as descriptor rings in contiguous
physical memory.

Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor

The driver supports multi-queue for both Tx and Rx.

Configuration information
-------------------------

**DPDK Configuration Parameters**

  The following configuration options are available for the ENA PMD:

   * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
     of the ENA PMD driver in the DPDK compilation.


   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT** (default y): Enables or disables debug
     logging of device initialization within the ENA PMD driver.

   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
     logging of RX logic within the ENA PMD driver.

   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
     logging of TX logic within the ENA PMD driver.

   * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
     logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.

**ENA Configuration Parameters**

   * **Number of Queues**

     This is the requested number of queues upon initialization, however, the actual
     number of receive and transmit queues to be created will be the minimum between
     the maximal number supported by the device and number of queues requested.

   * **Size of Queues**

     This is the requested size of receive/transmit queues, while the actual size
     will be the minimum between the requested size and the maximal receive/transmit
     supported by the device.

Building DPDK
-------------

See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
instructions on how to build DPDK.

By default the ENA PMD library will be built into the DPDK library.

For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
documentation that comes with DPDK suite <linux_gsg>`.

Supported ENA adapters
----------------------

Current ENA PMD supports the following ENA adapters including:

* ``1d0f:ec20`` - ENA VF
* ``1d0f:ec21`` - ENA VF with LLQ support

Supported Operating Systems
---------------------------

Any Linux distribution fulfilling the conditions described in ``System Requirements``
section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.

Supported features
------------------

* Jumbo frames up to 9K
* Port Hardware Statistics
* IPv4/TCP/UDP checksum offload
* TSO offload
* Multiple receive and transmit queues
* RSS
* Low Latency Queue for Tx

Unsupported features
--------------------

The features supported by the device and not yet supported by this PMD include:

* Asynchronous Event Notification Queue (AENQ)

Prerequisites
-------------

#. Prepare the system as recommended by DPDK suite.  This includes environment
   variables, hugepages configuration, tool-chains and configuration

#. Insert igb_uio kernel module using the command 'modprobe igb_uio'

#. Bind the intended ENA device to igb_uio module


At this point the system should be ready to run DPDK applications. Once the
application runs to completion, the ENA can be detached from igb_uio if necessary.

Usage example
-------------

This section demonstrates how to launch **testpmd** with Amazon ENA
devices managed by librte_pmd_ena.

#. Load the kernel modules:

   .. code-block:: console

      modprobe uio
      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

   .. note::

      Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module

#. Mount and request huge pages:

   .. code-block:: console

      mount -t hugetlbfs nodev /mnt/hugepages
      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

#. Bind UIO driver to ENA device (using provided by DPDK binding tool):

   .. code-block:: console

      ./tools/dpdk-devbind.py --bind=igb_uio 0000:02:00.1

#. Start testpmd with basic parameters:

   .. code-block:: console

      ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i

   Example output:

   .. code-block:: console

      [...]
      EAL: PCI device 0000:02:00.1 on NUMA socket -1
      EAL:   probe driver: 1d0f:ec20 rte_ena_pmd
      EAL:   PCI memory mapped at 0x7f9b6c400000
      PMD: eth_ena_dev_init(): Initializing 0:2:0.1
      Interactive-mode selected
      Configuring Port 0 (socket 0)
      Port 0: 00:00:00:11:00:01
      Checking link statuses...
      Port 0 Link Up - speed 10000 Mbps - full-duplex
      Done
      testpmd>
Commit	Line	Data
7c673cae FG	1	.. BSD LICENSE
	2
	3	Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates.
	4	All rights reserved.
	5
	6	Redistribution and use in source and binary forms, with or without
	7	modification, are permitted provided that the following conditions
	8	are met:
	9
	10	* Redistributions of source code must retain the above copyright
	11	notice, this list of conditions and the following disclaimer.
	12	* Redistributions in binary form must reproduce the above copyright
	13	notice, this list of conditions and the following disclaimer in
	14	the documentation and/or other materials provided with the
	15	distribution.
	16	* Neither the name of Amazon.com, Inc. nor the names of its
	17	contributors may be used to endorse or promote products derived
	18	from this software without specific prior written permission.
	19
	20	THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
	21	"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
	22	LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
	23	A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
	24	OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
	25	SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
	26	LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
	27	DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
	28	THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
	29	(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	30	OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	31
	32	ENA Poll Mode Driver
	33	====================
	34
	35	The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
	36	Network Adapter (ENA) family.
	37
	38	Overview
	39	--------
	40
	41	The ENA driver exposes a lightweight management interface with a
	42	minimal set of memory mapped registers and an extendable command set
	43	through an Admin Queue.
	44
	45	The driver supports a wide range of ENA adapters, is link-speed
	46	independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
	47	etc.), and it negotiates and supports an extendable feature set.
	48
	49	ENA adapters allow high speed and low overhead Ethernet traffic
	50	processing by providing a dedicated Tx/Rx queue pair per CPU core.
	51
	52	The ENA driver supports industry standard TCP/IP offload features such
	53	as checksum offload and TCP transmit segmentation offload (TSO).
	54
	55	Receive-side scaling (RSS) is supported for multi-core scaling.
	56
	57	Some of the ENA devices support a working mode called Low-latency
	58	Queue (LLQ), which saves several more microseconds.
	59
	60	Management Interface
	61	--------------------
	62
	63	ENA management interface is exposed by means of:
	64
65	* Device Registers
66	* Admin Queue (AQ) and Admin Completion Queue (ACQ)
67
68	ENA device memory-mapped PCIe space for registers (MMIO registers)
69	are accessed only during driver initialization and are not involved
70	in further normal device operation.
71
72	AQ is used for submitting management commands, and the
73	results/responses are reported asynchronously through ACQ.
74
75	ENA introduces a very small set of management commands with room for
76	vendor-specific extensions. Most of the management operations are
77	framed in a generic Get/Set feature command.
78
79	The following admin queue commands are supported:
80
81	* Create I/O submission queue
82	* Create I/O completion queue
83	* Destroy I/O submission queue
84	* Destroy I/O completion queue
85	* Get feature
86	* Set feature
87	* Get statistics
88
89	Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
90	properties.
91
92	Data Path Interface
93	-------------------
94
95	I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
96	SQ correspondingly). Each SQ has a completion queue (CQ) associated
97	with it.
98
99	The SQs and CQs are implemented as descriptor rings in contiguous
100	physical memory.
101
102	Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
103
104	The driver supports multi-queue for both Tx and Rx.
105
106	Configuration information
107	-------------------------
108
109	DPDK Configuration Parameters
110
111	The following configuration options are available for the ENA PMD:
112
113	* CONFIG_RTE_LIBRTE_ENA_PMD (default y): Enables or disables inclusion
114	of the ENA PMD driver in the DPDK compilation.
115
116
117	* CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT (default y): Enables or disables debug
118	logging of device initialization within the ENA PMD driver.
119
120	* CONFIG_RTE_LIBRTE_ENA_DEBUG_RX (default n): Enables or disables debug
121	logging of RX logic within the ENA PMD driver.
122
123	* CONFIG_RTE_LIBRTE_ENA_DEBUG_TX (default n): Enables or disables debug
124	logging of TX logic within the ENA PMD driver.
125
126	* CONFIG_RTE_LIBRTE_ENA_COM_DEBUG (default n): Enables or disables debug
127	logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
128
129	ENA Configuration Parameters
130
131	* Number of Queues
132
133	This is the requested number of queues upon initialization, however, the actual
134	number of receive and transmit queues to be created will be the minimum between
135	the maximal number supported by the device and number of queues requested.
136
137	* Size of Queues
138
139	This is the requested size of receive/transmit queues, while the actual size
140	will be the minimum between the requested size and the maximal receive/transmit
141	supported by the device.
142
143	Building DPDK
144	-------------
145
146	See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
147	instructions on how to build DPDK.
148
149	By default the ENA PMD library will be built into the DPDK library.
150
151	For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
152	documentation that comes with DPDK suite <linux_gsg>`.
153
154	Supported ENA adapters
155	----------------------
156
157	Current ENA PMD supports the following ENA adapters including:
158
159	* ``1d0f:ec20`` - ENA VF
160	* ``1d0f:ec21`` - ENA VF with LLQ support
161
162	Supported Operating Systems
163	---------------------------
164
165	Any Linux distribution fulfilling the conditions described in ``System Requirements``
166	section of :ref:`the DPDK documentation <linux_gsg>` or refer to DPDK Release Notes.
167
168	Supported features
169	------------------
170
171	* Jumbo frames up to 9K
172	* Port Hardware Statistics
173	* IPv4/TCP/UDP checksum offload
174	* TSO offload
175	* Multiple receive and transmit queues
176	* RSS
177	* Low Latency Queue for Tx
178
179	Unsupported features
180	--------------------
181
182	The features supported by the device and not yet supported by this PMD include:
183
184	* Asynchronous Event Notification Queue (AENQ)
185
186	Prerequisites
187	-------------
188
189	#. Prepare the system as recommended by DPDK suite. This includes environment
190	variables, hugepages configuration, tool-chains and configuration
191
192	#. Insert igb_uio kernel module using the command 'modprobe igb_uio'
193
194	#. Bind the intended ENA device to igb_uio module
195
196
197	At this point the system should be ready to run DPDK applications. Once the
198	application runs to completion, the ENA can be detached from igb_uio if necessary.
199
200	Usage example
201	-------------
202
203	This section demonstrates how to launch testpmd with Amazon ENA
204	devices managed by librte_pmd_ena.
205
206	#. Load the kernel modules:
207
208	.. code-block:: console
209
210	modprobe uio
211	insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
212
213	.. note::
214
215	Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module
216
217	#. Mount and request huge pages:
218
219	.. code-block:: console
220
221	mount -t hugetlbfs nodev /mnt/hugepages
222	echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
223
224	#. Bind UIO driver to ENA device (using provided by DPDK binding tool):
225
226	.. code-block:: console
227
228	./tools/dpdk-devbind.py --bind=igb_uio 0000:02:00.1
229
230	#. Start testpmd with basic parameters:
231
232	.. code-block:: console
233
234	./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i
235
236	Example output:
237
238	.. code-block:: console
239
240	[...]
241	EAL: PCI device 0000:02:00.1 on NUMA socket -1
242	EAL: probe driver: 1d0f:ec20 rte_ena_pmd
243	EAL: PCI memory mapped at 0x7f9b6c400000
244	PMD: eth_ena_dev_init(): Initializing 0:2:0.1
245	Interactive-mode selected
246	Configuring Port 0 (socket 0)
247	Port 0: 00:00:00:11:00:01
248	Checking link statuses...
249	Port 0 Link Up - speed 10000 Mbps - full-duplex
250	Done
251	testpmd>