projects / ceph.git / blame
| Commit | Line | Data |
|---|---|---|
| f67539c2 TL |
1 | .. SPDX-License-Identifier: BSD-3-Clause |
| 2 | Copyright 2019 Mellanox Technologies, Ltd | |
| 3 | ||
| 4 | .. include:: <isonum.txt> | |
| 5 | ||
| 6 | MLX5 vDPA driver | |
| 7 | ================ | |
| 8 | ||
| 9 | The MLX5 vDPA (vhost data path acceleration) driver library | |
| 10 | (**librte_pmd_mlx5_vdpa**) provides support for **Mellanox ConnectX-6**, | |
| 11 | **Mellanox ConnectX-6 Dx** and **Mellanox BlueField** families of | |
| 12 | 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in | |
| 13 | SR-IOV context. | |
| 14 | ||
| 15 | .. note:: | |
| 16 | ||
| 17 | Due to external dependencies, this driver is disabled in default | |
| 18 | configuration of the "make" build. It can be enabled with | |
| 19 | ``CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=y`` or by using "meson" build system which | |
| 20 | will detect dependencies. | |
| 21 | ||
| 22 | ||
| 23 | Design | |
| 24 | ------ | |
| 25 | ||
| 26 | For security reasons and robustness, this driver only deals with virtual | |
| 27 | memory addresses. The way resources allocations are handled by the kernel, | |
| 28 | combined with hardware specifications that allow to handle virtual memory | |
| 29 | addresses directly, ensure that DPDK applications cannot access random | |
| 30 | physical memory (or memory that does not belong to the current process). | |
| 31 | ||
| 32 | The PMD can use libibverbs and libmlx5 to access the device firmware | |
| 33 | or directly the hardware components. | |
| 34 | There are different levels of objects and bypassing abilities | |
| 35 | to get the best performances: | |
| 36 | ||
| 37 | - Verbs is a complete high-level generic API | |
| 38 | - Direct Verbs is a device-specific API | |
| 39 | - DevX allows to access firmware objects | |
| 40 | - Direct Rules manages flow steering at low-level hardware layer | |
| 41 | ||
| 42 | Enabling librte_pmd_mlx5_vdpa causes DPDK applications to be linked against | |
| 43 | libibverbs. | |
| 44 | ||
| 45 | A Mellanox mlx5 PCI device can be probed by either net/mlx5 driver or vdpa/mlx5 | |
| 46 | driver but not in parallel. Hence, the user should decide the driver by the | |
| 47 | ``class`` parameter in the device argument list. | |
| 48 | By default, the mlx5 device will be probed by the net/mlx5 driver. | |
| 49 | ||
| 50 | Supported NICs | |
| 51 | -------------- | |
| 52 | ||
| 53 | * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G) | |
| 54 | * Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 100G MCX623106AN-CDAT (2x100G) | |
| 55 | * Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 200G MCX623105AN-VDAT (1x200G) | |
| 56 | * Mellanox\ |reg| BlueField SmartNIC 25G MBF1M332A-ASCAT (2x25G) | |
| 57 | ||
| 58 | Prerequisites | |
| 59 | ------------- | |
| 60 | ||
| 61 | - Mellanox OFED version: **5.0** | |
| 62 | see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. | |
| 63 | ||
| 64 | Compilation options | |
| 65 | ~~~~~~~~~~~~~~~~~~~ | |
| 66 | ||
| 67 | These options can be modified in the ``.config`` file. | |
| 68 | ||
| 69 | - ``CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD`` (default **n**) | |
| 70 | ||
| 71 | Toggle compilation of librte_pmd_mlx5 itself. | |
| 72 | ||
| 73 | - ``CONFIG_RTE_IBVERBS_LINK_DLOPEN`` (default **n**) | |
| 74 | ||
| 75 | Build PMD with additional code to make it loadable without hard | |
| 76 | dependencies on **libibverbs** nor **libmlx5**, which may not be installed | |
| 77 | on the target system. | |
| 78 | ||
| 79 | In this mode, their presence is still required for it to run properly, | |
| 80 | however their absence won't prevent a DPDK application from starting (with | |
| 81 | ``CONFIG_RTE_BUILD_SHARED_LIB`` disabled) and they won't show up as | |
| 82 | missing with ``ldd(1)``. | |
| 83 | ||
| 84 | It works by moving these dependencies to a purpose-built rdma-core "glue" | |
| 85 | plug-in which must either be installed in a directory whose name is based | |
| 86 | on ``CONFIG_RTE_EAL_PMD_PATH`` suffixed with ``-glue`` if set, or in a | |
| 87 | standard location for the dynamic linker (e.g. ``/lib``) if left to the | |
| 88 | default empty string (``""``). | |
| 89 | ||
| 90 | This option has no performance impact. | |
| 91 | ||
| 92 | - ``CONFIG_RTE_IBVERBS_LINK_STATIC`` (default **n**) | |
| 93 | ||
| 94 | Embed static flavor of the dependencies **libibverbs** and **libmlx5** | |
| 95 | in the PMD shared library or the executable static binary. | |
| 96 | ||
| 97 | .. note:: | |
| 98 | ||
| 99 | For BlueField, target should be set to ``arm64-bluefield-linux-gcc``. This | |
| 100 | will enable ``CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD`` and set | |
| 101 | ``RTE_CACHE_LINE_SIZE`` to 64. Default armv8a configuration of make build and | |
| 102 | meson build set it to 128 then brings performance degradation. | |
| 103 | ||
| 104 | Run-time configuration | |
| 105 | ~~~~~~~~~~~~~~~~~~~~~~ | |
| 106 | ||
| 107 | - **ethtool** operations on related kernel interfaces also affect the PMD. | |
| 108 | ||
| 109 | - ``class`` parameter [string] | |
| 110 | ||
| 111 | Select the class of the driver that should probe the device. | |
| 112 | `vdpa` for the mlx5 vDPA driver. | |
| 113 |