ceph/src/spdk/dpdk/doc/guides/rawdevs/ntb.rst

   1 ..  SPDX-License-Identifier: BSD-3-Clause
   2     Copyright(c) 2018 Intel Corporation.
   3
   4 NTB Rawdev Driver
   5 =================
   6
   7 The ``ntb`` rawdev driver provides a non-transparent bridge between two
   8 separate hosts so that they can communicate with each other. Thus, many
   9 user cases can benefit from this, such as fault tolerance and visual
  10 acceleration.
  11
  12 This PMD allows two hosts to handshake for device start and stop, memory
  13 allocation for the peer to access and read/write allocated memory from peer.
  14 Also, the PMD allows to use doorbell registers to notify the peer and share
  15 some information by using scratchpad registers.
  16
  17 BIOS setting on Intel Skylake
  18 -----------------------------
  19
  20 Intel Non-transparent Bridge needs special BIOS setting. Since the PMD only
  21 supports Intel Skylake platform, introduce BIOS setting here. The referencce
  22 is https://www.intel.com/content/dam/support/us/en/documents/server-products/Intel_Xeon_Processor_Scalable_Family_BIOS_User_Guide.pdf
  23
  24 - Set the needed PCIe port as NTB to NTB mode on both hosts.
  25 - Enable NTB bars and set bar size of bar 23 and bar 45 as 12-29 (2K-512M)
  26   on both hosts. Note that bar size on both hosts should be the same.
  27 - Disable split bars for both hosts.
  28 - Set crosslink control override as DSD/USP on one host, USD/DSP on
  29   another host.
  30 - Disable PCIe PII SSC (Spread Spectrum Clocking) for both hosts. This
  31   is a hardware requirement.
  32
  33 Build Options
  34 -------------
  35
  36 - ``CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV`` (default ``y``)
  37
  38    Toggle compilation of the ``ntb`` driver.
  39
  40 Device Setup
  41 ------------
  42
  43 The Intel NTB devices need to be bound to a DPDK-supported kernel driver
  44 to use, i.e. igb_uio, vfio. The ``dpdk-devbind.py`` script can be used to
  45 show devices status and to bind them to a suitable kernel driver. They will
  46 appear under the category of "Misc (rawdev) devices".
  47
  48 Prerequisites
  49 -------------
  50
  51 NTB PMD needs kernel PCI driver to support write combining (WC) to get
  52 better performance. The difference will be more than 10 times.
  53 To enable WC, there are 2 ways.
  54
  55 - Insert igb_uio with ``wc_activate=1`` flag if use igb_uio driver.
  56
  57 .. code-block:: console
  58
  59   insmod igb_uio.ko wc_activate=1
  60
  61 - Enable WC for NTB device's Bar 2 and Bar 4 (Mapped memory) manually.
  62   The reference is https://www.kernel.org/doc/html/latest/x86/mtrr.html
  63   Get bar base address using ``lspci -vvv -s ae:00.0 | grep Region``.
  64
  65 .. code-block:: console
  66
  67   # lspci -vvv -s ae:00.0 | grep Region
  68   Region 0: Memory at 39bfe0000000 (64-bit, prefetchable) [size=64K]
  69   Region 2: Memory at 39bfa0000000 (64-bit, prefetchable) [size=512M]
  70   Region 4: Memory at 39bfc0000000 (64-bit, prefetchable) [size=512M]
  71
  72 Using the following command to enable WC.
  73
  74 .. code-block:: console
  75
  76   echo "base=0x39bfa0000000 size=0x20000000 type=write-combining" >> /proc/mtrr
  77   echo "base=0x39bfc0000000 size=0x20000000 type=write-combining" >> /proc/mtrr
  78
  79 And the results:
  80
  81 .. code-block:: console
  82
  83   # cat /proc/mtrr
  84   reg00: base=0x000000000 (    0MB), size= 2048MB, count=1: write-back
  85   reg01: base=0x07f000000 ( 2032MB), size=   16MB, count=1: uncachable
  86   reg02: base=0x39bfa0000000 (60553728MB), size=  512MB, count=1: write-combining
  87   reg03: base=0x39bfc0000000 (60554240MB), size=  512MB, count=1: write-combining
  88
  89 To disable WC for these regions, using the following.
  90
  91 .. code-block:: console
  92
  93      echo "disable=2" >> /proc/mtrr
  94      echo "disable=3" >> /proc/mtrr
  95
  96 Ring Layout
  97 -----------
  98
  99 Since read/write remote system's memory are through PCI bus, remote read
 100 is much more expensive than remote write. Thus, the enqueue and dequeue
 101 based on ntb ring should avoid remote read. The ring layout for ntb is
 102 like the following:
 103
 104 - Ring Format::
 105
 106    desc_ring:
 107
 108       0               16                                              64
 109       +---------------------------------------------------------------+
 110       |                        buffer address                         |
 111       +---------------+-----------------------------------------------+
 112       | buffer length |                      resv                     |
 113       +---------------+-----------------------------------------------+
 114
 115    used_ring:
 116
 117       0               16              32
 118       +---------------+---------------+
 119       | packet length |     flags     |
 120       +---------------+---------------+
 121
 122 - Ring Layout::
 123
 124       +------------------------+   +------------------------+
 125       | used_ring              |   | desc_ring              |
 126       | +---+                  |   | +---+                  |
 127       | |   |                  |   | |   |                  |
 128       | +---+      +--------+  |   | +---+                  |
 129       | |   | ---> | buffer | <+---+-|   |                  |
 130       | +---+      +--------+  |   | +---+                  |
 131       | |   |                  |   | |   |                  |
 132       | +---+                  |   | +---+                  |
 133       |  ...                   |   |  ...                   |
 134       |                        |   |                        |
 135       |            +---------+ |   |            +---------+ |
 136       |            | tx_tail | |   |            | rx_tail | |
 137       | System A   +---------+ |   | System B   +---------+ |
 138       +------------------------+   +------------------------+
 139                     <---------traffic---------
 140
 141 - Enqueue and Dequeue
 142   Based on this ring layout, enqueue reads rx_tail to get how many free
 143   buffers and writes used_ring and tx_tail to tell the peer which buffers
 144   are filled with data.
 145   And dequeue reads tx_tail to get how many packets are arrived, and
 146   writes desc_ring and rx_tail to tell the peer about the new allocated
 147   buffers.
 148   So in this way, only remote write happens and remote read can be avoid
 149   to get better performance.
 150
 151 Limitation
 152 ----------
 153
 154 - This PMD only supports Intel Skylake platform.