ceph/src/spdk/doc/peer_2_peer.md

   1 # Peer-2-Peer DMAs {#peer_2_peer}
   2
   3 Please note that the functionality discussed in this document is
   4 currently tagged as experimental.
   5
   6 # In this document {#p2p_toc}
   7
   8 * @ref p2p_overview
   9 * @ref p2p_nvme_api
  10 * @ref p2p_cmb_copy
  11 * @ref p2p_issues
  12
  13 # Overview {#p2p_overview}
  14
  15 Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI
  16 End Point (EP) to another without using a system memory buffer. The
  17 most obvious example of this from an SPDK perspective is using a NVMe
  18 Controller Memory Buffer (CMB) to enable direct copies of data between
  19 two NVMe SSDs.
  20
  21 In this section of documentation we outline how to perform P2P
  22 operations in SPDK and outline some of the issues that can occur when
  23 performing P2P operations.
  24
  25 # The P2P API for NVMe {#p2p_nvme_api}
  26
  27 The functions that provide access to the NVMe CMBs for P2P
  28 capabilities are given in the table below.
  29
  30 Key Functions                               | Description
  31 ------------------------------------------- | -----------
  32 spdk_nvme_ctrlr_alloc_cmb_io_buffer()       | @copybrief spdk_nvme_ctrlr_alloc_cmb_io_buffer()
  33 spdk_nvme_ctrlr_free_cmb_io_buffer()        | @copybrief spdk_nvme_ctrlr_free_cmb_io_buffer()
  34 spdk_nvme_ctrlr_get_regs_cmbsz()            | @copybrief spdk_nvme_ctrlr_get_regs_cmbsz()
  35
  36 # Determining device support {#p2p_support}
  37
  38 SPDK's identify example application displays whether a device has a controller
  39 memory buffer and which operations it supports. Run it as follows:
  40
  41 ~~~{.sh}
  42 ./examples/nvme/identify/identify -r traddr:<pci id of ssd>
  43 ~~~
  44
  45 # cmb_copy: An example P2P Application {#p2p_cmb_copy}
  46
  47 Run the cmb_copy example application.
  48
  49 ~~~{.sh}
  50 ./examples/nvme/cmb_copy/cmb_copy -r <pci id of write ssd>-1-0-1 -w <pci id of write ssd>-1-0-1 -c <pci id of the ssd with cmb>
  51 ~~~
  52 This should copy a single LBA (LBA 0) from namespace 1 on the read
  53 NVMe SSD to LBA 0 on namespace 1 on the write SSD using the CMB as the
  54 DMA buffer.
  55
  56 # Issues with P2P {#p2p_issues}
  57
  58 * In some systems when performing peer-2-peer DMAs between PCIe EPs
  59   that are directly connected to the Root Complex (RC) the DMA may
  60   fail or the performance may not be great. Basically your milage may
  61   vary. It is recommended that you use a PCIe switch (such as those
  62   provided by Broadcom or Microsemi) as that is know to provide good
  63   performance.
  64 * Even with a PCIe switch there may be occasions where peer-2-peer
  65   DMAs fail to work. This is probably due to PCIe Access Control
  66   Services (ACS) being enabled by the BIOS and/or OS. You can disable
  67   ACS using setpci or via out of tree kernel patches that can be found
  68   on the internet.
  69 * In more complex topologies involving several switches it may be
  70   possible to construct multiple paths between EPs. This could lead to
  71   TLP ordering problems. If you are working in these environments be
  72   careful!