ceph/src/spdk/examples/nvme/fio_plugin/README.md

   1 # Compiling fio
   2
   3 First, clone the fio source repository from https://github.com/axboe/fio
   4
   5     git clone https://github.com/axboe/fio
   6
   7 Then check out the fio 3.3:
   8
   9     cd fio && git checkout fio-3.3
  10
  11 Finally, compile the code:
  12
  13     make
  14
  15 # Compiling SPDK
  16
  17 First, clone the SPDK source repository from https://github.com/spdk/spdk
  18
  19     git clone https://github.com/spdk/spdk
  20     git submodule update --init
  21
  22 Then, run the SPDK configure script to enable fio (point it to the root of the fio repository):
  23
  24     cd spdk
  25     ./configure --with-fio=/path/to/fio/repo <other configuration options>
  26
  27 Finally, build SPDK:
  28
  29     make
  30
  31 **Note to advanced users**: These steps assume you're using the DPDK submodule. If you are using your
  32 own version of DPDK, the fio plugin requires that DPDK be compiled with -fPIC. You can compile DPDK
  33 with -fPIC by modifying your DPDK configuration file and adding the line:
  34
  35     EXTRA_CFLAGS=-fPIC
  36
  37 # Usage
  38
  39 To use the SPDK fio plugin with fio, specify the plugin binary using LD_PRELOAD when running
  40 fio and set ioengine=spdk in the fio configuration file (see example_config.fio in the same
  41 directory as this README).
  42
  43     LD_PRELOAD=<path to spdk repo>/examples/nvme/fio_plugin/fio_plugin fio
  44
  45 To select NVMe devices, you pass an SPDK Transport Identifier string as the filename. These are in the
  46 form:
  47
  48     filename=key=value [key=value] ... ns=value
  49
  50 Specifically, for local PCIe NVMe devices it will look like this:
  51
  52     filename=trtype=PCIe traddr=0000.04.00.0 ns=1
  53
  54 And remote devices accessed via NVMe over Fabrics will look like this:
  55
  56     filename=trtype=RDMA adrfam=IPv4 traddr=192.168.100.8 trsvcid=4420 ns=1
  57
  58
  59 **Note**: The specification of the PCIe address should not use the normal ':'
  60 and instead only use '.'. This is a limitation in fio - it splits filenames on
  61 ':'. Also, the NVMe namespaces start at 1, not 0, and the namespace must be
  62 specified at the end of the string.
  63
  64 Currently the SPDK fio plugin is limited to the thread usage model, so fio jobs must also specify thread=1
  65 when using the SPDK fio plugin.
  66
  67 fio also currently has a race condition on shutdown if dynamically loading the ioengine by specifying the
  68 engine's full path via the ioengine parameter - LD_PRELOAD is recommended to avoid this race condition.
  69
  70 When testing random workloads, it is recommended to set norandommap=1.  fio's random map
  71 processing consumes extra CPU cycles which will degrade performance over time with
  72 the fio_plugin since all I/O are submitted and completed on a single CPU core.
  73
  74 When testing FIO on multiple NVMe SSDs with SPDK plugin, it is recommended to use multiple jobs in FIO configurion.
  75 It has been observed that there are some performance gap between FIO(with SPDK plugin enabled) and SPDK perf
  76 (examples/nvme/perf/perf) on testing multiple NVMe SSDs. If you use one job(i.e., use one CPU core) configured for
  77 FIO test, the performance is worse than SPDK perf (also using one CPU core) against many NVMe SSDs. But if you use
  78 multiple jobs for FIO test, the performance of FIO is similiar with SPDK perf. After analyzing this phenomenon, we
  79 think that is caused by the FIO architecture. Mainly FIO can scale with multiple threads (i.e., using CPU cores),
  80 but it is not good to use one thread against many I/O devices.
  81
  82 # End-to-end Data Protection (Optional)
  83
  84 Running with PI setting, following settings steps are required.
  85 First, format device namespace with proper PI setting. For example:
  86
  87     nvme format /dev/nvme0n1 -l 1 -i 1 -p 0 -m 1
  88
  89 In fio configure file, add PRACT and set PRCHK by flags(GUARD|REFTAG|APPTAG) properly. For example:
  90
  91     pi_act=0
  92     pi_chk=GUARD
  93
  94 Blocksize should be set as the sum of data and metadata. For example, if data blocksize is 512 Byte, host generated
  95 PI metadata is 8 Byte, then blocksize in fio configure file should be 520 Byte:
  96
  97     bs=520