]>
Commit | Line | Data |
---|---|---|
11fdf7f2 | 1 | # Compiling fio |
7c673cae | 2 | |
11fdf7f2 | 3 | First, clone the fio source repository from https://github.com/axboe/fio |
7c673cae | 4 | |
11fdf7f2 | 5 | git clone https://github.com/axboe/fio |
7c673cae | 6 | |
11fdf7f2 | 7 | Then check out the fio 3.3: |
7c673cae | 8 | |
11fdf7f2 | 9 | cd fio && git checkout fio-3.3 |
7c673cae | 10 | |
11fdf7f2 | 11 | Finally, compile the code: |
7c673cae | 12 | |
11fdf7f2 | 13 | make |
7c673cae | 14 | |
11fdf7f2 | 15 | # Compiling SPDK |
7c673cae | 16 | |
11fdf7f2 TL |
17 | First, clone the SPDK source repository from https://github.com/spdk/spdk |
18 | ||
19 | git clone https://github.com/spdk/spdk | |
20 | git submodule update --init | |
21 | ||
22 | Then, run the SPDK configure script to enable fio (point it to the root of the fio repository): | |
23 | ||
24 | cd spdk | |
25 | ./configure --with-fio=/path/to/fio/repo <other configuration options> | |
26 | ||
27 | Finally, build SPDK: | |
28 | ||
29 | make | |
30 | ||
31 | **Note to advanced users**: These steps assume you're using the DPDK submodule. If you are using your | |
32 | own version of DPDK, the fio plugin requires that DPDK be compiled with -fPIC. You can compile DPDK | |
33 | with -fPIC by modifying your DPDK configuration file and adding the line: | |
7c673cae FG |
34 | |
35 | EXTRA_CFLAGS=-fPIC | |
36 | ||
11fdf7f2 TL |
37 | # Usage |
38 | ||
39 | To use the SPDK fio plugin with fio, specify the plugin binary using LD_PRELOAD when running | |
40 | fio and set ioengine=spdk in the fio configuration file (see example_config.fio in the same | |
41 | directory as this README). | |
42 | ||
43 | LD_PRELOAD=<path to spdk repo>/examples/nvme/fio_plugin/fio_plugin fio | |
44 | ||
45 | To select NVMe devices, you pass an SPDK Transport Identifier string as the filename. These are in the | |
46 | form: | |
47 | ||
48 | filename=key=value [key=value] ... ns=value | |
7c673cae | 49 | |
11fdf7f2 | 50 | Specifically, for local PCIe NVMe devices it will look like this: |
7c673cae | 51 | |
11fdf7f2 | 52 | filename=trtype=PCIe traddr=0000.04.00.0 ns=1 |
7c673cae | 53 | |
11fdf7f2 | 54 | And remote devices accessed via NVMe over Fabrics will look like this: |
7c673cae | 55 | |
11fdf7f2 | 56 | filename=trtype=RDMA adrfam=IPv4 traddr=192.168.100.8 trsvcid=4420 ns=1 |
7c673cae | 57 | |
7c673cae | 58 | |
11fdf7f2 TL |
59 | **Note**: The specification of the PCIe address should not use the normal ':' |
60 | and instead only use '.'. This is a limitation in fio - it splits filenames on | |
61 | ':'. Also, the NVMe namespaces start at 1, not 0, and the namespace must be | |
62 | specified at the end of the string. | |
7c673cae | 63 | |
11fdf7f2 TL |
64 | Currently the SPDK fio plugin is limited to the thread usage model, so fio jobs must also specify thread=1 |
65 | when using the SPDK fio plugin. | |
66 | ||
67 | fio also currently has a race condition on shutdown if dynamically loading the ioengine by specifying the | |
68 | engine's full path via the ioengine parameter - LD_PRELOAD is recommended to avoid this race condition. | |
7c673cae FG |
69 | |
70 | When testing random workloads, it is recommended to set norandommap=1. fio's random map | |
71 | processing consumes extra CPU cycles which will degrade performance over time with | |
72 | the fio_plugin since all I/O are submitted and completed on a single CPU core. | |
11fdf7f2 TL |
73 | |
74 | When testing FIO on multiple NVMe SSDs with SPDK plugin, it is recommended to use multiple jobs in FIO configurion. | |
75 | It has been observed that there are some performance gap between FIO(with SPDK plugin enabled) and SPDK perf | |
76 | (examples/nvme/perf/perf) on testing multiple NVMe SSDs. If you use one job(i.e., use one CPU core) configured for | |
77 | FIO test, the performance is worse than SPDK perf (also using one CPU core) against many NVMe SSDs. But if you use | |
78 | multiple jobs for FIO test, the performance of FIO is similiar with SPDK perf. After analyzing this phenomenon, we | |
79 | think that is caused by the FIO architecture. Mainly FIO can scale with multiple threads (i.e., using CPU cores), | |
80 | but it is not good to use one thread against many I/O devices. | |
81 | ||
82 | # End-to-end Data Protection (Optional) | |
83 | ||
84 | Running with PI setting, following settings steps are required. | |
85 | First, format device namespace with proper PI setting. For example: | |
86 | ||
87 | nvme format /dev/nvme0n1 -l 1 -i 1 -p 0 -m 1 | |
88 | ||
89 | In fio configure file, add PRACT and set PRCHK by flags(GUARD|REFTAG|APPTAG) properly. For example: | |
90 | ||
91 | pi_act=0 | |
92 | pi_chk=GUARD | |
93 | ||
94 | Blocksize should be set as the sum of data and metadata. For example, if data blocksize is 512 Byte, host generated | |
95 | PI metadata is 8 Byte, then blocksize in fio configure file should be 520 Byte: | |
96 | ||
97 | bs=520 |