ceph/src/spdk/doc/applications.md

   1
   2 # An Overview of SPDK Applications {#app_overview}
   3
   4 SPDK is primarily a development kit that delivers libraries and header files for
   5 use in other applications. However, SPDK also contains a number of applications.
   6 These applications are primarily used to test the libraries, but many are full
   7 featured and high quality. The major applications in SPDK are:
   8
   9 - @ref iscsi
  10 - @ref nvmf
  11 - @ref vhost
  12 - SPDK Target (a unified application combining the above three)
  13
  14 There are also a number of tools and examples in the `examples` directory.
  15
  16 The SPDK targets are all based on a common framework so they have much in
  17 common. The framework defines a concept called a `subsystem` and all
  18 functionality is implemented in various subsystems. Subsystems have a unified
  19 initialization and teardown path.
  20
  21 # Configuring SPDK Applications {#app_config}
  22
  23 ## Command Line Parameters {#app_cmd_line_args}
  24
  25 The SPDK application framework defines a set of base command line flags for all
  26 applications that use it. Specific applications may implement additional flags.
  27
  28 Param    | Long Param             | Type     | Default                | Description
  29 -------- | ---------------------- | -------- | ---------------------- | -----------
  30 -c       | --config               | string   |                        | @ref cmd_arg_config_file
  31 -d       | --limit-coredump       | flag     | false                  | @ref cmd_arg_limit_coredump
  32 -e       | --tpoint-group-mask    | integer  | 0x0                    | @ref cmd_arg_limit_tpoint_group_mask
  33 -g       | --single-file-segments | flag     |                        | @ref cmd_arg_single_file_segments
  34 -h       | --help                 | flag     |                        | show all available parameters and exit
  35 -i       | --shm-id               | integer  |                        | @ref cmd_arg_multi_process
  36 -m       | --cpumask              | CPU mask | 0x1                    | application @ref cpu_mask
  37 -n       | --mem-channels         | integer  | all channels           | number of memory channels used for DPDK
  38 -p       | --master-core          | integer  | first core in CPU mask | master (primary) core for DPDK
  39 -r       | --rpc-socket           | string   | /var/tmp/spdk.sock     | RPC listen address
  40 -s       | --mem-size             | integer  | all hugepage memory    | @ref cmd_arg_memory_size
  41 |        | --silence-noticelog    | flag     |                        | disable notice level logging to `stderr`
  42 -u       | --no-pci               | flag     |                        | @ref cmd_arg_disable_pci_access.
  43 |        | --wait-for-rpc         | flag     |                        | @ref cmd_arg_deferred_initialization
  44 -B       | --pci-blacklist        | B:D:F    |                        | @ref cmd_arg_pci_blacklist_whitelist.
  45 -W       | --pci-whitelist        | B:D:F    |                        | @ref cmd_arg_pci_blacklist_whitelist.
  46 -R       | --huge-unlink          | flag     |                        | @ref cmd_arg_huge_unlink
  47 |        | --huge-dir             | string   | the first discovered   | allocate hugepages from a specific mount
  48 -L       | --logflag              | string   |                        | @ref cmd_arg_debug_log_flags
  49
  50
  51 ### Configuration file {#cmd_arg_config_file}
  52
  53 Historically, the SPDK applications were configured using a configuration file.
  54 This is still supported, but is considered deprecated in favor of JSON RPC
  55 configuration. See @ref jsonrpc for details.
  56
  57 Note that `--config` and `--wait-for-rpc` cannot be used at the same time.
  58
  59 ### Limit coredump {#cmd_arg_limit_coredump}
  60
  61 By default, an SPDK application will set resource limits for core file sizes
  62 to RLIM_INFINITY.  Specifying `--limit-coredump` will not set the resource limits.
  63
  64 ### Tracepoint group mask {#cmd_arg_limit_tpoint_group_mask}
  65
  66 SPDK has an experimental low overhead tracing framework.  Tracepoints in this
  67 framework are organized into tracepoint groups.  By default, all tracepoint
  68 groups are disabled.  `--tpoint-group-mask` can be used to enable a specific
  69 subset of tracepoint groups in the application.
  70
  71 Note: Additional documentation on the tracepoint framework is in progress.
  72
  73 ### Deferred initialization {#cmd_arg_deferred_initialization}
  74
  75 SPDK applications progress through a set of states beginning with `STARTUP` and
  76 ending with `RUNTIME`.
  77
  78 If the `--wait-for-rpc` parameter is provided SPDK will pause just before starting
  79 subsystem initialization. This state is called `STARTUP`. The JSON RPC server is
  80 ready but only a small subsystem of commands are available to set up initialization
  81 parameters. Those parameters can't be changed after the SPDK application enters
  82 `RUNTIME` state. When the client finishes configuring the SPDK subsystems it
  83 needs to issue the @ref rpc_start_subsystem_init RPC command to begin the
  84 initialization process. After `rpc_start_subsystem_init` returns `true` SPDK
  85 will enter the `RUNTIME` state and the list of available commands becomes much
  86 larger.
  87
  88 To see which RPC methods are available in the current state, issue the
  89 `rpc_get_methods` with the parameter `current` set to `true`.
  90
  91 For more details see @ref jsonrpc documentation.
  92
  93 ### Create just one hugetlbfs file {#cmd_arg_single_file_segments}
  94
  95 Instead of creating one hugetlbfs file per page, this option makes SPDK create
  96 one file per hugepages per socket. This is needed for @ref virtio to be used
  97 with more than 8 hugepages. See @ref virtio_2mb.
  98
  99 ### Multi process mode {#cmd_arg_multi_process}
 100
 101 When `--shm-id` is specified, the application is started in multi-process mode.
 102 Applications using the same shm-id share their memory and
 103 [NVMe devices](@ref nvme_multi_process). The first app to start with a given id
 104 becomes a primary process, with the rest, called secondary processes, only
 105 attaching to it. When the primary process exits, the secondary ones continue to
 106 operate, but no new processes can be attached at this point. All processes within
 107 the same shm-id group must use the same
 108 [--single-file-segments setting](@ref cmd_arg_single_file_segments).
 109
 110 ### Memory size {#cmd_arg_memory_size}
 111
 112 Total size of the hugepage memory to reserve. If DPDK env layer is used, it will
 113 reserve memory from all available hugetlbfs mounts, starting with the one with
 114 the highest page size. This option accepts a number of bytes with a possible
 115 binary prefix, e.g. 1024, 1024M, 1G. The default unit is megabyte.
 116
 117 Starting with DPDK 18.05.1, it's possible to reserve hugepages at runtime, meaning
 118 that SPDK application can be started with 0 pre-reserved memory. Unlike hugepages
 119 pre-reserved at the application startup, the hugepages reserved at runtime will be
 120 released to the system as soon as they're no longer used.
 121
 122 ### Disable PCI access {#cmd_arg_disable_pci_access}
 123
 124 If SPDK is run with PCI access disabled it won't detect any PCI devices. This
 125 includes primarily NVMe and IOAT devices. Also, the VFIO and UIO kernel modules
 126 are not required in this mode.
 127
 128 ### PCI address blacklist and whitelist {#cmd_arg_pci_blacklist_whitelist}
 129
 130 If blacklist is used, then all devices with the provided PCI address will be
 131 ignored. If a whitelist is used, only whitelisted devices will be probed.
 132 `-B` or `-W` can be used more than once, but cannot be mixed together. That is,
 133 `-B` and `-W` cannot be used at the same time.
 134
 135 ### Unlink hugepage files after initialization {#cmd_arg_huge_unlink}
 136
 137 By default, each DPDK-based application tries to remove any orphaned hugetlbfs
 138 files during its initialization. This option removes hugetlbfs files of the current
 139 process as soon as they're created, but is not compatible with `--shm-id`.
 140
 141 ### Debug log {#cmd_arg_debug_log_flags}
 142
 143 Enable a specific debug log type. This option can be used more than once. A list of
 144 all available types is provided in the `--help` output, with `--logflag all`
 145 enabling all of them. Debug logs are only available in debug builds of SPDK.
 146
 147 ## CPU mask {#cpu_mask}
 148
 149 Whenever the `CPU mask` is mentioned it is a string in one of the following formats:
 150
 151 - Case insensitive hexadecimal string with or without "0x" prefix.
 152 - Comma separated list of CPUs or list of CPU ranges. Use '-' to define range.
 153
 154 ### Example
 155
 156 The following CPU masks are equal and correspond to CPUs 0, 1, 2, 8, 9, 10, 11 and 12:
 157
 158 ~~~
 159 0x1f07
 160 0x1F07
 161 1f07
 162 [0,1,2,8-12]
 163 [0, 1, 2, 8, 9, 10, 11, 12]
 164 ~~~