ceph/src/spdk/doc/libraries.md

   1 # SPDK Libraries {#libraries}
   2
   3 The SPDK repository is, first and foremost, a collection of high-performance
   4 storage-centric software libraries. With this in mind, much care has been taken
   5 to ensure that these libraries have consistent and robust naming and versioning
   6 conventions. The libraries themselves are also divided across two directories
   7 (`lib` and `module`) inside of the SPDK repository in a deliberate way to prevent
   8 mixing of SPDK event framework dependent code and lower level libraries. This document
   9 is aimed at explaining the structure, naming conventions, versioning scheme, and use cases
  10 of the libraries contained in these two directories.
  11
  12 # Directory Structure {#structure}
  13
  14 The SPDK libraries are divided into two directories. The `lib` directory contains the base libraries that
  15 compose SPDK. Some of these base libraries define plug-in systems. Instances of those plug-ins are called
  16 modules and are located in the `module` directory. For example, the `spdk_sock` library is contained in the
  17 `lib` directory while the implementations of socket abstractions, `sock_posix`, `sock_uring`, and `sock_vpp`
  18 are contained in the `module` directory.
  19
  20 ## lib {#lib}
  21
  22 The libraries in the `lib` directory can be readily divided into four categories:
  23
  24 - Utility Libraries: These libraries contain basic, commonly used functions that make more complex
  25 libraries easier to implement. For example, `spdk_log` contains macro definitions that provide a
  26 consistent logging paradigm and `spdk_json` is a general purpose JSON parsing library.
  27 - Protocol Libraries: These libraries contain the building blocks for a specific service. For example,
  28 `spdk_nvmf` and `spdk_vhost` each define the storage protocols after which they are named.
  29 - Storage Service Libraries: These libraries provide a specific abstraction that can be mapped to somewhere
  30 between the physical drive and the filesystem level of your typical storage stack. For example `spdk_bdev`
  31 provides a general block device abstraction layer, `spdk_lvol` provides a logical volume abstraction,
  32 `spdk_blobfs` provides a filesystem abstraction, and `spdk_ftl` provides a flash translation layer
  33 abstraction.
  34 - System Libraries: These libraries provide system level services such as a JSON based RPC service
  35 (see `spdk_jsonrpc`) and thread abstractions (see `spdk_thread`). The most notable library in this category
  36 is the `spdk_env_dpdk` library which provides a shim for the underlying Data Plane Development Kit (DPDK)
  37 environment and provides services like memory management.
  38
  39 The one library in the `lib` directory that doesn't fit into the above classification is the `spdk_event` library.
  40 This library defines a framework used by the applications contained in the `app` and `example` directories. Much
  41 care has been taken to keep the SPDK libraries independent from this framework. The libraries in `lib` are engineered
  42 to allow plugging directly into independent application frameworks such as Seastar or libuv with minimal effort.
  43
  44 Currently there are two exceptions in the `lib` directory which still rely on `spdk_event`, `spdk_vhost` and `spdk_iscsi`.
  45 There are efforts underway to remove all remaining dependencies these libraries have on the `spdk_event` library.
  46
  47 Much like the `spdk_event` library, the `spdk_env_dpdk` library has been architected in such a way that it
  48 can be readily replaced by an alternate environment shim. More information on replacing the `spdk_env_dpdk`
  49 module and the underlying `dpdk` environment can be found in the [environment](#env_replacement) section.
  50
  51 ## module {#module}
  52
  53 The component libraries in the `module` directory represent specific implementations of the base libraries in
  54 the `lib` directory. As with the `lib` directory, much care has been taken to avoid dependencies on the
  55 `spdk_event` framework except for those libraries which directly implement the `spdk_event` module plugin system.
  56
  57 There are seven sub-directories in the `module` directory which each hold a different class of libraries. These
  58 sub-directories can be divided into two types.
  59
  60 - plug-in libraries: These libraries are explicitly tied to one of the libraries in the `lib` directory and
  61 are registered with that library at runtime by way of a specific constructor function. The parent library in
  62 the `lib` directory then manages the module directly. These types of libraries each implement a function table
  63 defined by their parent library. The following table shows these directories and their corresponding parent
  64 libraries:
  65
  66 <center>
  67 | module directory | parent library | dependent on event library |
  68 |------------------|----------------|----------------------------|
  69 | module/accel     | spdk_accel     | no                         |
  70 | module/bdev      | spdk_bdev      | no                         |
  71 | module/event     | spdk_event     | yes                        |
  72 | module/sock      | spdk_sock      | no                         |
  73 </center>
  74
  75 - Free libraries: These libraries are highly dependent upon a library in the `lib` directory but are not
  76 explicitly registered to that library via a constructor. The libraries in the `blob`, `blobfs`, and `env_dpdk`
  77 directories fall into this category. None of the libraries in this category depend explicitly on the
  78 `spdk_event` library.
  79
  80 # Library Conventions {#conventions}
  81
  82 The SPDK libraries follow strict conventions for naming functions, logging, versioning, and header files.
  83
  84 ## Headers {#headers}
  85
  86 All public SPDK header files exist in the `include` directory of the SPDK repository. These headers
  87 are divided into two sub-directories.
  88
  89 `include/spdk` contains headers intended to be used by consumers of the SPDK libraries. All of the
  90 functions, variables, and types in these functions are intended for public consumption. Multiple headers
  91 in this directory may depend upon the same underlying library and work together to expose different facets
  92 of the library. The `spdk_bdev` library, for example, is exposed in three different headers. `bdev_module.h`
  93 defines the interfaces a bdev module library would need to implement, `bdev.h` contains general block device
  94 functions that would be used by an application consuming block devices exposed by SPDK, and `bdev_zone.h`
  95 exposes zoned bdev specific functions. Many of the other libraries exhibit a similar behavior of splitting
  96 headers between consumers of the library and those wishing to register a module with that library.
  97
  98 `include/spdk_internal`, as its name suggests contains header files intended to be consumed only by other
  99 libraries inside of the SPDK repository. These headers are typically used for sharing lower level functions
 100 between two libraries that both require similar functions. For example `spdk_internal/nvme_tcp.h` contains
 101 low level tcp functions used by both the `spdk_nvme` and `spdk_nvmf` libraries. These headers are *NOT*
 102 intended for general consumption.
 103
 104 Other header files contained directly in the `lib` and `module` directories are intended to be consumed *only*
 105 by source files of their corresponding library. Any symbols intended to be used across libraries need to be
 106 included in a header in the `include/spdk_internal` directory.
 107
 108 ## Naming Conventions {#naming}
 109
 110 All public types and functions in SPDK libraries begin with the prefix `spdk_`. They are also typically
 111 further namespaced using the spdk library name. The rest of the function or type name describes its purpose.
 112
 113 There are no internal library functions that begin with the `spdk_` prefix. This naming convention is
 114 enforced by the SPDK continuous Integration testing. Functions not intended for use outside of their home
 115 library should be namespaced with the name of the library only.
 116
 117 ## Map Files {#map}
 118
 119 SPDK libraries can be built as both static and shared object files. To facilitate building libraries as shared
 120 objects, each one has a corresponding map file (e.g. `spdk_nvmf` relies on `spdk_nvmf.map`). SPDK libraries
 121 not exporting any symbols rely on a blank map file located at `mk/spdk_blank.map`.
 122
 123 # SPDK Shared Objects {#shared_objects}
 124
 125 ## Shared Object Versioning {#versioning}
 126
 127 SPDK shared objects follow a semantic versioning pattern with a major and minor version. Any changes which
 128 break backwards compatibility (symbol removal or change) will cause a shared object major increment and
 129 backwards compatible changes will cause a minor version increment; i.e. an application that relies on
 130 `libspdk_nvmf.so.3.0` will be compatible with `libspdk_nvmf.so.3.1` but not with `libspdk_nvmf.so.4.0`.
 131
 132 Shared object versions are incremented only once between each release cycle. This means that at most, the
 133 major version of each SPDK shared library will increment only once between each SPDK release.
 134
 135 There are currently no guarantees in SPDK of ABI compatibility between two major SPDK releases.
 136
 137 The point releases of an LTS release will be ABI compatible with the corresponding LTS major release.
 138
 139 Shared objects are versioned independently of one another. This means that `libspdk_nvme.so.3.0` and
 140 `libspdk_bdev.so.3.0` do not necessarily belong to the same release. This also means that shared objects
 141 with the same suffix are not necessarily compatible with each other. It is important to source all of your
 142 SPDK libraries from the same repository and version to ensure inter-library compatibility.
 143
 144 ## Linking to Shared Objects {#so_linking}
 145
 146 Shared objects in SPDK are created on a per-library basis. There is a top level `libspdk.so` object
 147 which is a linker script. It simply contains references to all of the other spdk shared objects.
 148
 149 There are essentially two ways of linking to SPDK libraries.
 150
 151 1. An application can link to the top level shared object library as follows:
 152 ~~~{.sh}
 153         gcc -o my_app ./my_app.c -lspdk -lspdk_env_dpdk -ldpdk
 154 ~~~
 155
 156 2. An application can link to only a subset of libraries by linking directly to the ones it relies on:
 157 ~~~{.sh}
 158         gcc -o my_app ./my_app.c -lpassthru_external -lspdk_event_bdev -lspdk_bdev -lspdk_bdev_malloc
 159         -lspdk_log -lspdk_thread -lspdk_util -lspdk_event -lspdk_env_dpdk -ldpdk
 160 ~~~
 161
 162 In the second instance, please note that applications need only link to the libraries upon which they
 163 directly depend. All SPDK libraries have their dependencies specified at object compile time. This means
 164 that when linking to `spdk_net`, one does not also have to specify `spdk_log`, `spdk_util`, `spdk_json`,
 165 `spdk_jsonrpc`, and `spdk_rpc`. However, this dependency inclusion does not extend to the application
 166 itself; i.e. if an application directly uses symbols from both `spdk_bdev` and `spdk_log`, both libraries
 167 will need to be supplied to the linker when linking the application even though `spdk_log` is a dependency
 168 of `spdk_bdev`.
 169
 170 Please also note that when linking to SPDK libraries, both the spdk_env shim library and the env library
 171 itself need to be supplied to the linker. In the examples above, these are `spdk_env_dpdk` and `dpdk`
 172 respectively. This was intentional and allows one to easily swap out both the environment and the
 173 environment shim.
 174
 175 ## Replacing the env abstraction {#env_replacement}
 176
 177 SPDK depends on an environment abstraction that provides crucial pinned memory management and PCIe
 178 bus management operations. The interface for this environment abstraction is defined in the
 179 `include/env.h` header file. The default implementation of this environment is located in `spdk_env_dpdk`.
 180 This abstraction in turn relies upon the DPDK libraries. This two part implementation was deliberate
 181 and allows for easily swapping out the dpdk version upon which the spdk libraries rely without making
 182 modifications to the spdk source directly.
 183
 184 Any environment can replace the `spdk_env_dpdk` environment by implementing the `include/env.h` header
 185 file. The environment can either be implemented wholesale in a single library or as a two-part
 186 shim/implementation library system.
 187 ~~~{.sh}
 188         # single library
 189         gcc -o my_app ./my_app.c -lspdk -lcustom_env_implementation
 190
 191         # two libraries
 192         gcc -o my_app ./my_app.c -lspdk -lcustom_env_shim -lcustom_env_implementation
 193 ~~~