]>
Commit | Line | Data |
---|---|---|
f67539c2 TL |
1 | # SPDK Libraries {#libraries} |
2 | ||
3 | The SPDK repository is, first and foremost, a collection of high-performance | |
4 | storage-centric software libraries. With this in mind, much care has been taken | |
5 | to ensure that these libraries have consistent and robust naming and versioning | |
6 | conventions. The libraries themselves are also divided across two directories | |
7 | (`lib` and `module`) inside of the SPDK repository in a deliberate way to prevent | |
8 | mixing of SPDK event framework dependent code and lower level libraries. This document | |
9 | is aimed at explaining the structure, naming conventions, versioning scheme, and use cases | |
10 | of the libraries contained in these two directories. | |
11 | ||
12 | # Directory Structure {#structure} | |
13 | ||
14 | The SPDK libraries are divided into two directories. The `lib` directory contains the base libraries that | |
15 | compose SPDK. Some of these base libraries define plug-in systems. Instances of those plug-ins are called | |
16 | modules and are located in the `module` directory. For example, the `spdk_sock` library is contained in the | |
17 | `lib` directory while the implementations of socket abstractions, `sock_posix`, `sock_uring`, and `sock_vpp` | |
18 | are contained in the `module` directory. | |
19 | ||
20 | ## lib {#lib} | |
21 | ||
22 | The libraries in the `lib` directory can be readily divided into four categories: | |
23 | ||
24 | - Utility Libraries: These libraries contain basic, commonly used functions that make more complex | |
25 | libraries easier to implement. For example, `spdk_log` contains macro definitions that provide a | |
26 | consistent logging paradigm and `spdk_json` is a general purpose JSON parsing library. | |
27 | - Protocol Libraries: These libraries contain the building blocks for a specific service. For example, | |
28 | `spdk_nvmf` and `spdk_vhost` each define the storage protocols after which they are named. | |
29 | - Storage Service Libraries: These libraries provide a specific abstraction that can be mapped to somewhere | |
30 | between the physical drive and the filesystem level of your typical storage stack. For example `spdk_bdev` | |
31 | provides a general block device abstraction layer, `spdk_lvol` provides a logical volume abstraction, | |
32 | `spdk_blobfs` provides a filesystem abstraction, and `spdk_ftl` provides a flash translation layer | |
33 | abstraction. | |
34 | - System Libraries: These libraries provide system level services such as a JSON based RPC service | |
35 | (see `spdk_jsonrpc`) and thread abstractions (see `spdk_thread`). The most notable library in this category | |
36 | is the `spdk_env_dpdk` library which provides a shim for the underlying Data Plane Development Kit (DPDK) | |
37 | environment and provides services like memory management. | |
38 | ||
39 | The one library in the `lib` directory that doesn't fit into the above classification is the `spdk_event` library. | |
40 | This library defines a framework used by the applications contained in the `app` and `example` directories. Much | |
41 | care has been taken to keep the SPDK libraries independent from this framework. The libraries in `lib` are engineered | |
42 | to allow plugging directly into independent application frameworks such as Seastar or libuv with minimal effort. | |
43 | ||
44 | Currently there are two exceptions in the `lib` directory which still rely on `spdk_event`, `spdk_vhost` and `spdk_iscsi`. | |
45 | There are efforts underway to remove all remaining dependencies these libraries have on the `spdk_event` library. | |
46 | ||
47 | Much like the `spdk_event` library, the `spdk_env_dpdk` library has been architected in such a way that it | |
48 | can be readily replaced by an alternate environment shim. More information on replacing the `spdk_env_dpdk` | |
49 | module and the underlying `dpdk` environment can be found in the [environment](#env_replacement) section. | |
50 | ||
51 | ## module {#module} | |
52 | ||
53 | The component libraries in the `module` directory represent specific implementations of the base libraries in | |
54 | the `lib` directory. As with the `lib` directory, much care has been taken to avoid dependencies on the | |
55 | `spdk_event` framework except for those libraries which directly implement the `spdk_event` module plugin system. | |
56 | ||
57 | There are seven sub-directories in the `module` directory which each hold a different class of libraries. These | |
58 | sub-directories can be divided into two types. | |
59 | ||
60 | - plug-in libraries: These libraries are explicitly tied to one of the libraries in the `lib` directory and | |
61 | are registered with that library at runtime by way of a specific constructor function. The parent library in | |
62 | the `lib` directory then manages the module directly. These types of libraries each implement a function table | |
63 | defined by their parent library. The following table shows these directories and their corresponding parent | |
64 | libraries: | |
65 | ||
66 | <center> | |
67 | | module directory | parent library | dependent on event library | | |
68 | |------------------|----------------|----------------------------| | |
69 | | module/accel | spdk_accel | no | | |
70 | | module/bdev | spdk_bdev | no | | |
71 | | module/event | spdk_event | yes | | |
72 | | module/sock | spdk_sock | no | | |
73 | </center> | |
74 | ||
75 | - Free libraries: These libraries are highly dependent upon a library in the `lib` directory but are not | |
76 | explicitly registered to that library via a constructor. The libraries in the `blob`, `blobfs`, and `env_dpdk` | |
77 | directories fall into this category. None of the libraries in this category depend explicitly on the | |
78 | `spdk_event` library. | |
79 | ||
80 | # Library Conventions {#conventions} | |
81 | ||
82 | The SPDK libraries follow strict conventions for naming functions, logging, versioning, and header files. | |
83 | ||
84 | ## Headers {#headers} | |
85 | ||
86 | All public SPDK header files exist in the `include` directory of the SPDK repository. These headers | |
87 | are divided into two sub-directories. | |
88 | ||
89 | `include/spdk` contains headers intended to be used by consumers of the SPDK libraries. All of the | |
90 | functions, variables, and types in these functions are intended for public consumption. Multiple headers | |
91 | in this directory may depend upon the same underlying library and work together to expose different facets | |
92 | of the library. The `spdk_bdev` library, for example, is exposed in three different headers. `bdev_module.h` | |
93 | defines the interfaces a bdev module library would need to implement, `bdev.h` contains general block device | |
94 | functions that would be used by an application consuming block devices exposed by SPDK, and `bdev_zone.h` | |
95 | exposes zoned bdev specific functions. Many of the other libraries exhibit a similar behavior of splitting | |
96 | headers between consumers of the library and those wishing to register a module with that library. | |
97 | ||
98 | `include/spdk_internal`, as its name suggests contains header files intended to be consumed only by other | |
99 | libraries inside of the SPDK repository. These headers are typically used for sharing lower level functions | |
100 | between two libraries that both require similar functions. For example `spdk_internal/nvme_tcp.h` contains | |
101 | low level tcp functions used by both the `spdk_nvme` and `spdk_nvmf` libraries. These headers are *NOT* | |
102 | intended for general consumption. | |
103 | ||
104 | Other header files contained directly in the `lib` and `module` directories are intended to be consumed *only* | |
105 | by source files of their corresponding library. Any symbols intended to be used across libraries need to be | |
106 | included in a header in the `include/spdk_internal` directory. | |
107 | ||
108 | ## Naming Conventions {#naming} | |
109 | ||
110 | All public types and functions in SPDK libraries begin with the prefix `spdk_`. They are also typically | |
111 | further namespaced using the spdk library name. The rest of the function or type name describes its purpose. | |
112 | ||
113 | There are no internal library functions that begin with the `spdk_` prefix. This naming convention is | |
114 | enforced by the SPDK continuous Integration testing. Functions not intended for use outside of their home | |
115 | library should be namespaced with the name of the library only. | |
116 | ||
117 | ## Map Files {#map} | |
118 | ||
119 | SPDK libraries can be built as both static and shared object files. To facilitate building libraries as shared | |
120 | objects, each one has a corresponding map file (e.g. `spdk_nvmf` relies on `spdk_nvmf.map`). SPDK libraries | |
121 | not exporting any symbols rely on a blank map file located at `mk/spdk_blank.map`. | |
122 | ||
123 | # SPDK Shared Objects {#shared_objects} | |
124 | ||
125 | ## Shared Object Versioning {#versioning} | |
126 | ||
127 | SPDK shared objects follow a semantic versioning pattern with a major and minor version. Any changes which | |
128 | break backwards compatibility (symbol removal or change) will cause a shared object major increment and | |
129 | backwards compatible changes will cause a minor version increment; i.e. an application that relies on | |
130 | `libspdk_nvmf.so.3.0` will be compatible with `libspdk_nvmf.so.3.1` but not with `libspdk_nvmf.so.4.0`. | |
131 | ||
132 | Shared object versions are incremented only once between each release cycle. This means that at most, the | |
133 | major version of each SPDK shared library will increment only once between each SPDK release. | |
134 | ||
135 | There are currently no guarantees in SPDK of ABI compatibility between two major SPDK releases. | |
136 | ||
137 | The point releases of an LTS release will be ABI compatible with the corresponding LTS major release. | |
138 | ||
139 | Shared objects are versioned independently of one another. This means that `libspdk_nvme.so.3.0` and | |
140 | `libspdk_bdev.so.3.0` do not necessarily belong to the same release. This also means that shared objects | |
141 | with the same suffix are not necessarily compatible with each other. It is important to source all of your | |
142 | SPDK libraries from the same repository and version to ensure inter-library compatibility. | |
143 | ||
144 | ## Linking to Shared Objects {#so_linking} | |
145 | ||
146 | Shared objects in SPDK are created on a per-library basis. There is a top level `libspdk.so` object | |
147 | which is a linker script. It simply contains references to all of the other spdk shared objects. | |
148 | ||
149 | There are essentially two ways of linking to SPDK libraries. | |
150 | ||
151 | 1. An application can link to the top level shared object library as follows: | |
152 | ~~~{.sh} | |
153 | gcc -o my_app ./my_app.c -lspdk -lspdk_env_dpdk -ldpdk | |
154 | ~~~ | |
155 | ||
156 | 2. An application can link to only a subset of libraries by linking directly to the ones it relies on: | |
157 | ~~~{.sh} | |
158 | gcc -o my_app ./my_app.c -lpassthru_external -lspdk_event_bdev -lspdk_bdev -lspdk_bdev_malloc | |
159 | -lspdk_log -lspdk_thread -lspdk_util -lspdk_event -lspdk_env_dpdk -ldpdk | |
160 | ~~~ | |
161 | ||
162 | In the second instance, please note that applications need only link to the libraries upon which they | |
163 | directly depend. All SPDK libraries have their dependencies specified at object compile time. This means | |
164 | that when linking to `spdk_net`, one does not also have to specify `spdk_log`, `spdk_util`, `spdk_json`, | |
165 | `spdk_jsonrpc`, and `spdk_rpc`. However, this dependency inclusion does not extend to the application | |
166 | itself; i.e. if an application directly uses symbols from both `spdk_bdev` and `spdk_log`, both libraries | |
167 | will need to be supplied to the linker when linking the application even though `spdk_log` is a dependency | |
168 | of `spdk_bdev`. | |
169 | ||
170 | Please also note that when linking to SPDK libraries, both the spdk_env shim library and the env library | |
171 | itself need to be supplied to the linker. In the examples above, these are `spdk_env_dpdk` and `dpdk` | |
172 | respectively. This was intentional and allows one to easily swap out both the environment and the | |
173 | environment shim. | |
174 | ||
175 | ## Replacing the env abstraction {#env_replacement} | |
176 | ||
177 | SPDK depends on an environment abstraction that provides crucial pinned memory management and PCIe | |
178 | bus management operations. The interface for this environment abstraction is defined in the | |
179 | `include/env.h` header file. The default implementation of this environment is located in `spdk_env_dpdk`. | |
180 | This abstraction in turn relies upon the DPDK libraries. This two part implementation was deliberate | |
181 | and allows for easily swapping out the dpdk version upon which the spdk libraries rely without making | |
182 | modifications to the spdk source directly. | |
183 | ||
184 | Any environment can replace the `spdk_env_dpdk` environment by implementing the `include/env.h` header | |
185 | file. The environment can either be implemented wholesale in a single library or as a two-part | |
186 | shim/implementation library system. | |
187 | ~~~{.sh} | |
188 | # single library | |
189 | gcc -o my_app ./my_app.c -lspdk -lcustom_env_implementation | |
190 | ||
191 | # two libraries | |
192 | gcc -o my_app ./my_app.c -lspdk -lcustom_env_shim -lcustom_env_implementation | |
193 | ~~~ |