]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | # Changelog |
2 | ||
f67539c2 TL |
3 | ## v20.07: |
4 | ||
5 | ### accel | |
6 | ||
7 | A new API was added `spdk_accel_get_capabilities` that allows applications to | |
8 | query the capabilities of the currently enabled accel engine back-end. | |
9 | ||
10 | A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. | |
11 | ||
12 | The software accel engine implemenation has added support for CRC-32C. | |
13 | ||
14 | A new capability, compare, was added via `spdk_accel_submit_compare`. | |
15 | ||
16 | The software accel engine implemenation has added support for compare. | |
17 | ||
18 | Several APIs were added to `accel_engine.h` to support batched submission | |
19 | of operations. | |
20 | ||
21 | Several APIs were added to `accel_engine.h` to support dualcast operations. | |
22 | ||
23 | ### accel_fw | |
24 | ||
25 | The accel_fw was updated to no longer require the app to allocate an | |
26 | accel_task on its behalf. All public APIs now take a callback arg as | |
27 | the parameter that used to be the accel_task. | |
28 | ||
29 | The accel_fw API `spdk_accel_task_size` was removed as it is no longer | |
30 | required. | |
31 | ||
32 | The accel_fw was updated to support compare, dualcast, crc32c. | |
33 | ||
34 | The accel_fw introduced batching support for all commands in all plug-ins. | |
35 | See docs for detailed information. | |
36 | ||
37 | ### bdev | |
38 | ||
39 | A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os | |
40 | whose callback context match to the bdev on the given channel. | |
41 | ||
42 | ### build | |
43 | ||
44 | The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. | |
45 | Existing fio configuration files will need to be updated. | |
46 | ||
47 | ### dpdk | |
48 | ||
49 | Updated DPDK submodule to DPDK 20.05. | |
50 | ||
51 | ### env | |
52 | ||
53 | Several new APIs have been added to provide greater flexibility in registering and | |
54 | accessing polled mode PCI drivers. See `env.h` for more details. | |
55 | ||
56 | ### idxd | |
57 | ||
58 | The idxd library and plug-in module for the accel_fw were updated to support | |
59 | all accel_fw commands as well as batching. Batching is supported both | |
60 | through the library and the plug-in module. | |
61 | ||
62 | IDXD engine support for CRC-32C has been added. | |
63 | ||
64 | ### ioat | |
65 | ||
66 | A new API `spdk_ioat_get_max_descriptors` was added. | |
67 | ||
68 | ### nvme | |
69 | ||
70 | An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure | |
71 | to solve the ABI compatiblity issue between different SPDK version. | |
72 | ||
73 | A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted | |
74 | commands whose callback argument match. | |
75 | ||
76 | Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added | |
77 | to the public API. | |
78 | ||
79 | A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe | |
80 | controller. | |
81 | ||
82 | A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl | |
83 | implementations of the transport interface will have to implement that function. | |
84 | ||
85 | ### nvmf | |
86 | ||
87 | The NVMe-oF target no longer supports connecting scheduling configuration and instead | |
88 | always uses what was previously called "transport" scheduling. | |
89 | ||
90 | `spdk_nvmf_tgt_accept` no longer takes a function pointer as an argument. New connections | |
91 | are automatically assigned to poll groups by the underlying transport. Further, | |
92 | `spdk_nvmf_transport_ops` has changed such that the accept function pointer no longer | |
93 | takes a function pointer as an argument. Instead, transports should call | |
94 | `spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. | |
95 | ||
96 | The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, | |
97 | the NVMe-oF target could abort only Asynchronous Event Request commands. | |
98 | ||
99 | ### rdma | |
100 | ||
101 | A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. | |
102 | Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly | |
103 | using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). | |
104 | Using mlx5_dv requires libmlx5 installed on the system. | |
105 | ||
106 | ### rpc | |
107 | ||
108 | Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the | |
109 | rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. | |
110 | ||
111 | Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. | |
112 | ||
113 | Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server | |
114 | must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing | |
115 | proper authentication (if needed) is now a responsibility of the user. | |
116 | ||
117 | The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function | |
118 | of bdev modules. | |
119 | ||
120 | New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. | |
121 | See `sock` section for more details. | |
122 | ||
123 | ### sock | |
124 | ||
125 | Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration | |
126 | options. Options can be set independently for each implementation. | |
127 | ||
128 | Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. | |
129 | ||
130 | Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only | |
131 | available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring | |
132 | based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. | |
133 | ||
134 | Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. | |
135 | New option is used only in posix implementation. | |
136 | ||
137 | Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. | |
138 | New option is used only in posix implementation. | |
139 | ||
140 | ### util | |
141 | ||
142 | Some previously exposed CRC32 functions have been removed from the public API - | |
143 | `spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. | |
144 | ||
145 | ### vhost | |
146 | ||
147 | The function `spdk_vhost_blk_get_dev` has been removed. | |
148 | ||
149 | ## v20.04: | |
150 | ||
151 | IDXD engine support for compare has been added. | |
152 | ||
153 | ### configuration | |
154 | ||
155 | Legacy INI style configuration for SPDK applications has been deprecated and will be | |
156 | removed in future release. | |
157 | Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. | |
158 | ||
159 | ### copy | |
160 | ||
161 | The copy engine library, modules and public APIs have been renamed. Use of the word `copy` | |
162 | has been replaced with the word `accel` short for accelerator in preparation for adding new | |
163 | capabilities in the future. Additionally, APIs for what was previously called the `memcpy` | |
164 | engine have been renamed to identify the engine as a software accelerator. | |
165 | ||
166 | ### crypto | |
167 | ||
168 | Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC | |
169 | `bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either | |
170 | AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. | |
171 | The key2 parameter is the second key required for AES_XTS. | |
172 | ||
173 | ### event | |
174 | ||
175 | Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. | |
176 | ||
177 | ### idxd | |
178 | ||
179 | IDXD support was added in the form of a low level library that can directly | |
180 | utilize the IDXD offload engine as well as an accel plug-in module allowing for use | |
181 | with the generic accel framework API. IDXD is the first in a family of offload | |
182 | engines that share the same interface, specifically DSA is added here. More info | |
183 | can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator | |
184 | ||
185 | Much of the implementation models IOAT, however the low level interfaces are very | |
186 | different. The RPC to enable IDXD requires a configuration number as well. The | |
187 | code includes two pre-defined configurations of IDXD groups/work queues/engines. A future | |
188 | version will provide an interface to allow for setting of individual configuration | |
189 | parameters. | |
190 | ||
191 | IDXD is not yet available so this feature should be considered experimental. It will | |
192 | be built up with additional documentation as an ongoing activity. | |
193 | ||
194 | ### iscsi | |
195 | ||
196 | The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool | |
197 | of lightweight threads already exist at start up time. A poll group is a collection of | |
198 | unrelated iSCSI connections. Each poll group is only accessed from the associated | |
199 | lightweight thread. | |
200 | ||
201 | ### ftl | |
202 | ||
203 | Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and | |
204 | `spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. | |
205 | ||
206 | ### miscellaneous | |
207 | ||
208 | The `--json-ignore-init-errors` command line parameter has been added to ignore | |
209 | initialization errors on JSON config load. | |
210 | ||
211 | The public header file io_channel.h has been removed. Please use thread.h which has the | |
212 | exact same API. | |
213 | ||
214 | ### nvme | |
215 | ||
216 | Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" | |
217 | prefix. | |
218 | ||
219 | Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority | |
220 | of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. | |
221 | ||
222 | A new poll group API has been added to allow for pooling of nvme qpairs across a single | |
223 | entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` | |
224 | family of functions. As a result of this new API, all NVMe transports are expected to implement | |
225 | several poll group related functions. | |
226 | ||
227 | A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag | |
228 | allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair | |
229 | within the context of that call. | |
230 | ||
231 | As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` | |
232 | and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created | |
233 | qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without | |
234 | destroying them (for example to disconnect a qpair before migrating it to a new poll group and | |
235 | reconnecting it). | |
236 | ||
237 | The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` | |
238 | have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. | |
239 | ||
240 | An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving | |
241 | the entire size of the controller memory buffer for data transfer. | |
242 | ||
243 | ### nvme_cuse | |
244 | ||
245 | `spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which | |
246 | stores the pointer to the controller name, and `size_t *size` which stores the length of | |
247 | the name. The return type has also been changed from char * to int. | |
248 | ||
249 | `spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which | |
250 | stores the pointer to the namespace name, and `size_t *size` which stores the length of | |
251 | the name. The return type has also been changed from char * to int. | |
252 | ||
253 | ### nvme_opal | |
254 | ||
255 | Several public OPAL structure definitions have been changed since the last release. | |
256 | ||
257 | ### nvmf | |
258 | ||
259 | `spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. | |
260 | ||
261 | The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool | |
262 | of lightweight threads already exist at start up time. A poll group is a collection of | |
263 | unrelated NVMe-oF connections. Each poll group is only accessed from the associated | |
264 | lightweight thread. | |
265 | ||
266 | A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific | |
267 | nature of a listener object. | |
268 | ||
269 | `spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an | |
270 | int to indicate the status of the listen call. | |
271 | ||
272 | The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts | |
273 | a cb_fn and cb_arg to call upon completion. | |
274 | ||
275 | The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts | |
276 | a cb_fn and cb_arg to call upon completion. | |
277 | ||
278 | The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate | |
279 | with NVMe-oF libraries without using internal APIs. | |
280 | ||
281 | ### ocf | |
282 | ||
283 | Updated the OCF submodule to OCF v20.03 | |
284 | ||
285 | New version of OCF comes with API changes and bug fixes | |
286 | ||
287 | ### rpc | |
288 | ||
289 | A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread | |
290 | to the specified value. | |
291 | ||
292 | A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. | |
293 | ||
294 | A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. | |
295 | ||
296 | A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. | |
297 | ||
298 | ### sock | |
299 | ||
300 | The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority | |
301 | will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. | |
302 | Users may now specify the priority of the socket in the opts that they want to use. | |
303 | ||
304 | ### spdk_top | |
305 | ||
306 | A new application, `spdk_top`, has been added which allows users to monitor resource consumption | |
307 | by a running SPDK application. More information on this application can be found in | |
308 | `app/spdk_top/README`. | |
309 | ||
310 | ### thread | |
311 | ||
312 | A new function `spdk_thread_lib_init_ext` has been added, and the function | |
313 | `spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is | |
314 | expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. | |
315 | `spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation | |
316 | is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. | |
317 | Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. | |
318 | The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. | |
319 | The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. | |
320 | ||
321 | A unique ID has been added for each created SPDK thread, it is retrieved by a new function | |
322 | `spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by | |
323 | a new function `spdk_thread_get_by_id`. | |
324 | ||
325 | A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask | |
326 | to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` | |
327 | is supported. | |
328 | ||
329 | A new function `spdk_poller_register_named` has been added to set arbitrary name to the | |
330 | created poller. If NULL, the name is set to the pointer of the poller function. | |
331 | ||
332 | The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK | |
333 | threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. | |
334 | ||
335 | Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` | |
336 | and `spdk_thread_poll`. | |
337 | ||
338 | ### util | |
339 | ||
340 | New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence | |
341 | numbers based on serial number arithmetic. | |
342 | ||
343 | ### vhost | |
344 | ||
345 | Poll groups per session have been replaced by SPDK threads per vhost controller. | |
346 | ||
347 | ### vmd | |
348 | ||
349 | A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD | |
350 | library through the `spdk_vmd_init` call. | |
351 | ||
352 | ## v20.01 | |
353 | ||
354 | ### bdev | |
355 | ||
356 | A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. | |
357 | ||
358 | A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support | |
359 | of block comparisons and compare-and-write. | |
360 | ||
361 | A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs | |
362 | which expose raw media to alert all I/O channels of pending media management events. | |
363 | ||
364 | A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request | |
365 | an auxiliary buffer for its own private use. The API is used in the same manner that | |
366 | `spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the | |
367 | bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary | |
368 | buffer. | |
369 | ||
370 | ### blobfs | |
371 | ||
372 | Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. | |
373 | ||
374 | Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. | |
375 | ||
376 | ### blobstore | |
377 | ||
378 | Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. | |
379 | Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation | |
380 | for thin provisioned blobs. Extent Table descriptor is enabled by default. | |
381 | See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. | |
382 | ||
383 | ### dpdk | |
384 | ||
385 | Updated DPDK submodule to DPDK 19.11. | |
386 | ||
387 | ### env_dpdk | |
388 | ||
389 | `spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. | |
390 | ||
391 | A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by | |
392 | the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. | |
393 | ||
394 | ### event | |
395 | ||
396 | The functions `spdk_reactor_enable_framework_monitor_context_switch()` and | |
397 | `spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to | |
398 | `spdk_framework_enable_context_switch_monitor()` and | |
399 | `spdk_framework_context_switch_monitor_enabled()`, respectively. | |
400 | ||
401 | ### ftl | |
402 | ||
403 | All NVMe dependencies were removed from ftl library. | |
404 | From now ftl library is using bdev_zone API. | |
405 | bdev_ftl becomes virtual bdev. | |
406 | ||
407 | `ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by | |
408 | `base_bdev_desc`. | |
409 | ||
410 | `bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` | |
411 | parameter. | |
412 | ||
413 | `spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. | |
414 | ||
415 | `spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. | |
416 | ||
417 | ### isa-l | |
418 | ||
419 | Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and | |
420 | optimization for aarch64. | |
421 | ||
422 | Enabled ISA-L on aarch64 by default in addition to x86. | |
423 | ||
424 | ### nvme | |
425 | ||
426 | `delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` | |
427 | to allow reuse in other transports. | |
428 | ||
429 | Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together | |
430 | and posted with a single call to ibv_post_send(receive) in the next call to qpair completion | |
431 | processing function. Batching is controlled by 'delay_cmd_submit' qpair option. | |
432 | ||
433 | The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order | |
434 | to facilitate this feature, several small API changes have been made: | |
435 | ||
436 | The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. | |
437 | A new function, `spdk_nvme_transport_available_by_name`, has been added. | |
438 | A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which | |
439 | enable registering out of tree transports. | |
440 | ||
441 | A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare | |
442 | operation. | |
443 | ||
444 | A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. | |
445 | ||
446 | A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in | |
447 | log page requests. | |
448 | ||
449 | ### nvmf | |
450 | ||
451 | Add SockPriority option in [Transport] section, this can be used for NVMe-oF target | |
452 | on TCP transport to set sock priority for the incomming TCP connections. | |
453 | ||
454 | The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order | |
455 | to facilitate this feature, several small API changes have been made: | |
456 | ||
457 | The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. | |
458 | `spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been | |
459 | modified to take a string. | |
460 | A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which | |
461 | enable registering out of tree transports. | |
462 | ||
463 | Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for | |
464 | incoming connections for specified target and trid. Listener is not stopped | |
465 | implicitly upon destruction of a subsystem any more. | |
466 | ||
467 | A custom NVMe admin command handler has been added which allows the user to use the real drive | |
468 | attributes from one of the target NVMe drives when reporting drive attributes to the initiator. | |
469 | This handler can be enabled via the `nvmf_set_config` RPC. | |
470 | Note: In a future version of SPDK, this handler will be enabled by default. | |
471 | ||
472 | The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, | |
473 | the target expects the initiator to send both the compare command and write command either with, or without inline data. The | |
474 | SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting | |
475 | compatibility with this version of SPDK. | |
476 | ||
477 | The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows | |
478 | to configure transport ACK timeout. Applicable for RDMA transport only. | |
479 | ||
480 | ### rpc | |
481 | ||
482 | A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. | |
483 | ||
484 | A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. | |
485 | ||
486 | A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. | |
487 | ||
488 | A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory | |
489 | consumption stats. Please see the env_dpdk section above for more details. | |
490 | ||
491 | A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. | |
492 | ||
493 | `bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. | |
494 | ||
495 | `bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag | |
496 | ||
497 | `bdev_nvme_cuse_register` now accepts a `name` parameter. | |
498 | ||
499 | `bdev_uring_create` now takes arguments for `bdev_name` and `block_size` | |
500 | ||
501 | `nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. | |
502 | Please see the nvmf section above for more details. | |
503 | ||
504 | ### scsi | |
505 | ||
506 | `spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. | |
507 | ||
508 | ### sock | |
509 | ||
510 | Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will | |
511 | never return EAGAIN, instead queueing internally until the data has all been sent. This can | |
512 | simplify many code flows that create pollers to continue attempting to flush writes | |
513 | on sockets. | |
514 | ||
515 | Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now | |
516 | specify the sock layer implementation they'd prefer to use. Valid implementations are currently | |
517 | "vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. | |
518 | ||
519 | ### thread | |
520 | ||
521 | `spdk_thread_send_msg` now returns int indicating if the message was successfully | |
522 | sent. | |
523 | ||
524 | A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from | |
525 | a context that may be interrupted, e.g. a signal handler. | |
526 | ||
527 | Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control | |
528 | of pollers to the application owner. | |
529 | ||
530 | ### util | |
531 | ||
532 | `spdk_pipe`, a new utility for buffering data from sockets or files for parsing | |
533 | has been added. The public API is available at `include/spdk/pipe.h`. | |
534 | ||
535 | ### bdev | |
536 | ||
537 | Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status | |
538 | code for fused compare-and-write operation. | |
539 | ||
540 | Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. | |
541 | ||
542 | ## v19.10 | |
543 | ||
544 | ### rpc | |
545 | ||
546 | Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. | |
547 | The old names will continue to function, but will display a deprecation warning. | |
548 | ||
549 | Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to | |
550 | 'bdev_nvme_set_options' RPC method. | |
551 | ||
552 | Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. | |
553 | ||
554 | Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' | |
555 | RPC method. | |
556 | ||
557 | Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. | |
558 | ||
559 | Added `blobfs_create` RPC method to build blobfs on given bdev. | |
560 | ||
561 | Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. | |
562 | Then on the host path, user can directly do some file operations which will be mapped | |
563 | to blobfs. | |
564 | ||
565 | ### bdev | |
566 | ||
567 | Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and | |
568 | `spdk_bdev_io_get_nvme_status()` that allows setting/getting | |
569 | the NVMe completion queue DW0 entry. This allows vendor specific IO commands | |
570 | to return commmand specific completion info back to the initiator. | |
571 | ||
572 | Added `spdk_bdev_get_write_unit_size()` function for retrieving required number | |
573 | of logical blocks for write operation. | |
574 | ||
575 | New zone-related fields were added to the result of the `get_bdevs` RPC call: | |
576 | ||
577 | - `zoned`: indicates whether the device is zoned or a regular | |
578 | block device | |
579 | - `zone_size`: number of blocks in a single zone | |
580 | - `max_open_zones`: maximum number of open zones | |
581 | - `optimal_open_zones`: optimal number of open zones | |
582 | ||
583 | The `zoned` field is a boolean and is always present, while the rest is only available for zoned | |
584 | bdevs. | |
585 | ||
586 | A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. | |
587 | The new open function introduces requirement to provide callback function that will be called by | |
588 | asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as | |
589 | an argument instead of bdev structure to avoid a race condition that can happen when the bdev | |
590 | is being removed between a call to get its structure based on a name and actually openning it. | |
591 | ||
592 | New 'resize' event has been added to notify about change of block count property of block device. | |
593 | Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. | |
594 | ||
595 | ### bdev zone | |
596 | ||
597 | Added new public header for zoned bdev. Zoned bdev is an extension | |
598 | of the bdev interface. | |
599 | ||
600 | `spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` | |
601 | APIs were added for retrieving zoned device information. | |
602 | `spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned | |
603 | device. | |
604 | Added `spdk_bdev_zone_management()` API for changing zone state. | |
605 | `spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for | |
606 | appending data to a zone. | |
607 | Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. | |
608 | Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. | |
609 | ||
610 | ### bdev opal | |
611 | ||
612 | EXPERIMENTAL: A new opal bdev has been added to support management of | |
613 | NVMe self-encrypting drives through the Opal specification. Users can | |
614 | create opal bdevs from an NVMe namespace bdev, if the controller | |
615 | containing that namespace supports Opal. Currently this is only | |
616 | supported for namespace ID=1. The following RPCs have been added to | |
617 | support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, | |
618 | `bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, | |
619 | `bdev_opal_new_user`, `bdev_opal_set_lock_state`. | |
620 | It does not yet support recreating the opal bdevs after application restart. | |
621 | This bdev module should be considered very experimental, and the RPCs may | |
622 | change significantly in future releases. | |
623 | ||
624 | ### delay bdev | |
625 | ||
626 | The `bdev_delay_update_latency` has been added to allow users to update | |
627 | a latency value for a given delay bdev. | |
628 | ||
629 | ### compress bdev | |
630 | ||
631 | A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs | |
632 | that were not loaded due to a missing pm metadata file. In this state they | |
633 | can only be deleted. | |
634 | ||
635 | ### null bdev | |
636 | ||
637 | Metadata support has been added to Null bdev module. | |
638 | ||
639 | Protection information support has been added to Null bdev module. | |
640 | ||
641 | ### nvme | |
642 | ||
643 | Added `no_shn_notification` to NVMe controller initialization options, users can enable | |
644 | it for NVMe controllers. When the option is enabled, the controller will not do the | |
645 | shutdown process and just disable the controller, users can start their application | |
646 | later again to initialize the controller to the ready state. | |
647 | ||
648 | A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller | |
649 | can support weighted round robin arbitration feature with submission queue. | |
650 | ||
651 | Added `arbitration_burst` option for arbitration feature, and added three | |
652 | `low/medium/high_priority_weight` options for weighted round robin arbitration. | |
653 | ||
654 | Added `spdk_nvme_ns_cmd_write_uncorrectable`. | |
655 | ||
656 | Added new error handling and reporting functionality. This includes several | |
657 | new API functions to facilitate applications recovering when a qpair or | |
658 | controller fails. | |
659 | ||
660 | `spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. | |
661 | `spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to | |
662 | change the trid for failover cases. | |
663 | `spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. | |
664 | `spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. | |
665 | ||
666 | Modified the return behavior of several API functions to better indicate to | |
667 | applications when a qpair is failed. This list of functions includes: | |
668 | ||
669 | `spdk_nvme_qpair_process_completions` | |
670 | `spdk_nvme_ns_cmd_*` | |
671 | `spdk_nvme_ctrlr_process_admin_completions` | |
672 | `spdk_nvme_ctrlr_cmd_*` | |
673 | ||
674 | These functions now return -ENXIO when the qpair or controller on which they | |
675 | operate is failed. | |
676 | ||
677 | EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux | |
678 | kernel for controller as well as for namespace and process ioctl requests as usual | |
679 | from linux environment. | |
680 | ||
681 | ### nvmf | |
682 | ||
683 | The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` | |
684 | as its only parameter. This new structure contains the max_subsystems parameter previously | |
685 | passed into that function. | |
686 | ||
687 | A new public API function `spdk_nvmf_get_tgt` has been added which allows users to | |
688 | retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special | |
689 | case where an RPC or application only creates a single target, this function can accept | |
690 | a null name parameter and will return the only available target. | |
691 | ||
692 | The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will | |
693 | allow those RPCs to work with applications that create more than one target. | |
694 | ||
695 | Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and | |
696 | `nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target | |
697 | objects. In SPDK the target object defines a unique discovery service. As of this release, | |
698 | these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and | |
699 | nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py | |
700 | ||
701 | Three new header functions have also been added to help deal with multiple targets. | |
702 | `spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. | |
703 | `spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. | |
704 | `spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. | |
705 | ||
706 | The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context | |
707 | information to the `new_qpair` callback. This will simplify the code when having multiple | |
708 | nvmf targets or when retrieving the context information from globals is not suitable. | |
709 | ||
710 | ### blobstore | |
711 | ||
712 | A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` | |
713 | function has been deprecated. | |
714 | The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback | |
715 | function that will be called by asynchronous event such as bdev removal. | |
716 | ||
717 | ### blobfs_bdev | |
718 | ||
719 | A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. | |
720 | ||
721 | Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. | |
722 | ||
723 | Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. | |
724 | ||
725 | Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to | |
726 | a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle | |
727 | FUSE request by blobfs API. | |
728 | ||
729 | ### build | |
730 | ||
731 | Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. | |
732 | It requires the installation of libfuse3. By default, it is disabled. And it will be | |
733 | enabled if run `./configure` with `--with-fuse` option. | |
734 | ||
735 | ### iSCSI | |
736 | ||
737 | Portals may no longer be associated with a cpumask. The scheduling of | |
738 | connections is moving to a more dynamic model. | |
739 | ||
740 | An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication | |
741 | for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites | |
742 | the setting by the global parameters for the iSCSI portal group. | |
743 | ||
744 | ### socket | |
745 | ||
746 | Added `spdk_sock_is_connected` to check whether the socket is currently connected. | |
747 | `spdk_sock_group_poll` now returns number of events on success. | |
748 | ||
749 | ### env | |
750 | ||
751 | Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. | |
752 | ||
753 | ### event | |
754 | ||
755 | `framework_start_init` RPC no longer stops the application on error during | |
756 | initialization. | |
757 | ||
758 | ### DPDK | |
759 | ||
760 | Updated DPDK submodule to DPDK 19.08. | |
761 | ||
762 | ### ocf | |
763 | ||
764 | Updated OCF submodule to OCF v19.06 | |
765 | ||
766 | Along with update, new cache mode 'write only' was added. | |
767 | ||
768 | New cache modes added to use via RPC, wi - write invalidate and wa - write around. | |
769 | ||
770 | New version of OCF provides fully asynchronous management API. | |
771 | ||
772 | ## v19.07 | |
773 | ||
774 | ### ftl | |
775 | ||
776 | EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache | |
777 | to allow for recovery from dirty shutdown event. | |
778 | ||
779 | Added handling of Asynchronous Nand Management Events (ANM). | |
780 | ||
781 | ### vmd | |
782 | ||
783 | EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated | |
784 | controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected | |
785 | NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into | |
786 | SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect | |
787 | NVMe driver to the device located at the given transport ID. | |
788 | ||
789 | To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. | |
790 | ||
791 | Current implementation does not support hotplug. | |
792 | ||
793 | ### blobfs | |
794 | ||
795 | Blobfs file asynchronous operations were added to public APIs. | |
9f95a23c TL |
796 | |
797 | ### util | |
798 | ||
799 | A new file API `spdk_posix_file_load` was added to load file content into a data buffer. | |
800 | ||
f67539c2 TL |
801 | New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, |
802 | `spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, | |
803 | and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and | |
804 | `spdk_dif_generate_stream` have been refined to insert or strip DIF by stream | |
805 | fasion with any alignment. | |
806 | ||
807 | New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, | |
808 | and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. | |
809 | ||
810 | New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been | |
811 | added to compute CRC-32C checksum for extended LBA payload. | |
812 | ||
813 | Bdevperf and bdevio applications now support starting tests with application specific | |
814 | RPCs. Please see helper Python scripts in their respective directories. | |
815 | This is a move towards simpler RPC-only configuration for all main | |
816 | and auxiliary applications. | |
817 | ||
818 | Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, | |
819 | and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or | |
820 | RPC driven run-time configuration. | |
821 | ||
822 | ### nvmf | |
823 | ||
824 | EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been | |
825 | added. This depends on the FC HBA driver at | |
826 | https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) | |
827 | for more information. | |
828 | ||
829 | Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation | |
830 | state is stored in a JSON file on the local filesystem between target restart. To support this, | |
831 | an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. | |
832 | This allows the user to specify which file to store the persistent reservation state in. Note | |
833 | that this is done per namespace. | |
834 | ||
835 | The c2h success optimization under which a command capsule response is not sent | |
836 | for reads is turned on by default. A config knob was added to allow disabling | |
837 | the optimization. This will mostly be used for integration testing with 5.0.x kernels | |
838 | while some compatibility fixes make their way down the pipeline for 5.1.x kernels. | |
839 | ||
840 | The sock priority setting of the TCP connection owned by the tcp transport is added. It is | |
841 | used to optimize the TCP connection performance under designated traffic classes. And the | |
842 | priority is used to differeniate the sock priority between SPDK NVMe-oF TCP target application | |
843 | and other TCP based applications. | |
9f95a23c TL |
844 | |
845 | Shared receive queue can now be disabled even for NICs that support it using the | |
846 | `nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared | |
847 | receive queue is predicated on hardware support when this flag is not used. | |
848 | ||
f67539c2 TL |
849 | spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal |
850 | poll group for the qpair. And `ConnectionScheduler` configuration is added into the | |
851 | [Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection | |
852 | scheduling strategy among different spdk threads. | |
853 | ||
854 | Added infrastructure to retrieve global and per poll group NVMf statistics. | |
855 | ||
856 | DIF strip and insert is now supported for TCP transport. When it is enabled, DIF | |
857 | setting is not exposed to the NVMe-oF initiator, and DIF is attached into data | |
858 | for write I/O and stripped from data for read I/O. | |
859 | ||
860 | Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and | |
861 | updated the related rpc function nvmf_create_transport to make this | |
862 | configurable parameter available to users. The `dif_insert_or_strip` is relevant | |
863 | for TCP transport for now and used to configure the DIF strip and insert. | |
864 | ||
865 | Added infrastructure to retrieve NVMf transport statistics. | |
866 | ||
9f95a23c TL |
867 | ### notify |
868 | ||
869 | The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were | |
870 | renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, | |
871 | respectively. And update type name of callback accordingly. | |
872 | ||
873 | ### bdev | |
874 | ||
875 | The format of the data returned by the get_bdevs_iostat RPC has changed to | |
876 | make it easier to parse. It now returns an object with a "ticks" object | |
877 | and "bdevs" array with the per-bdev statistics. | |
878 | ||
879 | A new bdev module `delay` has been added which simulates a drive latency when placed | |
880 | on top of a Null bdev. This module is intended only for testing and can be created using | |
881 | the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well | |
882 | as average and p99 latency arguments for both read and write operations. Average latency is | |
883 | defined as a value close to what you would expect a perf tool such as FIO to report back as | |
884 | the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one | |
885 | would expect the drive to see the slowest 1% of I/O report. For underlying drives with already | |
886 | significant latency, the latency values provided to the drive will be additive. This should be | |
887 | taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. | |
888 | ||
f67539c2 TL |
889 | DIF reference tag remapping is now supported for partition type virtual bdev |
890 | modules. When using partition type virtual bdevs, block address space is | |
891 | remapped during I/O processing and DIF reference tag is remapped accordingly. | |
892 | ||
893 | Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in | |
894 | separate buffer. To check support for separatate metadata, use spdk_bdev_is_md_separate(). | |
895 | ||
896 | All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, | |
897 | one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically | |
898 | generated may change. | |
899 | ||
900 | A new virtual bdev module `compress` has been added to provide compression services on top of | |
901 | a thinly provisioned logical volume. See documentation for complete details. | |
902 | ||
9f95a23c TL |
903 | ### nvme |
904 | ||
f67539c2 TL |
905 | Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which |
906 | can be used to change the number of requests allocated for one NVMe I/O queue. For | |
907 | very big I/O size, e.g. 128MiB, with this option user will not get an error due to | |
908 | limited requests in NVMe driver layer. | |
909 | ||
9f95a23c TL |
910 | Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a |
911 | previously attached controller. | |
912 | ||
913 | Nvme Opal library spdk_opal_cmd deprecated. Adding seperate command APIs. | |
f67539c2 TL |
914 | NVMe Opal library add support for activating locking SP which will make the transaction |
915 | from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking | |
916 | of this method, lock and unlock features will be enabled. | |
917 | NVMe Opal library add support for locking/unlocking range and list locking range info. | |
918 | NVMe opal library add support for multiuser. Admin can enable user and add user to specific | |
919 | locking range and the user can lock/unlock his range. | |
920 | ||
921 | Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass | |
922 | a completely formed command to an NVMe submission queue (buffer addresses and all). | |
923 | This is supported on the PCIe transport only. | |
924 | ||
925 | Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address | |
926 | of the NVMe controller registers. This is supported on the PCIe transport only. | |
927 | ||
928 | Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter | |
929 | structure to allow caller to override the virtual and optionally physical address | |
930 | of the submission and completion queue pair to be created. This is supported on | |
931 | the PCIe transport only. | |
932 | ||
933 | Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables | |
934 | logging of failed requests. By default logging is enabled. | |
935 | ||
936 | Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and | |
937 | spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. | |
938 | ||
939 | Added support for NVMe Sanitize command. | |
9f95a23c TL |
940 | |
941 | ### env | |
942 | ||
943 | The parameter `free_space` has been added to spdk_ring_enqueue() to wait when | |
944 | the ring is almost full and resume when there is enough space available in | |
945 | the ring. | |
946 | ||
f67539c2 TL |
947 | A new API `spdk_mempool_lookup` has been added to lookup the memory pool created |
948 | by the primary process. | |
949 | ||
950 | Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow | |
951 | iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs | |
952 | to attach/detach PCI devices are no longer thread safe. They are now meant to | |
953 | be called from only a single thread only, the same only that called spdk_env_init(). | |
954 | This applies to the newly added APIs as well. | |
955 | ||
956 | ### vpp | |
957 | ||
958 | SPDK now supports VPP version 19.04.2, up from VPP 18.01. | |
959 | ||
960 | VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). | |
961 | This allows for better control over sessions and queues. | |
962 | Please see VPP documentation for more details: | |
963 | [VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) | |
964 | ||
965 | ### sock | |
966 | ||
967 | Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for | |
968 | this socket. When a socket is created, it is often assigned to a sock group using | |
969 | spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. | |
970 | For some network devices, it is optimal to assign particular sockets to specific | |
971 | sock groups. This API is intended to provide the user with that information. | |
972 | ||
973 | spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. | |
974 | spdk_sock_group_create() is updated to allow input the user provided ctx. | |
975 | spdk_sock_set_priority() is added to set the priority of the socket. | |
976 | ||
977 | ### rpc | |
978 | ||
979 | Added thread_get_stats RPC method to retrieve existing statistics. | |
980 | ||
981 | Added nvmf_get_stats RPC method to retrieve NVMf susbsystem statistics. | |
982 | ||
983 | Response buffers for RPC requests are now always pre-allocated, which implies | |
984 | that all spdk_jsonrpc_begin_result() calls always succeed and return a valid | |
985 | buffer for JSON response. RPC calls no longer need to check if the buffer is | |
986 | non-NULL. | |
987 | ||
988 | Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when | |
989 | renaming existing RPC. First time a deprecated alias is used, it will print | |
990 | a warning message. | |
991 | ||
992 | RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, | |
993 | but is now deprecated. | |
994 | ||
995 | ### blobstore | |
996 | ||
997 | A snapshot can now be deleted if there is only a single clone on top of it. | |
998 | ||
999 | ### build | |
1000 | ||
1001 | Preliminary support for cross compilation is now available. Targeting an older | |
1002 | CPU on the same architecture using your native compiler can be accomplished by | |
1003 | using the `--target-arch` option to `configure` as follows: | |
1004 | ||
1005 | ~~~ | |
1006 | ./configure --target-arch=broadwell | |
1007 | ~~~ | |
1008 | ||
1009 | Additionally, some support for cross-compiling to other architectures has been | |
1010 | added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC | |
1011 | and CXX to the cross compilers, then run configure as follows: | |
1012 | ||
1013 | ~~~ | |
1014 | ./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu | |
1015 | ~~~ | |
1016 | ||
1017 | ### vhost | |
1018 | ||
1019 | A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. | |
1020 | A malicious client (e.g. a virtual machine) could send a carefully prepared, | |
1021 | invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI | |
1022 | target are strongly recommended to update. All SPDK versions < 19.07 are affected. | |
1023 | ||
1024 | By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork | |
1025 | located inside SPDK repo. The internal fork is still kept around to support older | |
1026 | DPDK versions, but is considered legacy and will be eventually removed. | |
1027 | ||
1028 | `configure` will now automatically use the upstream rte_vhost if the used DPDK | |
1029 | version is >= 19.05. | |
1030 | ||
1031 | spdk_vhost_init() is now asynchronous and accepts a completion callback. | |
1032 | ||
1033 | ### iscsi target | |
1034 | ||
1035 | A security vulnerability has been identified and fixed in SPDK iSCSI target. | |
1036 | A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, | |
1037 | invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI | |
1038 | target are strongly recommended to update. All SPDK versions < 19.07 are affected. | |
1039 | ||
1040 | ### thread | |
1041 | ||
1042 | Exposed spdk_set_thread() in order for applications to associate | |
1043 | with SPDK thread when necessary. | |
1044 | ||
1045 | Added spdk_thread_destroy() to allow framework polling the thread to | |
1046 | release resources associated with that thread. | |
1047 | ||
1048 | ## v19.04 | |
9f95a23c TL |
1049 | |
1050 | ### nvme | |
1051 | ||
1052 | Added asynchronous probe support. New APIs spdk_nvme_probe_async(), | |
1053 | spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable | |
1054 | this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a | |
1055 | context associated with the specified controllers. Users then call | |
1056 | spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation | |
1057 | completed. | |
1058 | ||
1059 | A new qpair creation option, delay_pcie_doorbell, was added. This can be passed | |
1060 | to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as | |
1061 | spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the | |
1062 | doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). | |
1063 | This can result in significantly fewer MMIO writes to the doorbell register | |
1064 | under heavy load, greatly improving performance. | |
1065 | ||
1066 | spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller | |
1067 | memory buffer, if available. | |
1068 | ||
1069 | spdk_nvme_ctrlr_get_flags() was added to return controller feature | |
1070 | flags. Two flags are currently tracked: | |
1071 | SPDK_NVME_CTRLR_SGL_SUPPORTED | |
1072 | SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED | |
1073 | ||
1074 | The NVMe hotplug poller is now able to detach devices hot-removed from the system | |
1075 | via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. | |
1076 | ||
1077 | Opal support was added for scan, take ownership, revert TPer, and dumping device | |
1078 | info. The nvme_manage tool can be used to perform these operations. The public | |
1079 | API functions are spdk_nvme_ctrlr_security_receive() and | |
1080 | spdk_nvme_ctrlr_security_send(). This module should be considered experimental | |
1081 | pending additional features and tests. | |
1082 | ||
f67539c2 TL |
1083 | The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB |
1084 | if the corresponding NVMe-oF target allows. | |
1085 | ||
9f95a23c TL |
1086 | ### raid |
1087 | ||
1088 | Added new strip_size_kb rpc param on create to replace the more ambiguous | |
1089 | strip_size. The strip_size rpc param is deprecated. | |
1090 | ||
1091 | Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" | |
1092 | ||
1093 | ### thread | |
1094 | ||
1095 | Added spdk_thread_has_pollers() function to verify if there are any registered | |
1096 | pollers to be run on the thread. Added spdk_thread_is_idle() function to check | |
1097 | if there are any scheduled operations to be performed on the thread at given | |
1098 | time. | |
1099 | ||
1100 | spdk_thread_create() now takes an optional CPU affinity mask that is passed to | |
1101 | the scheduler. | |
1102 | ||
1103 | spdk_thread_lib_init() now takes an optional context size. For each thread | |
1104 | created, an additional region of memory of this size will be allocated. A | |
1105 | pointer to this region of memory can be obtained by calling | |
1106 | spdk_thread_get_ctx(). The inverse operation is also available via | |
1107 | spdk_thread_get_from_ctx(). | |
1108 | ||
1109 | spdk_thread_poll() now optionally accepts the current time, in ticks. This can | |
1110 | be used to avoid some calls to spdk_get_ticks() internally. | |
1111 | ||
1112 | spdk_app_start() now only accepts a single context argument. | |
1113 | ||
1114 | ### bdev | |
1115 | ||
1116 | An new API `spdk_bdev_get_data_block_size` has been added to get size of data | |
1117 | block except for metadata. | |
1118 | ||
1119 | spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used | |
1120 | instead. | |
1121 | ||
1122 | A mechanism for acquiring and releasing data buffers from bdev modules, used | |
1123 | to perform zero copy operations, was added. | |
1124 | ||
1125 | New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), | |
1126 | spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been | |
1127 | added to get metadata and DIF settings. | |
1128 | ||
1129 | Bdevs claimed by the `examine_config` callback will be now further examined in the | |
1130 | `examine_disk` callback. | |
1131 | ||
1132 | spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel | |
1133 | from a bdev_io. | |
1134 | ||
f67539c2 TL |
1135 | ### ocf |
1136 | ||
1137 | Updated OCF submodule to OCF v19.3.2 | |
1138 | ||
1139 | Added support for many-to-one configuration for OCF bdev. | |
1140 | Multiple core devices can now be cached on single cache device. | |
1141 | ||
1142 | Added persistent metadata support, allowing to restore cache state after shutdown. | |
1143 | During start of SPDK application, the devices are examined and if OCF metadata | |
1144 | is present - appropriate OCF bdevs will be recreated. | |
1145 | ||
1146 | Added Write-Back mode support. In this mode, data is first written to | |
1147 | caching device and periodically synchronized to the core devices. | |
1148 | Dirty data is saved as persistent metadata on cache device, | |
1149 | allowing for safe restore during application restart. | |
1150 | For more details please see OCF documentation: | |
1151 | [OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) | |
1152 | ||
9f95a23c TL |
1153 | ### NVMe-oF Target |
1154 | ||
1155 | Support for per-device shared receive queues in the RDMA transport has been added. | |
1156 | It is enabled by default for any device that supports it. | |
1157 | ||
1158 | The size of a shared receive queue is defined by transport configuration file parameter | |
1159 | `MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. | |
1160 | Default size is 4096. | |
1161 | ||
1162 | Add model number as parameter to construct_nvmf_subsystem (-d option), | |
1163 | rather than using hardcoded define. | |
1164 | ||
1165 | DIF passthrough feature has been added. DIF setting of the allocated bdevs is | |
1166 | exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is | |
1167 | passed through to the allocated bdevs. | |
1168 | ||
1169 | ### env | |
1170 | ||
1171 | The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. | |
1172 | For retrieving physical addresses, spdk_vtophys() should be used instead. | |
1173 | ||
1174 | spdk_realloc() has been added to reallocate DMA/shared memory. | |
1175 | ||
1176 | spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know | |
1177 | that device has a pending external hotremove request. | |
1178 | ||
1179 | spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources | |
1180 | allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected | |
1181 | that common usage of those functions is to call them just before terminating the process. | |
1182 | ||
1183 | Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. | |
1184 | ||
1185 | ### DPDK | |
1186 | ||
1187 | Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. | |
1188 | ||
1189 | Updated DPDK submodule to DPDK 19.02. | |
1190 | ||
1191 | ### rpc | |
1192 | ||
1193 | New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. | |
1194 | ||
1195 | The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device | |
1196 | is specified, use that specified nbd device. If it's not specified, pick available one. | |
1197 | ||
1198 | ### iSCSI target | |
1199 | ||
1200 | DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. | |
1201 | DIF is attached into data for write I/O and stripped from data for read I/O. | |
1202 | ||
1203 | ### vhost | |
1204 | ||
1205 | Added experimental support for running with the external, upstream rte_vhost library. | |
1206 | This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. | |
1207 | The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. | |
1208 | ||
1209 | As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and | |
1210 | fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks | |
1211 | if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. | |
1212 | ||
1213 | The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first | |
1214 | unoccupied target id will be used. | |
1215 | ||
1216 | ### AIO | |
1217 | ||
1218 | AIO bdev module can now reap I/O completions directly from userspace, significantly improving | |
1219 | the overall performance. | |
1220 | ||
1221 | ### blobfs | |
1222 | ||
1223 | Synchronous IO operations no longer use spdk_io_channel, but instead use | |
1224 | spdk_fs_thread_ctx. The behavior is otherwise identical. | |
1225 | ||
1226 | ### OCF | |
1227 | ||
1228 | Added support for caching multiple bdevs using a single bdev as a cache. | |
1229 | ||
1230 | ### notify | |
1231 | ||
1232 | Added the notify library that provides a high performance local event bus | |
1233 | between libraries. Example usage was added to bdev module, which reports | |
1234 | notifications for added and removed bdevs via RPC. | |
1235 | ||
1236 | ### sock | |
1237 | ||
1238 | Added new API spdk_sock_readv() to the sock library for performing vectored | |
1239 | reads. | |
1240 | ||
1241 | ### event | |
1242 | ||
1243 | The function spdk_subsystem_init() no longer requires spdk_event as an argument. | |
1244 | ||
1245 | Changed API of spdk_subsystem_config_json() to no longer be asynchronous. | |
1246 | ||
1247 | ### io_uring | |
1248 | ||
1249 | A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel | |
1250 | API was added. This module requires liburing. | |
1251 | ||
1252 | ### build | |
1253 | ||
1254 | Options to easily compile with profile guided optimization have been added to | |
1255 | `configure`. To use profile guided optimization with SPDK, run | |
1256 | `./configure --with-pgo-capture`, build SPDK, then run a workload of your | |
1257 | choosing. Then, simply run `./configure --with-pgo-enable` and recompile to | |
1258 | build using the generated profile data. Profile guided optimization can yield | |
1259 | very large performance improvements, especially on GCC 8 and clang 7. This may | |
1260 | be combined with link time optimization which has been available under the | |
1261 | `--enable-lto` configure option for several releases. | |
1262 | ||
1263 | ### compression bdev/reduce library | |
1264 | ||
1265 | Added "reduce" block compression scheme based on using SSDs for storing | |
1266 | compressed blocks of storage and presistent memory for metadata. Please see | |
1267 | [compression](https://spdk.io/doc/bdev.html) for more details. | |
1268 | ||
f67539c2 | 1269 | ## v19.01 |
9f95a23c TL |
1270 | |
1271 | ### ocf bdev | |
1272 | ||
1273 | New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. | |
1274 | This module allows for the use of one bdev to act as a high performance cache in front of another bdev. | |
1275 | Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. | |
1276 | Only write through mode is currently supported and this feature is considered experimental. | |
1277 | ||
1278 | ### event framework | |
1279 | ||
1280 | For `spdk_app_parse_args`, add return value to the callback which parses application | |
1281 | specific command line parameters to protect SPDK applications from crashing by invalid | |
1282 | values from user input. | |
1283 | ||
1284 | By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved | |
1285 | memory size can be still set with `-s` or `--mem-size` option, although the default value | |
1286 | was reduced down to 0. | |
1287 | ||
1288 | A custom hugetlbfs directory can now be specified via spdk_app_opts. | |
1289 | This can be used to configure hugepages with different sizes, a different size limit, | |
1290 | or different access permissions than the system's default hugepage pool. | |
1291 | SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. | |
1292 | ||
1293 | ### environment | |
1294 | ||
1295 | spdk_vtophys() has been refactored to accept length of the translated region as a new | |
1296 | parameter. The function will now update that parameter with the largest possible value | |
1297 | for which the memory is contiguous in the physical memory address space. | |
1298 | ||
1299 | The following functions were removed: | |
f67539c2 | 1300 | |
9f95a23c TL |
1301 | - spdk_pci_nvme_device_attach() |
1302 | - spdk_pci_nvme_enumerate() | |
1303 | - spdk_pci_ioat_device_attach() | |
1304 | - spdk_pci_ioat_enumerate() | |
1305 | - spdk_pci_virtio_device_attach() | |
1306 | - spdk_pci_virtio_enumerate() | |
1307 | ||
1308 | They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which | |
1309 | require a new spdk_pci_driver object to be provided. It can be one of the following: | |
f67539c2 | 1310 | |
9f95a23c TL |
1311 | - spdk_pci_nvme_get_driver() |
1312 | - spdk_pci_ioat_get_driver() | |
1313 | - spdk_pci_virtio_get_driver() | |
1314 | ||
1315 | spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual | |
1316 | spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for | |
1317 | each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() | |
1318 | or spdk_pci_enumerate() like any other device. | |
1319 | ||
1320 | A new spdk_pause() function was added to pause CPU execution for an implementation specific | |
1321 | amount of time. Quoting from DPDK function this is based on: "This call is intended for | |
1322 | tight loops which poll a shared resource or wait for an event. A short pause within the loop | |
1323 | may reduce the power consumption." | |
1324 | ||
1325 | A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init | |
1326 | is added into it. If user is using DPDK, and already called rte_eal_init, then include | |
1327 | include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. | |
1328 | ||
1329 | ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture | |
1330 | to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L | |
1331 | by explicitly passing --without-isal to the configure script. | |
1332 | ||
1333 | ### util | |
1334 | ||
1335 | A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. | |
1336 | ||
1337 | An new parameter `init_crc` representing the initial CRC value was added to | |
1338 | `spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning | |
1339 | multiple separate buffers. | |
1340 | ||
1341 | New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX | |
1342 | formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF | |
1343 | insert and strip. | |
1344 | ||
1345 | Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` | |
1346 | and `strtoll`. | |
1347 | ||
1348 | Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string | |
1349 | with automatic buffer re-allocation. | |
1350 | ||
1351 | ### nvme | |
1352 | ||
1353 | Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are | |
1354 | introduced to support further security protocol development. | |
1355 | ||
1356 | admin_timeout_ms was added to NVMe controller initialization options, users | |
1357 | can change the default value when probing a controller. | |
1358 | ||
1359 | Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, | |
1360 | it will be used to enable the digest support for the NVMe/TCP transport. | |
1361 | ||
1362 | Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With | |
1363 | this new transport, it can be used to connect the NVMe-oF target with the | |
1364 | same TCP/IP support. | |
1365 | ||
1366 | Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr | |
1367 | arg refers to a Discovery Controller or not. | |
1368 | ||
1369 | Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` | |
1370 | for parsing host address and host service ID arguments on a per connection basis. | |
1371 | ||
1372 | The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to | |
1373 | use when connecting to the controller backing the NVMe bdev. | |
1374 | ||
1375 | ### NVMe-oF Target | |
1376 | ||
1377 | The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. | |
1378 | Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` | |
1379 | will also no longer implicitly initialize a transport with the default target options (since | |
1380 | there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` | |
1381 | prior to calling `spdk_nvmf_tgt_listen`. | |
1382 | ||
1383 | Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to | |
1384 | `set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. | |
1385 | ||
1386 | Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, | |
1387 | and also updated the related rpc function nvmf_create_transport, to make this | |
1388 | configurable parameter available to users. The `num_shared_buffers` is used to | |
1389 | configure the shared buffer numbers of the transport used by RDMA or TCP transport. | |
1390 | `buf_cache_size` configures number of shared buffers to cache per poll group. | |
1391 | ||
1392 | ### nvmf | |
1393 | ||
1394 | Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this tranport, | |
1395 | the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF | |
1396 | protocol via TCP/IP from the host. | |
1397 | ||
1398 | Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. | |
1399 | By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. | |
1400 | Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. | |
1401 | ||
1402 | ### bdev | |
1403 | ||
1404 | Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. | |
1405 | Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. | |
1406 | ||
1407 | Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. | |
1408 | Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. | |
1409 | ||
1410 | On shutdown, bdev unregister now proceeds in top-down fashion, with | |
1411 | claimed bdevs skipped (these will be unregistered later, when virtual | |
1412 | bdev built on top of the respective base bdev unclaims it). This | |
1413 | allows virtual bdevs to be shut down cleanly as opposed to the | |
1414 | previous behavior that didn't differentiate between hotremove and | |
1415 | planned shutdown. | |
1416 | ||
1417 | The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC | |
1418 | for the bdev type to be removed (i.e. delete_malloc_bdev). | |
1419 | ||
1420 | Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. | |
1421 | ||
1422 | Bdev I/O statistics now track unmap opertations. | |
1423 | ||
1424 | ### logical volumes | |
1425 | ||
1426 | Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. | |
1427 | This allows for basing clones on top of lvol_bdev without first creating a snapshot. | |
1428 | ||
1429 | Added option to change method for data erasure when deleting lvol or resizing down. | |
1430 | Default of unmapping clusters can now be changed to writing zeroes or no operation. | |
1431 | ||
1432 | Added option to change method for erasing data region on lvol store creation. | |
1433 | Default of unmapping can now be changed to writing zeroes or no operation. | |
1434 | ||
1435 | ### log | |
1436 | ||
1437 | "trace flags" are now referred to as "log flags" in the SPDK log API. The | |
1438 | set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, | |
1439 | and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. | |
1440 | ||
1441 | ### trace | |
1442 | ||
1443 | New `trace_record` application was added. It can be used to poll spdk trace shm file and | |
1444 | append any new trace entries into another specified file. This can help retain those entries | |
1445 | that would otherwise be overwritten in the shm file. See | |
1446 | [Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) | |
1447 | for more details. | |
1448 | ||
1449 | Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app | |
1450 | with argument "--num-trace-entries <NUM>" provided. | |
1451 | ||
1452 | New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and | |
1453 | each tpoint group status. | |
1454 | New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or | |
1455 | disable a specific tpoint group. | |
1456 | ||
1457 | ### ftl | |
1458 | ||
1459 | EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as | |
1460 | block devices. The module is split into the library (located in lib/ftl) and bdev_ftl | |
1461 | (lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. | |
1462 | ||
1463 | ### vhost | |
1464 | ||
1465 | A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious | |
1466 | vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which | |
1467 | would result in a partial denial of service in the SPDK vhost target. These types of descriptor | |
1468 | chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted | |
1469 | vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny | |
1470 | Yakovlev.) | |
1471 | ||
1472 | Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. | |
1473 | Each connection (internally called a vhost session) will have access to the same storage, but | |
1474 | will use different virtqueues, different features and possibly different memory. | |
1475 | ||
1476 | ### vhost scsi | |
1477 | ||
1478 | SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. | |
1479 | Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. | |
1480 | ||
1481 | ### DPDK | |
1482 | ||
1483 | DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new | |
1484 | multi-process device hotplug yet and continues to work the same way it always did. | |
1485 | ||
1486 | Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. | |
1487 | ||
1488 | ### RPC | |
1489 | ||
1490 | The following RPC commands deprecated in the previous release are now removed: | |
f67539c2 | 1491 | |
9f95a23c TL |
1492 | - construct_virtio_user_scsi_bdev |
1493 | - construct_virtio_pci_scsi_bdev | |
1494 | - construct_virtio_user_blk_bdev | |
1495 | - construct_virtio_pci_blk_bdev | |
1496 | - remove_virtio_scsi_bdev | |
1497 | - construct_nvmf_subsystem | |
1498 | ||
1499 | ### Miscellaneous | |
1500 | ||
1501 | The configure options `--with-raid` and `--without-raid` that were deprecated in the previous | |
1502 | release are now removed. | |
1503 | ||
1504 | ### nbd | |
1505 | ||
1506 | Starting nbd using `spdk_nbd_start` is now performed asynchronously. | |
1507 | ||
1508 | ### net framework | |
1509 | ||
1510 | Net framework initialization and finish is now done asynchronously. | |
1511 | ||
1512 | ### rpc | |
1513 | ||
1514 | Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. | |
1515 | Added `spdk_rpc_get_state` to check current state of RPC server. | |
1516 | RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized. | |
1517 | ||
1518 | ### json rpc | |
1519 | ||
1520 | JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. | |
1521 | JSON RPC server can now recieve a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` | |
1522 | and `spdk_jsonrpc_conn_del_close_cb`. | |
1523 | ||
f67539c2 | 1524 | ## v18.10 |
11fdf7f2 TL |
1525 | |
1526 | ### nvme | |
1527 | ||
1528 | spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() | |
1529 | were added to support sending or receiving security protocol data to or from | |
1530 | nvme controller. | |
1531 | ||
1532 | spdk_nvme_ns_get_extended_sector_size() was added. This function includes | |
1533 | the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still | |
1534 | returns only the data size per sector, not including metadata. | |
1535 | ||
9f95a23c TL |
1536 | New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. |
1537 | See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation | |
1538 | for more details. | |
1539 | ||
11fdf7f2 TL |
1540 | ### Build System |
1541 | ||
1542 | New `configure` options, `--with-shared` and `--without-shared` | |
1543 | [default], provide the capability to build, or not, SPDK shared libraries. | |
1544 | This includes the single SPDK shared lib encompassing all of the SPDK | |
1545 | static libs as well as individual SPDK shared libs corresponding to | |
1546 | each of the SPDK static ones. Although the production of the shared | |
1547 | libs conforms with conventional version naming practices, such naming | |
1548 | does not at this time confer any SPDK ABI compatibility claims. | |
1549 | ||
1550 | ### bdev | |
1551 | ||
1552 | spdk_bdev_alias_del_all() was added to delete all alias from block device. | |
1553 | ||
1554 | A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev | |
1555 | Framework. The module initially uses a software AESNI CBC cipher with experimental support for the | |
1556 | Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work | |
9f95a23c | 1557 | may include additional ciphers as well as consideration for authentication. |
11fdf7f2 TL |
1558 | |
1559 | The RAID virtual bdev module is now always enabled by default. The configure --with-raid and | |
1560 | --without-raid options are now ignored and deprecated and will be removed in the next release. | |
1561 | ||
1562 | Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. | |
1563 | See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) | |
1564 | documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at | |
1565 | 18.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both | |
1566 | bandwidth and IOPS limits. | |
1567 | ||
9f95a23c TL |
1568 | spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. |
1569 | ||
11fdf7f2 TL |
1570 | ### Environment Abstraction Layer and Event Framework |
1571 | ||
1572 | The size parameter of spdk_mem_map_translate is now a pointer. This allows the | |
1573 | function to report back the actual size of the translation relative to the original | |
1574 | request made by the user. | |
1575 | ||
1576 | A new structure spdk_mem_map_ops has been introduced to hold memory map related | |
1577 | callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc | |
1578 | in lieu of the notify callback. | |
1579 | ||
9f95a23c TL |
1580 | ### DPDK 18.08 |
1581 | ||
1582 | The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically | |
1583 | utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. | |
1584 | ||
1585 | Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, | |
1586 | but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try | |
1587 | to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using | |
1588 | only as many hugepages as it is really needed. | |
1589 | ||
1590 | Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed | |
1591 | to be physically contiguous. | |
1592 | ||
1593 | ### I/OAT | |
1594 | ||
1595 | I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. | |
1596 | ||
11fdf7f2 TL |
1597 | ### iscsi target |
1598 | ||
1599 | Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC | |
1600 | method for CHAP authentication in discovery sessions have been changed to | |
1601 | align with `construct_target_node` RPC method. Old names are still usable | |
1602 | but will be removed in future release. | |
1603 | ||
1604 | `set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have | |
1605 | been added to set CHAP authentication for discovery sessions and existing | |
1606 | target nodes, respectively. | |
1607 | ||
1608 | The SPDK iSCSI target supports an AuthFile which can be used to load CHAP | |
1609 | shared secrets when the iSCSI target starts. SPDK previously provided a | |
1610 | default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was | |
1611 | specified. This default has been removed. Users must now explicitly specify | |
1612 | the location of this file to load CHAP shared secrets from a file, or use | |
1613 | the related iSCSI RPC methods to add them at runtime. | |
1614 | ||
1615 | ### iscsi initiator | |
1616 | ||
1617 | The SPDK iSCSI initiator is no longer considered experimental and becomes | |
1618 | a first-class citizen among bdev modules. The basic usage has been briefly | |
1619 | described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) | |
1620 | ||
1621 | ### Miscellaneous | |
1622 | ||
9f95a23c TL |
1623 | The SPDK application framework can now parse long name command line parameters. |
1624 | Most single-character parameters have a long name equivalent now. See the | |
1625 | [Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation | |
1626 | for details or use the `--help` command line parameter to list all available | |
1627 | params. | |
1628 | ||
1629 | bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already | |
1630 | used by existing apps for memory size. | |
1631 | ||
1632 | bdevio can now accept all SPDK command line parameters. The config now has to | |
1633 | be provided with `-c` or `--config` param. | |
1634 | ||
1635 | The following ioat/perf and nvme/perf parameters were renamed as well: | |
1636 | `-s` (io size) to `-o` | |
1637 | `-d` (mem size) to `-s` | |
1638 | ||
11fdf7f2 TL |
1639 | The ReactorMask config file parameter has been deprecated. Users should |
1640 | use the -m or --cpumask command line option to specify the CPU core mask | |
1641 | for the application. | |
1642 | ||
1643 | Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt | |
1644 | and vhost. Config file pathnames may now only be specified using the | |
1645 | -c command line option. | |
1646 | ||
1647 | Users may no longer set DPDK_DIR in their environment to specify the | |
1648 | location of the DPDK installation used to build SPDK. Using DPDK_DIR | |
1649 | has not been the documented nor recommended way to specify the DPDK | |
1650 | location for several releases, but removing it ensures no unexpected | |
1651 | surprises for users who may have DPDK_DIR defined for other reasons. | |
1652 | Users should just use the "configure" script to specify the DPDK | |
1653 | location before building SPDK. | |
1654 | ||
1655 | Although we know that many developers still use Python 2 we are officially | |
1656 | switching to Python3 with requirement that all new code must be valid also | |
1657 | for Python 2 up to the EOL which is year 2020. | |
1658 | ||
1659 | Invoking interpreter explicitly is forbidden for executable scripts. There | |
1660 | is no need to use syntax like "python ./scripts/rpc.py". All executable | |
1661 | scripts must contain proper shebang pointing to the right interpreter. | |
1662 | Scripts without shebang musn't be executable. | |
1663 | ||
1664 | A Python script has been added to enable conversion of old INI config file | |
1665 | to new JSON-RPC config file format. This script can be found at | |
1666 | scripts/config_converter.py. Example how this script can be used: | |
1667 | ~~~{.sh} | |
1668 | cat old_format.ini | scripts/config_converter.py > new_json_format.json | |
1669 | ~~~ | |
1670 | ||
1671 | ### Sock | |
1672 | ||
1673 | Two additional parameters were added to spdk_sock_get_addr() for the server | |
1674 | port and client port. These parameters are named "sport" and "cport" | |
1675 | respectively. | |
1676 | ||
1677 | ### Virtio | |
1678 | ||
1679 | The following RPC commands have been deprecated: | |
f67539c2 | 1680 | |
11fdf7f2 TL |
1681 | - construct_virtio_user_scsi_bdev |
1682 | - construct_virtio_pci_scsi_bdev | |
1683 | - construct_virtio_user_blk_bdev | |
1684 | - construct_virtio_pci_blk_bdev | |
1685 | - remove_virtio_scsi_bdev | |
1686 | ||
1687 | The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` | |
1688 | command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` | |
1689 | was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI | |
1690 | devices. | |
1691 | ||
9f95a23c TL |
1692 | ### Blobfs |
1693 | ||
1694 | spdk_file_get_id() returning unique ID for the file was added. | |
1695 | ||
1696 | ### JSON | |
1697 | ||
1698 | Added jsonrpc-client C library intended for issuing RPC commands from applications. | |
1699 | ||
1700 | Added API enabling iteration over JSON object: | |
f67539c2 | 1701 | |
9f95a23c TL |
1702 | - spdk_json_find() |
1703 | - spdk_json_find_string() | |
1704 | - spdk_json_find_array() | |
1705 | - spdk_json_object_first() | |
1706 | - spdk_json_array_first() | |
1707 | - spdk_json_next() | |
1708 | ||
1709 | ### Blobstore | |
1710 | ||
1711 | Blobstore I/O operations are now based on io_units, instead of blobstore page size. | |
1712 | The io_unit size is now the same as the underlying block device's block size. | |
1713 | Logical volumes built on a block device with 512B block size can now be used as boot devices | |
1714 | in QEMU. | |
1715 | ||
1716 | ### SPDKCLI | |
1717 | ||
1718 | The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. | |
1719 | Support for the iSCSI and NVMe-oF targets has been added. | |
1720 | ||
f67539c2 | 1721 | ## v18.07 |
11fdf7f2 TL |
1722 | |
1723 | ### bdev | |
1724 | ||
1725 | A new public header file bdev_module.h has been introduced to facilitate the | |
1726 | development of new bdev modules. This header includes an interface for the | |
1727 | spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of | |
1728 | multiple virtual bdevs on top of a single base bdev and should act as the | |
1729 | primary API for module authors. | |
1730 | ||
1731 | spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide | |
1732 | options. | |
1733 | ||
1734 | A mechanism for handling out of memory condition errors (ENOMEM) returned from | |
1735 | I/O submission requests at the bdev layer has been added. See | |
1736 | spdk_bdev_queue_io_wait(). | |
1737 | ||
1738 | The spdk_bdev_get_io_stat() function now returns cumulative totals instead of | |
1739 | resetting on each call. This allows multiple callers to query I/O statistics | |
1740 | without conflicting with each other. Existing users will need to adjust their | |
1741 | code to record the previous I/O statistics to calculate the delta between calls. | |
1742 | ||
1743 | I/O queue depth tracking and samples options have been added. See | |
1744 | spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and | |
1745 | spdk_bdev_set_qd_sampling_period(). | |
1746 | ||
1747 | ### RAID module | |
f67539c2 | 1748 | |
11fdf7f2 TL |
1749 | A new bdev module called "raid" has been added as experimental module which |
1750 | aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note | |
1751 | that vhost will not work with this module because it does not yet have support | |
1752 | for multi-element io vectors. | |
1753 | ||
1754 | ### Log | |
1755 | ||
1756 | The debug log component flag available on several SPDK applications has been | |
1757 | renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the | |
1758 | option to be added to tools that already use `-t` to mean something else. | |
1759 | ||
1760 | ### Blobstore | |
1761 | ||
1762 | A new function, spdk_bs_dump(), has been added that dumps all of the contents of | |
1763 | a blobstore to a file pointer. This includes the metadata and is very useful for | |
1764 | debugging. | |
1765 | ||
1766 | Two new operations have been added for thin-provisioned blobs. | |
1767 | spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions | |
1768 | of the blob and populate them with the correct data by reading from the backing | |
1769 | blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate | |
1770 | clusters that correspond to data in the blob's immediate parent. Clusters | |
1771 | allocated to grandparents or that aren't allocated at all will remain | |
1772 | thin-provisioned. | |
1773 | ||
1774 | ### BlobFS | |
1775 | ||
1776 | Changed the return type of spdk_file_truncate() from void to int to allow the | |
1777 | propagation of `ENOMEM` errors. | |
1778 | ||
1779 | ### NVMe Driver | |
1780 | ||
1781 | The new API functions spdk_nvme_qpair_add_cmd_error_injection() and | |
1782 | spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error | |
1783 | emulation. Users can set a specified command to fail with a particular error | |
1784 | status. | |
1785 | ||
1786 | Changed the name `timeout_sec` parameter to `timeout_us` in | |
1787 | spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from | |
1788 | uint32_t to uint64_t. This will give users more fine-grained control over the | |
1789 | timeout period. | |
1790 | ||
1791 | Basic support for Open Channel SSDs was added. See nvme_ocssd.h | |
1792 | ||
1793 | ### NVMe Over Fabrics | |
1794 | ||
1795 | The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback | |
1796 | as a parameter. | |
1797 | ||
1798 | spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. | |
1799 | ||
1800 | spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed | |
1801 | number of namespaces for a given subsystem. | |
1802 | ||
1803 | ### Build System | |
1804 | ||
1805 | The build system now generates a combined shared library (libspdk.so) that may | |
1806 | be used in place of the individual static libraries (libspdk_*.a). The combined | |
1807 | library includes all components of SPDK and is intended to make linking against | |
1808 | SPDK easier. The static libraries are also still provided for users that prefer | |
1809 | to link only the minimal set of components required. | |
1810 | ||
1811 | ### git pre-commit and pre-push hooks | |
1812 | ||
1813 | The pre-commit hook will run `scripts/check_format.sh` and verify there are no | |
1814 | formating errors before allowing `git commit` to run. The pre-push hook runs | |
1815 | `make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and | |
1816 | clang compiler before allowing `git push` to run. Following each DEBUG build | |
1817 | `test/unit/unittest.sh` is run and verified. Results are recorded in the | |
1818 | `make.log` file. | |
1819 | ||
1820 | To enable type: 'git config core.hooksPath .githooks'. To override after | |
1821 | configuration use the `git --no-verify` flag. | |
1822 | ||
1823 | ### RPC | |
1824 | ||
1825 | The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node | |
1826 | rather than always returning `true`. | |
1827 | ||
1828 | ### DPDK 18.05 | |
1829 | ||
1830 | The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports | |
1831 | dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, | |
1832 | that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use | |
1833 | the legacy memory allocation model. The plan is to enable dynamic memory allocation | |
1834 | after the DPDK 18.08 release which should fix these issues. | |
1835 | ||
1836 | ### Environment Abstraction Layer and Event Framework | |
1837 | ||
1838 | The spdk_mem_map_translate() function now takes a size parameter to indicate the size of | |
1839 | the memory region. This can be used by environment implementations to validate the | |
1840 | requested translation. | |
1841 | ||
1842 | The I/O Channel implementation has been moved to its own library - lib/thread. The | |
1843 | public API that was previously in spdk/io_channel.h is now in spdk/thread.h The | |
1844 | file spdk/io_channel.h remains and includes spdk/thread.h. | |
1845 | ||
1846 | spdk_reactor_get_tsc_stats was added to return interesting statistics for each | |
1847 | reactor. | |
1848 | ||
1849 | ### IOAT | |
1850 | ||
1851 | IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable | |
1852 | option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is | |
1853 | now deprecated and will be removed in a future release. | |
1854 | ||
1855 | ## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP | |
1856 | ||
1857 | ### vhost | |
1858 | ||
1859 | The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the | |
1860 | DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). | |
1861 | Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) | |
1862 | for additional information on the DPDK vulnerability. | |
1863 | ||
1864 | Workarounds have been added to ensure vhost compatibility with QEMU 2.12. | |
1865 | ||
1866 | EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the | |
1867 | [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. | |
1868 | ||
1869 | ### Unified Target Application | |
1870 | ||
1871 | A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the | |
1872 | functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, | |
1873 | and vhost target. The new application can be managed through the existing configuration file and | |
1874 | [JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. | |
1875 | ||
1876 | ### Env | |
1877 | ||
1878 | spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). | |
1879 | ||
1880 | New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. | |
1881 | These new functions have a `flags` parameter that allows the user to specify whether the allocated | |
1882 | memory needs to be suitable for DMA and whether it should be shared across processes with the same | |
1883 | shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will | |
1884 | eventually be deprecated and removed. | |
1885 | ||
1886 | ### Bdev | |
1887 | ||
1888 | A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules | |
1889 | when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require | |
1890 | notification that the set of initialization examine() calls is complete. | |
1891 | ||
1892 | The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with | |
1893 | the spdk_bdev_get_uuid() function. | |
1894 | ||
1895 | Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the | |
1896 | [set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation | |
1897 | for more details. | |
1898 | ||
1899 | ### RPC | |
1900 | ||
1901 | The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. | |
1902 | Users should switch to the `-r` command-line parameter instead. | |
1903 | ||
1904 | The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as | |
1905 | needed; previously, the response was limited to 32 kilobytes. | |
1906 | ||
1907 | ### SPDKCLI | |
1908 | ||
1909 | EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. | |
1910 | See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. | |
1911 | ||
1912 | ### NVMe Driver | |
1913 | ||
1914 | EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is | |
1915 | experimental pending a functional allocator to free and reallocate CMB buffers. | |
1916 | ||
1917 | spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. | |
1918 | ||
1919 | New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() | |
1920 | have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if | |
1921 | a namespace ID is active. | |
1922 | ||
1923 | ### NVMe-oF Target | |
1924 | ||
1925 | Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters | |
1926 | to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes | |
1927 | the backing bdev's UUID as the namespace UUID when available. | |
1928 | ||
1929 | spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. | |
1930 | ||
1931 | ### Blobstore | |
1932 | ||
1933 | A number of functions have been renamed: | |
1934 | ||
1935 | - spdk_bs_io_write_blob() => spdk_blob_io_write() | |
1936 | - spdk_bs_io_read_blob() => spdk_blob_io_read() | |
1937 | - spdk_bs_io_writev_blob() => spdk_blob_io_writev() | |
1938 | - spdk_bs_io_readv_blob() => spdk_blob_io_readv() | |
1939 | - spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() | |
1940 | - spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() | |
1941 | ||
1942 | The old names still exist but are deprecated. They will be removed in the v18.07 release. | |
1943 | ||
1944 | spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O | |
1945 | are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still | |
1946 | required to sync the updated metadata to disk. | |
1947 | ||
1948 | ### Logical Volumes | |
1949 | ||
1950 | A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. | |
1951 | ||
1952 | Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. | |
1953 | ||
1954 | New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols | |
1955 | and Clones of existing Snapshots. | |
1956 | See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation | |
1957 | for more details. | |
1958 | ||
1959 | Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. | |
1960 | ||
1961 | ### Lib | |
1962 | ||
1963 | A set of changes were made in the SPDK's lib code altering | |
1964 | instances of calls to `exit()` and `abort()` to return a failure instead | |
1965 | wherever reasonably possible. | |
1966 | ||
1967 | spdk_app_start() no longer exit()'s on an internal failure, but | |
1968 | instead returns a non-zero error status. | |
1969 | ||
1970 | spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid | |
1971 | option, but instead returns SPDK_APP_PARSE_ARGS_HELP and | |
1972 | SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS | |
1973 | on success. | |
1974 | ||
1975 | spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. | |
1976 | ||
1977 | ### I/O Channels | |
1978 | ||
1979 | The prototype for spdk_poller_fn() has been modified; it now returns a value indicating | |
1980 | whether or not the poller did any work. Existing pollers will need to be updated to | |
1981 | return a value. | |
1982 | ||
1983 | ### iSCSI Target | |
1984 | ||
1985 | The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace | |
1986 | TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more | |
1987 | details. | |
1988 | ||
1989 | ### iSCSI initiator | |
1990 | ||
1991 | An iSCSI initiator bdev module has been added to SPDK. This module should be considered | |
1992 | experimental pending additional features and tests. More details can be found in | |
1993 | lib/bdev/iscsi/README. | |
1994 | ||
1995 | ### PMDK | |
1996 | ||
1997 | The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of | |
1998 | `--with-nvml`. This reflects the renaming of the persistent memory library from NVML to | |
1999 | PMDK. | |
2000 | ||
2001 | ### Virtio Block driver | |
2002 | ||
2003 | A userspace driver for Virtio Block devices has been added. It was built on top of the | |
2004 | [Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to | |
2005 | the Virtio SCSI driver. See the | |
2006 | [Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for | |
2007 | more information. | |
2008 | ||
2009 | ### Virtio with 2MB hugepages | |
2010 | ||
2011 | The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option | |
2012 | enables SPDK Virtio to work with 2MB hugepages. | |
2013 | See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. | |
2014 | ||
2015 | ## v18.01: Blobstore Thin Provisioning | |
2016 | ||
2017 | ### Build System | |
2018 | ||
2019 | The build system now includes a `make install` rule, including support for the common | |
2020 | `DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix | |
2021 | may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. | |
2022 | ||
2023 | ### RPC | |
2024 | ||
2025 | A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/run/spdk.sock. | |
2026 | A -r option command line option has been added to enable an alternative UNIX domain socket location, | |
2027 | or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file | |
2028 | section is now deprecated and will be removed in the v18.04 release. | |
2029 | ||
2030 | ### I/O Channels | |
2031 | ||
2032 | spdk_poller_register() and spdk_poller_unregister() were moved from the event | |
2033 | framework (include/spdk/event.h) to the I/O channel library | |
2034 | (include/spdk/io_channel.h). This allows code that doesn't depend on the event | |
2035 | framework to request registration and unregistration of pollers. | |
2036 | ||
2037 | spdk_for_each_channel() now allows asynchronous operations during iteration. | |
2038 | Instead of immediately continuing the interation upon returning from the iteration | |
2039 | callback, the user must call spdk_for_each_channel_continue() to resume iteration. | |
2040 | ||
2041 | ### Block Device Abstraction Layer (bdev) | |
2042 | ||
2043 | The poller abstraction was removed from the bdev layer. There is now a general purpose | |
2044 | abstraction for pollers available in include/spdk/io_channel.h | |
2045 | ||
2046 | ### Lib | |
2047 | ||
2048 | A set of changes were made in the SPDK's lib code altering, | |
2049 | instances of calls to `exit()` and `abort()` to return a failure instead | |
2050 | wherever reasonably possible. This has resulted in return type changes of | |
2051 | the API for: | |
2052 | ||
2053 | - spdk_env_init() from type `void` to `int`. | |
2054 | - spdk_mem_map_init() from type `void` to `int`. | |
2055 | ||
2056 | Applications making use of these APIs should be modified to check for | |
2057 | a non-zero return value instead of relying on them to fail without return. | |
2058 | ||
2059 | ### NVMe Driver | |
2060 | ||
2061 | SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: | |
2062 | Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. | |
2063 | ||
2064 | ### NVMe-oF Target | |
2065 | ||
2066 | Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available | |
2067 | cores round-robin. The "Core" option in the configuration file has been removed. | |
2068 | ||
2069 | ### Blobstore | |
2070 | ||
2071 | A number of functions have been renamed: | |
2072 | ||
2073 | - spdk_bs_md_resize_blob() => spdk_blob_resize() | |
2074 | - spdk_bs_md_sync_blob() => spdk_blob_sync_md() | |
2075 | - spdk_bs_md_close_blob() => spdk_blob_close() | |
2076 | - spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() | |
2077 | - spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() | |
2078 | - spdk_blob_md_set_xattr() => spdk_blob_set_xattr() | |
2079 | - spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() | |
2080 | - spdk_bs_md_create_blob() => spdk_bs_create_blob() | |
2081 | - spdk_bs_md_open_blob() => spdk_bs_open_blob() | |
2082 | - spdk_bs_md_delete_blob() => spdk_bs_delete_blob() | |
2083 | - spdk_bs_md_iter_first() => spdk_bs_iter_first() | |
2084 | - spdk_bs_md_iter_next() => spdk_bs_iter_next() | |
2085 | ||
2086 | The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument | |
2087 | rather than struct spdk_blob **. | |
2088 | ||
2089 | The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument | |
2090 | rather than struct spdk_blob **. | |
2091 | ||
2092 | Thin provisioning support has been added to the blobstore. It can be enabled by setting the | |
2093 | `thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). | |
2094 | ||
2095 | ### NBD device | |
2096 | ||
2097 | The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be | |
2098 | achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. | |
2099 | See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. | |
2100 | ||
2101 | ### FIO plugin | |
2102 | ||
2103 | SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, | |
2104 | although it still remains to work for now. The new FIO contains huge amount of bugfixes and | |
2105 | it's recommended to do an update. | |
2106 | ||
2107 | ### Virtio library | |
2108 | ||
2109 | Previously a part of the bdev_virtio module, now a separate library. Virtio is now available | |
2110 | via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing | |
2111 | new Virtio backends, namely Virtio-BLK. | |
2112 | ||
2113 | ### iSCSI | |
2114 | ||
2115 | The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated | |
2116 | to an epoll/kqueue descriptor on the master core when idle. | |
2117 | ||
2118 | ## v17.10: Logical Volumes | |
2119 | ||
2120 | ### New dependencies | |
2121 | ||
2122 | libuuid was added as new dependency for logical volumes. | |
2123 | ||
2124 | libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. | |
2125 | ||
2126 | ### Block Device Abstraction Layer (bdev) | |
2127 | ||
2128 | An [fio](http://github.com/axboe/fio) plugin was added that can route | |
2129 | I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) | |
2130 | for more information. | |
2131 | ||
2132 | spdk_bdev_unmap() was modified to take an offset and a length in bytes as | |
2133 | arguments instead of requiring the user to provide an array of SCSI | |
2134 | unmap descriptors. This limits unmaps to a single contiguous range. | |
2135 | ||
2136 | spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. | |
2137 | If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using | |
2138 | write commands. | |
2139 | ||
2140 | New API functions that accept I/O parameters in units of blocks instead of bytes | |
2141 | have been added: | |
f67539c2 | 2142 | |
11fdf7f2 TL |
2143 | - spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() |
2144 | - spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() | |
2145 | - spdk_bdev_write_zeroes_blocks() | |
2146 | - spdk_bdev_unmap_blocks() | |
2147 | ||
2148 | The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be | |
2149 | retried later. | |
2150 | ||
2151 | ### Linux AIO bdev | |
2152 | ||
2153 | The AIO bdev now allows the user to override the auto-detected block size. | |
2154 | ||
2155 | ### NVMe driver | |
2156 | ||
2157 | The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. | |
2158 | NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take | |
2159 | into account when splitting I/O requests. | |
2160 | ||
2161 | The HotplugEnable option in `[Nvme]` sections of the configuration file is now | |
2162 | "No" by default. It was previously "Yes". | |
2163 | ||
2164 | The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the | |
2165 | NVMe Controller associated with a given namespace. | |
2166 | ||
2167 | The NVMe library now allows the user to specify a host identifier when attaching | |
2168 | to a controller. The host identifier is used as part of the Reservations feature, | |
2169 | as well as in the NVMe-oF Connect command. The default host ID is also now a | |
2170 | randomly-generated UUID, and the default host NQN uses the host ID to generate | |
2171 | a UUID-based NQN. | |
2172 | ||
2173 | spdk_nvme_connect() was added to allow the user to connect directly to a single | |
2174 | NVMe or NVMe-oF controller. | |
2175 | ||
2176 | ### NVMe-oF Target (nvmf_tgt) | |
2177 | ||
2178 | The NVMe-oF target no longer requires any in-capsule data buffers to run, and | |
2179 | the feature is now entirely optional. Previously, at least 4 KiB in-capsule | |
2180 | data buffers were required. | |
2181 | ||
2182 | NVMe-oF subsytems have a new configuration option, AllowAnyHost, to control | |
2183 | whether the host NQN whitelist is enforced when accepting new connections. | |
2184 | If no Host options have been specified and AllowAnyHost is disabled, the | |
2185 | connection will be denied; this is a behavior change from previous releases, | |
2186 | which allowed any host NQN to connect if the Host list was empty. | |
2187 | AllowAnyHost is disabled by default. | |
2188 | ||
2189 | NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number | |
2190 | of namespaces per subsystem is no longer limited. | |
2191 | ||
2192 | The NVMe-oF target now supports the Write Zeroes command. | |
2193 | ||
2194 | ### Environment Abstraction Layer | |
2195 | ||
2196 | A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide | |
2197 | additional clarity when constructing spdk_mempools. Previously, -1 could be | |
2198 | passed and the library would choose a reasonable default, but this new value | |
2199 | makes it explicit that the default is being used. | |
2200 | ||
2201 | ### Blobstore | |
2202 | ||
2203 | The blobstore super block now contains a bstype field to identify the type of the blobstore. | |
2204 | Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). | |
2205 | ||
2206 | spdk_bs_destroy() was added to allow destroying blobstore on device | |
2207 | with an initialized blobstore. | |
2208 | ||
2209 | spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable | |
2210 | scattered payloads. | |
2211 | ||
2212 | A CLI tool for blobstore has been added, allowing basic operations through either command | |
2213 | line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) | |
2214 | documentation for more details. | |
2215 | ||
2216 | ### Event Framework | |
2217 | ||
2218 | The ability to set a thread name, previously only used by the reactor code, is | |
2219 | now part of the spdk_thread_allocate() API. Users may specify a thread name | |
2220 | which will show up in tools like `gdb`. | |
2221 | ||
2222 | ### Log | |
2223 | ||
2224 | The spdk_trace_dump() function now takes a new parameter to allow the caller to | |
2225 | specify an output file handle (stdout or stderr, for example). | |
2226 | ||
2227 | ### Logical Volumes | |
2228 | ||
2229 | Logical volumes library built on top of SPDK blobstore has been added. | |
2230 | It is possible to create logical volumes on top of other devices using RPC. | |
2231 | ||
2232 | See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. | |
2233 | ||
2234 | ### Persistent Memory | |
2235 | ||
2236 | A new persistent memory bdev type has been added. | |
2237 | The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). | |
2238 | It is possible to create pmem devices on top of pmem pool files using RPC. | |
2239 | ||
2240 | See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. | |
2241 | ||
2242 | ### Virtio SCSI driver | |
2243 | ||
2244 | A userspace driver for Virtio SCSI devices has been added. | |
2245 | The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. | |
2246 | ||
2247 | See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and [Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. | |
2248 | ||
2249 | ### Vhost target | |
2250 | ||
2251 | The vhost target application now supports live migration between QEMU instances. | |
2252 | ||
11fdf7f2 TL |
2253 | ## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev |
2254 | ||
2255 | ### Build System | |
2256 | ||
2257 | A `configure` script has been added to simplify the build configuration process. | |
2258 | The existing CONFIG file and `make CONFIG_...` options are also still supported. | |
2259 | Run `./configure --help` for information about available configuration options. | |
2260 | ||
2261 | A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` | |
2262 | option is specified to configure, the SPDK build system will automatically build a | |
2263 | known-good configuration of DPDK with the minimal options enabled. See the Building | |
2264 | section of README.md for more information. | |
2265 | ||
2266 | A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to | |
2267 | develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant | |
2268 | section of README.md for more information. | |
2269 | ||
2270 | ### Userspace vhost-blk target | |
2271 | ||
2272 | The vhost library and example app have been updated to support the vhost-blk | |
2273 | protocol in addition to the existing vhost-scsi protocol. | |
2274 | See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. | |
2275 | ||
2276 | ### Block device abstraction layer (bdev) | |
2277 | ||
2278 | A GPT virtual block device has been added, which automatically exposes GPT partitions | |
2279 | with a special SPDK-specific partition type as bdevs. | |
2280 | See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for | |
2281 | more information. | |
2282 | ||
2283 | ### NVMe driver | |
2284 | ||
2285 | The NVMe driver has been updated to support recent Intel SSDs, including the Intel® | |
2286 | Optane™ SSD DC P4800X series. | |
2287 | ||
2288 | A workaround has been added for devices that failed to recognize register writes | |
2289 | during controller reset. | |
2290 | ||
2291 | The NVMe driver now allocates request tracking objects on a per-queue basis. The | |
2292 | number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by | |
2293 | modifying `io_queue_requests` in the opts structure. | |
2294 | ||
2295 | The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). | |
2296 | ||
2297 | spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override | |
2298 | controller-level options for each individual I/O queue pair. | |
2299 | Existing callers with qprio == 0 can be updated to: | |
2300 | ~~~ | |
2301 | ... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); | |
2302 | ~~~ | |
2303 | Callers that need to specify a non-default qprio should be updated to: | |
2304 | ~~~ | |
2305 | struct spdk_nvme_io_qpair_opts opts; | |
2306 | spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); | |
2307 | opts.qprio = SPDK_NVME_QPRIO_...; | |
2308 | ... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); | |
2309 | ~~~ | |
2310 | ||
2311 | ### Environment Abstraction Layer | |
2312 | ||
2313 | The environment abstraction layer has been updated to include several new functions | |
2314 | in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the | |
2315 | current set of functions. | |
2316 | ||
2317 | ### SPDK Performance Analysis with Intel® VTune™ Amplifier | |
2318 | ||
2319 | Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. | |
2320 | ||
2321 | This analysis provides: | |
f67539c2 | 2322 | |
11fdf7f2 TL |
2323 | - I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) |
2324 | - Tuning insights on the interplay of I/O and compute devices by estimating how many cores | |
2325 | would be reasonable to provide for SPDK to keep up with a current storage workload. | |
2326 | ||
2327 | See the VTune Amplifier documentation for more information. | |
2328 | ||
7c673cae FG |
2329 | ## v17.03: Blobstore and userspace vhost-scsi target |
2330 | ||
2331 | ### Blobstore and BlobFS | |
2332 | ||
2333 | The blobstore is a persistent, power-fail safe block allocator designed to be | |
2334 | used as the local storage system backing a higher-level storage service. | |
2335 | See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. | |
2336 | ||
2337 | BlobFS adds basic filesystem functionality like filenames on top of the blobstore. | |
2338 | This release also includes a RocksDB Env implementation using BlobFS in place of the | |
2339 | kernel filesystem. | |
2340 | See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. | |
2341 | ||
2342 | ### Userspace vhost-scsi target | |
2343 | ||
2344 | A userspace implementation of the QEMU vhost-scsi protocol has been added. | |
2345 | The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. | |
2346 | See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. | |
2347 | ||
2348 | ### Event framework | |
2349 | ||
2350 | The overhead of the main reactor event loop was reduced by optimizing the number of | |
2351 | calls to spdk_get_ticks() per iteration. | |
2352 | ||
2353 | ### NVMe library | |
2354 | ||
2355 | The NVMe library will now automatically split readv/writev requests with scatter-gather | |
2356 | lists that do not map to valid PRP lists when the NVMe controller does not natively | |
2357 | support SGLs. | |
2358 | ||
2359 | The `identify` and `perf` NVMe examples were modified to add a consistent format for | |
2360 | specifying remote NVMe over Fabrics devices via the `-r` option. | |
2361 | This is implemented using the new `spdk_nvme_transport_id_parse()` function. | |
2362 | ||
2363 | ### iSCSI Target | |
2364 | ||
2365 | The [Nvme] section of the configuration file was modified to remove the `BDF` directive | |
2366 | and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) | |
2367 | devices can now be specified as the backing block device. A script to generate an | |
2368 | entire [Nvme] section based on the local NVMe devices attached was added at | |
2369 | `scripts/gen_nvme.sh`. | |
2370 | ||
2371 | ### NVMe-oF Target | |
2372 | ||
2373 | The [Nvme] section of the configuration file was modified to remove the `BDF` directive | |
2374 | and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) | |
2375 | devices can now be specified as the backing block device. A script to generate an | |
2376 | entire [Nvme] section based on the local NVMe devices attached was added at | |
2377 | `scripts/gen_nvme.sh`. | |
2378 | ||
2379 | ## v16.12: NVMe over Fabrics host, hotplug, and multi-process | |
2380 | ||
2381 | ### NVMe library | |
2382 | ||
2383 | The NVMe library has been changed to create its own request memory pool rather than | |
2384 | requiring the user to initialize the global `request_mempool` variable. Apps can be | |
2385 | updated by simply removing the initialization of `request_mempool`. Since the NVMe | |
2386 | library user no longer needs to know the size of the internal NVMe request | |
2387 | structure to create the pool, the `spdk_nvme_request_size()` function was also removed. | |
2388 | ||
2389 | The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become | |
2390 | `spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe | |
2391 | Dataset Management command's parameters. Existing callers can be updated to use | |
2392 | `spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the | |
2393 | `type` parameter. | |
2394 | ||
2395 | The NVMe library SGL callback prototype has been changed to return virtual addresses | |
2396 | rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and | |
2397 | `spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. | |
2398 | ||
2399 | The NVMe library now supports NVMe over Fabrics devices in addition to the existing | |
2400 | support for local PCIe-attached NVMe devices. For an example of how to enable | |
2401 | NVMe over Fabrics support in an application, see `examples/nvme/identify` and | |
2402 | `examples/nvme/perf`. | |
2403 | ||
2404 | Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug | |
2405 | support, an application should call the `spdk_nvme_probe()` function on a regular | |
2406 | basis to probe for new devices (reported via the existing `probe_cb` callback) and | |
2407 | removed devices (reported via a new `remove_cb` callback). Hotplug is currently | |
2408 | only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe | |
2409 | devices must be bound to `uio_pci_generic` by an external script or tool. | |
2410 | ||
2411 | Multiple processes may now coordinate and use a single NVMe device simultaneously | |
2412 | using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). | |
2413 | ||
2414 | ### NVMe over Fabrics target (`nvmf_tgt`) | |
2415 | ||
2416 | The `nvmf_tgt` configuration file format has been updated significantly to enable | |
2417 | new features. See the example configuration file `etc/spdk/nvmf.conf.in` for | |
2418 | more details on the new and changed options. | |
2419 | ||
2420 | The NVMe over Fabrics target now supports virtual mode subsystems, which allow the | |
2421 | user to export devices from the SPDK block device abstraction layer as NVMe over | |
2422 | Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, | |
2423 | and a single `nvmf_tgt` may export both types of subsystems simultaneously. | |
2424 | ||
2425 | ### Block device abstraction layer (bdev) | |
2426 | ||
2427 | The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe | |
2428 | blockdev driver has been updated to support scatter/gather. Apps can use the | |
2429 | new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` | |
2430 | functions. | |
2431 | ||
2432 | The bdev status returned from each I/O has been extended to pass through NVMe | |
2433 | or SCSI status codes directly in cases where the underlying device can provide | |
2434 | a more specific status code. | |
2435 | ||
2436 | A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the | |
2437 | `iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or | |
2438 | NVMe namespaces. | |
2439 | ||
2440 | ### General changes | |
2441 | ||
2442 | `libpciaccess` has been removed as a dependency and DPDK PCI enumeration is | |
2443 | used instead. Prior to DPDK 16.07 enumeration by class code was not supported, | |
2444 | so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered | |
2445 | by the NVMe library. | |
2446 | ||
2447 | The `env` environment abstraction library has been introduced, and a default | |
2448 | DPDK-based implementation is provided as part of SPDK. The goal of the `env` | |
2449 | layer is to enable use of alternate user-mode memory allocation and PCI access | |
2450 | libraries. See `doc/porting.md` for more details. | |
2451 | ||
2452 | The build process has been modified to produce all of the library files in the | |
2453 | `build/lib` directory. This is intended to simplify the use of SPDK from external | |
2454 | projects, which can now link to SPDK libraries by adding the `build/lib` directory | |
2455 | to the library path via `-L` and linking the SPDK libraries by name (for example, | |
2456 | `-lspdk_nvme -lspdk_log -lspdk_util`). | |
2457 | ||
2458 | `nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user | |
2459 | to query and modify the configuration at runtime. The RPC service is disabled by | |
2460 | default, since it currently does not provide any authentication or security | |
2461 | mechanisms; it should only be enabled on systems with controlled user access | |
2462 | behind a firewall. An example RPC client implemented in Python is provided in | |
2463 | `scripts/rpc.py`. | |
2464 | ||
2465 | ## v16.08: iSCSI target, NVMe over Fabrics maturity | |
2466 | ||
2467 | This release adds a userspace iSCSI target. The iSCSI target is capable of exporting | |
2468 | NVMe devices over a network using the iSCSI protocol. The application is located | |
2469 | in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. | |
2470 | ||
2471 | This release also significantly improves the existing NVMe over Fabrics target. | |
f67539c2 | 2472 | |
7c673cae FG |
2473 | - The configuration file format was changed, which will require updates to |
2474 | any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): | |
2475 | - `SubsystemGroup` was renamed to `Subsystem`. | |
2476 | - `AuthFile` was removed (it was unimplemented). | |
2477 | - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) | |
2478 | when naming subsystems. The default node name was changed to reflect this; | |
2479 | it is now "nqn.2016-06.io.spdk". | |
2480 | - `Port` and `Host` sections were merged into the `Subsystem` section | |
2481 | - Global options to control max queue depth, number of queues, max I/O | |
2482 | size, and max in-capsule data size were added. | |
2483 | - The Nvme section was removed. Now a list of devices is specified by | |
2484 | bus/device/function directly in the Subsystem section. | |
2485 | - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt | |
2486 | to future-proof the interface, so the only mode supported by this release | |
2487 | is "Direct". | |
2488 | - Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. | |
2489 | - The target now supports discovery. | |
2490 | ||
2491 | This release also adds one new feature and provides some better examples and tools | |
2492 | for the NVMe driver. | |
f67539c2 | 2493 | |
7c673cae FG |
2494 | - The Weighted Round Robin arbitration method is now supported. This allows |
2495 | the user to specify different priorities on a per-I/O-queue basis. To | |
2496 | enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. | |
2497 | - A simplified "Hello World" example was added to show the proper way to use | |
2498 | the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. | |
2499 | - A test for measuring software overhead was added. See `test/lib/nvme/overhead`. | |
2500 | ||
2501 | ## v16.06: NVMf userspace target | |
2502 | ||
2503 | This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the | |
2504 | newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe | |
2505 | devices from a host machine over the network via RDMA. Currently, the target is | |
2506 | limited to directly exporting physical NVMe devices, and the discovery subsystem | |
2507 | is not supported. | |
2508 | ||
2509 | This release includes a general API cleanup, including renaming all declarations | |
2510 | in public headers to include a `spdk` prefix to prevent namespace clashes with | |
2511 | user code. | |
2512 | ||
2513 | - NVMe | |
2514 | - The `nvme_attach()` API was reworked into a new probe/attach model, which | |
2515 | moves device detection into the NVMe library. The new API also allows | |
2516 | parallel initialization of NVMe controllers, providing a major reduction in | |
2517 | startup time when using multiple controllers. | |
2518 | - I/O queue allocation was changed to be explicit in the API. Each function | |
2519 | that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) | |
2520 | argument, and I/O queues may be allocated using | |
2521 | `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of | |
2522 | queue pairs than the previous model, which only allowed a single queue | |
2523 | per thread and limited the total number of I/O queues to the lowest number | |
2524 | supported on any attached controller. | |
2525 | - Added support for the Write Zeroes command. | |
2526 | - `examples/nvme/perf` can now report I/O command latency from the | |
2527 | the controller's viewpoint using the Intel vendor-specific read/write latency | |
2528 | log page. | |
2529 | - Added namespace reservation command support, which can be used to coordinate | |
2530 | sharing of a namespace between multiple hosts. | |
2531 | - Added hardware SGL support, which enables use of scattered buffers that | |
2532 | don't conform to the PRP list alignment and length requirements on supported | |
2533 | NVMe controllers. | |
2534 | - Added end-to-end data protection support, including the ability to write and | |
2535 | read metadata in extended LBA (metadata appended to each block of data in the | |
2536 | buffer) and separate metadata buffer modes. | |
2537 | See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` | |
2538 | for details. | |
2539 | - IOAT | |
2540 | - The DMA block fill feature is now exposed via the `ioat_submit_fill()` | |
2541 | function. This is functionally similar to `memset()`, except the memory is | |
2542 | filled with an 8-byte repeating pattern instead of a single byte like memset. | |
2543 | - PCI | |
2544 | - Added support for using DPDK for PCI device mapping in addition to the | |
2545 | existing libpciaccess option. Using the DPDK PCI support also allows use of | |
2546 | the Linux VFIO driver model, which means that SPDK userspace drivers will work | |
2547 | with the IOMMU enabled. Additionally, SPDK applications may be run as an | |
2548 | unprivileged user with access restricted to a specific set of PCIe devices. | |
2549 | - The PCI library API was made more generic to abstract away differences | |
2550 | between the underlying PCI access implementations. | |
2551 | ||
2552 | ## v1.2.0: IOAT user-space driver | |
2553 | ||
2554 | This release adds a user-space driver with support for the Intel I/O Acceleration Technology (I/OAT, also known as "Crystal Beach") DMA offload engine. | |
2555 | ||
2556 | - IOAT | |
2557 | - New user-space driver supporting DMA memory copy offload | |
2558 | - Example programs `ioat/perf` and `ioat/verify` | |
2559 | - Kernel-mode DMA engine test driver `kperf` for performance comparison | |
2560 | - NVMe | |
2561 | - Per-I/O flags for Force Unit Access (FUA) and Limited Retry | |
2562 | - Public API for retrieving log pages | |
2563 | - Reservation register/acquire/release/report command support | |
2564 | - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks | |
2565 | - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features | |
2566 | - Updated to support DPDK 2.2.0 | |
2567 | ||
7c673cae FG |
2568 | ## v1.0.0: NVMe user-space driver |
2569 | ||
2570 | This is the initial open source release of the Storage Performance Development Kit (SPDK). | |
2571 | ||
2572 | Features: | |
f67539c2 | 2573 | |
7c673cae FG |
2574 | - NVMe user-space driver |
2575 | - NVMe example programs | |
2576 | - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver | |
2577 | - `examples/nvme/identify` displays NVMe controller information in a human-readable format | |
2578 | - Linux and FreeBSD support |