3 ## v19.07: (Upcoming Release)
7 A new file API `spdk_posix_file_load` was added to load file content into a data buffer.
11 Shared receive queue can now be disabled even for NICs that support it using the
12 `nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared
13 receive queue is predicated on hardware support when this flag is not used.
17 The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were
18 renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`,
19 respectively. And update type name of callback accordingly.
23 The format of the data returned by the get_bdevs_iostat RPC has changed to
24 make it easier to parse. It now returns an object with a "ticks" object
25 and "bdevs" array with the per-bdev statistics.
27 A new bdev module `delay` has been added which simulates a drive latency when placed
28 on top of a Null bdev. This module is intended only for testing and can be created using
29 the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well
30 as average and p99 latency arguments for both read and write operations. Average latency is
31 defined as a value close to what you would expect a perf tool such as FIO to report back as
32 the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one
33 would expect the drive to see the slowest 1% of I/O report. For underlying drives with already
34 significant latency, the latency values provided to the drive will be additive. This should be
35 taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device.
39 Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a
40 previously attached controller.
42 Nvme Opal library spdk_opal_cmd deprecated. Adding seperate command APIs.
46 The parameter `free_space` has been added to spdk_ring_enqueue() to wait when
47 the ring is almost full and resume when there is enough space available in
54 Added asynchronous probe support. New APIs spdk_nvme_probe_async(),
55 spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable
56 this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a
57 context associated with the specified controllers. Users then call
58 spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation
61 A new qpair creation option, delay_pcie_doorbell, was added. This can be passed
62 to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as
63 spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the
64 doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions().
65 This can result in significantly fewer MMIO writes to the doorbell register
66 under heavy load, greatly improving performance.
68 spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller
69 memory buffer, if available.
71 spdk_nvme_ctrlr_get_flags() was added to return controller feature
72 flags. Two flags are currently tracked:
73 SPDK_NVME_CTRLR_SGL_SUPPORTED
74 SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED
76 The NVMe hotplug poller is now able to detach devices hot-removed from the system
77 via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`.
79 Opal support was added for scan, take ownership, revert TPer, and dumping device
80 info. The nvme_manage tool can be used to perform these operations. The public
81 API functions are spdk_nvme_ctrlr_security_receive() and
82 spdk_nvme_ctrlr_security_send(). This module should be considered experimental
83 pending additional features and tests.
87 Added new strip_size_kb rpc param on create to replace the more ambiguous
88 strip_size. The strip_size rpc param is deprecated.
90 Changed the raid bdev product_name from "Pooled Device" to "Raid Volume"
94 Added spdk_thread_has_pollers() function to verify if there are any registered
95 pollers to be run on the thread. Added spdk_thread_is_idle() function to check
96 if there are any scheduled operations to be performed on the thread at given
99 spdk_thread_create() now takes an optional CPU affinity mask that is passed to
102 spdk_thread_lib_init() now takes an optional context size. For each thread
103 created, an additional region of memory of this size will be allocated. A
104 pointer to this region of memory can be obtained by calling
105 spdk_thread_get_ctx(). The inverse operation is also available via
106 spdk_thread_get_from_ctx().
108 spdk_thread_poll() now optionally accepts the current time, in ticks. This can
109 be used to avoid some calls to spdk_get_ticks() internally.
111 spdk_app_start() now only accepts a single context argument.
115 An new API `spdk_bdev_get_data_block_size` has been added to get size of data
116 block except for metadata.
118 spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used
121 A mechanism for acquiring and releasing data buffers from bdev modules, used
122 to perform zero copy operations, was added.
124 New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(),
125 spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been
126 added to get metadata and DIF settings.
128 Bdevs claimed by the `examine_config` callback will be now further examined in the
129 `examine_disk` callback.
131 spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel
136 Support for per-device shared receive queues in the RDMA transport has been added.
137 It is enabled by default for any device that supports it.
139 The size of a shared receive queue is defined by transport configuration file parameter
140 `MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`.
141 Default size is 4096.
143 Add model number as parameter to construct_nvmf_subsystem (-d option),
144 rather than using hardcoded define.
146 DIF passthrough feature has been added. DIF setting of the allocated bdevs is
147 exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is
148 passed through to the allocated bdevs.
152 The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated.
153 For retrieving physical addresses, spdk_vtophys() should be used instead.
155 spdk_realloc() has been added to reallocate DMA/shared memory.
157 spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know
158 that device has a pending external hotremove request.
160 spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources
161 allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected
162 that common usage of those functions is to call them just before terminating the process.
164 Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA.
168 Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now.
170 Updated DPDK submodule to DPDK 19.02.
174 New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application.
176 The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device
177 is specified, use that specified nbd device. If it's not specified, pick available one.
181 DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator.
182 DIF is attached into data for write I/O and stripped from data for read I/O.
186 Added experimental support for running with the external, upstream rte_vhost library.
187 This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag.
188 The minimum supported rte_vhost version (DPDK version) is 19.05-rc1.
190 As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and
191 fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks
192 if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting.
194 The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first
195 unoccupied target id will be used.
199 AIO bdev module can now reap I/O completions directly from userspace, significantly improving
200 the overall performance.
204 Synchronous IO operations no longer use spdk_io_channel, but instead use
205 spdk_fs_thread_ctx. The behavior is otherwise identical.
209 Added support for caching multiple bdevs using a single bdev as a cache.
213 Added the notify library that provides a high performance local event bus
214 between libraries. Example usage was added to bdev module, which reports
215 notifications for added and removed bdevs via RPC.
219 Added new API spdk_sock_readv() to the sock library for performing vectored
224 The function spdk_subsystem_init() no longer requires spdk_event as an argument.
226 Changed API of spdk_subsystem_config_json() to no longer be asynchronous.
230 A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel
231 API was added. This module requires liburing.
235 Options to easily compile with profile guided optimization have been added to
236 `configure`. To use profile guided optimization with SPDK, run
237 `./configure --with-pgo-capture`, build SPDK, then run a workload of your
238 choosing. Then, simply run `./configure --with-pgo-enable` and recompile to
239 build using the generated profile data. Profile guided optimization can yield
240 very large performance improvements, especially on GCC 8 and clang 7. This may
241 be combined with link time optimization which has been available under the
242 `--enable-lto` configure option for several releases.
244 ### compression bdev/reduce library
246 Added "reduce" block compression scheme based on using SSDs for storing
247 compressed blocks of storage and presistent memory for metadata. Please see
248 [compression](https://spdk.io/doc/bdev.html) for more details.
254 New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added.
255 This module allows for the use of one bdev to act as a high performance cache in front of another bdev.
256 Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details.
257 Only write through mode is currently supported and this feature is considered experimental.
261 For `spdk_app_parse_args`, add return value to the callback which parses application
262 specific command line parameters to protect SPDK applications from crashing by invalid
263 values from user input.
265 By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved
266 memory size can be still set with `-s` or `--mem-size` option, although the default value
267 was reduced down to 0.
269 A custom hugetlbfs directory can now be specified via spdk_app_opts.
270 This can be used to configure hugepages with different sizes, a different size limit,
271 or different access permissions than the system's default hugepage pool.
272 SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option.
276 spdk_vtophys() has been refactored to accept length of the translated region as a new
277 parameter. The function will now update that parameter with the largest possible value
278 for which the memory is contiguous in the physical memory address space.
280 The following functions were removed:
281 - spdk_pci_nvme_device_attach()
282 - spdk_pci_nvme_enumerate()
283 - spdk_pci_ioat_device_attach()
284 - spdk_pci_ioat_enumerate()
285 - spdk_pci_virtio_device_attach()
286 - spdk_pci_virtio_enumerate()
288 They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which
289 require a new spdk_pci_driver object to be provided. It can be one of the following:
290 - spdk_pci_nvme_get_driver()
291 - spdk_pci_ioat_get_driver()
292 - spdk_pci_virtio_get_driver()
294 spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual
295 spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for
296 each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach()
297 or spdk_pci_enumerate() like any other device.
299 A new spdk_pause() function was added to pause CPU execution for an implementation specific
300 amount of time. Quoting from DPDK function this is based on: "This call is intended for
301 tight loops which poll a shared resource or wait for an event. A short pause within the loop
302 may reduce the power consumption."
304 A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init
305 is added into it. If user is using DPDK, and already called rte_eal_init, then include
306 include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init.
308 ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture
309 to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L
310 by explicitly passing --without-isal to the configure script.
314 A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid.
316 An new parameter `init_crc` representing the initial CRC value was added to
317 `spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning
318 multiple separate buffers.
320 New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX
321 formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF
324 Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol`
327 Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string
328 with automatic buffer re-allocation.
332 Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are
333 introduced to support further security protocol development.
335 admin_timeout_ms was added to NVMe controller initialization options, users
336 can change the default value when probing a controller.
338 Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts,
339 it will be used to enable the digest support for the NVMe/TCP transport.
341 Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With
342 this new transport, it can be used to connect the NVMe-oF target with the
345 Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr
346 arg refers to a Discovery Controller or not.
348 Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id`
349 for parsing host address and host service ID arguments on a per connection basis.
351 The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to
352 use when connecting to the controller backing the NVMe bdev.
356 The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`.
357 Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen`
358 will also no longer implicitly initialize a transport with the default target options (since
359 there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create`
360 prior to calling `spdk_nvmf_tgt_listen`.
362 Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to
363 `set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit.
365 Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts,
366 and also updated the related rpc function nvmf_create_transport, to make this
367 configurable parameter available to users. The `num_shared_buffers` is used to
368 configure the shared buffer numbers of the transport used by RDMA or TCP transport.
369 `buf_cache_size` configures number of shared buffers to cache per poll group.
373 Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this tranport,
374 the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF
375 protocol via TCP/IP from the host.
377 Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory.
378 By default, the RDMA transport will use the ibverbs library to create protection domains and register memory.
379 Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration.
383 Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev.
384 Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details.
386 Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io.
387 Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module.
389 On shutdown, bdev unregister now proceeds in top-down fashion, with
390 claimed bdevs skipped (these will be unregistered later, when virtual
391 bdev built on top of the respective base bdev unclaims it). This
392 allows virtual bdevs to be shut down cleanly as opposed to the
393 previous behavior that didn't differentiate between hotremove and
396 The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC
397 for the bdev type to be removed (i.e. delete_malloc_bdev).
399 Added support for separate bandwidth rate limits for read and write to QoS in bdev layer.
401 Bdev I/O statistics now track unmap opertations.
405 Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC.
406 This allows for basing clones on top of lvol_bdev without first creating a snapshot.
408 Added option to change method for data erasure when deleting lvol or resizing down.
409 Default of unmapping clusters can now be changed to writing zeroes or no operation.
411 Added option to change method for erasing data region on lvol store creation.
412 Default of unmapping can now be changed to writing zeroes or no operation.
416 "trace flags" are now referred to as "log flags" in the SPDK log API. The
417 set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated,
418 and set_log_flag, clear_log_flag and get_log_flags RPCs have been added.
422 New `trace_record` application was added. It can be used to poll spdk trace shm file and
423 append any new trace entries into another specified file. This can help retain those entries
424 that would otherwise be overwritten in the shm file. See
425 [Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events)
428 Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app
429 with argument "--num-trace-entries <NUM>" provided.
431 New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and
432 each tpoint group status.
433 New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or
434 disable a specific tpoint group.
438 EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as
439 block devices. The module is split into the library (located in lib/ftl) and bdev_ftl
440 (lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details.
444 A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious
445 vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which
446 would result in a partial denial of service in the SPDK vhost target. These types of descriptor
447 chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted
448 vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny
451 Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file.
452 Each connection (internally called a vhost session) will have access to the same storage, but
453 will use different virtqueues, different features and possibly different memory.
457 SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated.
458 Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes.
462 DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new
463 multi-process device hotplug yet and continues to work the same way it always did.
465 Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now.
469 The following RPC commands deprecated in the previous release are now removed:
470 - construct_virtio_user_scsi_bdev
471 - construct_virtio_pci_scsi_bdev
472 - construct_virtio_user_blk_bdev
473 - construct_virtio_pci_blk_bdev
474 - remove_virtio_scsi_bdev
475 - construct_nvmf_subsystem
479 The configure options `--with-raid` and `--without-raid` that were deprecated in the previous
480 release are now removed.
484 Starting nbd using `spdk_nbd_start` is now performed asynchronously.
488 Net framework initialization and finish is now done asynchronously.
492 Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state.
493 Added `spdk_rpc_get_state` to check current state of RPC server.
494 RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized.
498 JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll.
499 JSON RPC server can now recieve a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb`
500 and `spdk_jsonrpc_conn_del_close_cb`.
506 spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive()
507 were added to support sending or receiving security protocol data to or from
510 spdk_nvme_ns_get_extended_sector_size() was added. This function includes
511 the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still
512 returns only the data size per sector, not including metadata.
514 New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller.
515 See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation
520 New `configure` options, `--with-shared` and `--without-shared`
521 [default], provide the capability to build, or not, SPDK shared libraries.
522 This includes the single SPDK shared lib encompassing all of the SPDK
523 static libs as well as individual SPDK shared libs corresponding to
524 each of the SPDK static ones. Although the production of the shared
525 libs conforms with conventional version naming practices, such naming
526 does not at this time confer any SPDK ABI compatibility claims.
530 spdk_bdev_alias_del_all() was added to delete all alias from block device.
532 A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev
533 Framework. The module initially uses a software AESNI CBC cipher with experimental support for the
534 Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work
535 may include additional ciphers as well as consideration for authentication.
537 The RAID virtual bdev module is now always enabled by default. The configure --with-raid and
538 --without-raid options are now ignored and deprecated and will be removed in the next release.
540 Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer.
541 See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit)
542 documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at
543 18.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both
544 bandwidth and IOPS limits.
546 spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed.
548 ### Environment Abstraction Layer and Event Framework
550 The size parameter of spdk_mem_map_translate is now a pointer. This allows the
551 function to report back the actual size of the translation relative to the original
552 request made by the user.
554 A new structure spdk_mem_map_ops has been introduced to hold memory map related
555 callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc
556 in lieu of the notify callback.
560 The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically
561 utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1.
563 Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup,
564 but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try
565 to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using
566 only as many hugepages as it is really needed.
568 Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed
569 to be physically contiguous.
573 I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors.
577 Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC
578 method for CHAP authentication in discovery sessions have been changed to
579 align with `construct_target_node` RPC method. Old names are still usable
580 but will be removed in future release.
582 `set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have
583 been added to set CHAP authentication for discovery sessions and existing
584 target nodes, respectively.
586 The SPDK iSCSI target supports an AuthFile which can be used to load CHAP
587 shared secrets when the iSCSI target starts. SPDK previously provided a
588 default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was
589 specified. This default has been removed. Users must now explicitly specify
590 the location of this file to load CHAP shared secrets from a file, or use
591 the related iSCSI RPC methods to add them at runtime.
595 The SPDK iSCSI initiator is no longer considered experimental and becomes
596 a first-class citizen among bdev modules. The basic usage has been briefly
597 described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi)
601 The SPDK application framework can now parse long name command line parameters.
602 Most single-character parameters have a long name equivalent now. See the
603 [Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation
604 for details or use the `--help` command line parameter to list all available
607 bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already
608 used by existing apps for memory size.
610 bdevio can now accept all SPDK command line parameters. The config now has to
611 be provided with `-c` or `--config` param.
613 The following ioat/perf and nvme/perf parameters were renamed as well:
614 `-s` (io size) to `-o`
615 `-d` (mem size) to `-s`
617 The ReactorMask config file parameter has been deprecated. Users should
618 use the -m or --cpumask command line option to specify the CPU core mask
621 Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt
622 and vhost. Config file pathnames may now only be specified using the
623 -c command line option.
625 Users may no longer set DPDK_DIR in their environment to specify the
626 location of the DPDK installation used to build SPDK. Using DPDK_DIR
627 has not been the documented nor recommended way to specify the DPDK
628 location for several releases, but removing it ensures no unexpected
629 surprises for users who may have DPDK_DIR defined for other reasons.
630 Users should just use the "configure" script to specify the DPDK
631 location before building SPDK.
633 Although we know that many developers still use Python 2 we are officially
634 switching to Python3 with requirement that all new code must be valid also
635 for Python 2 up to the EOL which is year 2020.
637 Invoking interpreter explicitly is forbidden for executable scripts. There
638 is no need to use syntax like "python ./scripts/rpc.py". All executable
639 scripts must contain proper shebang pointing to the right interpreter.
640 Scripts without shebang musn't be executable.
642 A Python script has been added to enable conversion of old INI config file
643 to new JSON-RPC config file format. This script can be found at
644 scripts/config_converter.py. Example how this script can be used:
646 cat old_format.ini | scripts/config_converter.py > new_json_format.json
651 Two additional parameters were added to spdk_sock_get_addr() for the server
652 port and client port. These parameters are named "sport" and "cport"
657 The following RPC commands have been deprecated:
658 - construct_virtio_user_scsi_bdev
659 - construct_virtio_pci_scsi_bdev
660 - construct_virtio_user_blk_bdev
661 - construct_virtio_pci_blk_bdev
662 - remove_virtio_scsi_bdev
664 The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev`
665 command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev`
666 was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI
671 spdk_file_get_id() returning unique ID for the file was added.
675 Added jsonrpc-client C library intended for issuing RPC commands from applications.
677 Added API enabling iteration over JSON object:
679 - spdk_json_find_string()
680 - spdk_json_find_array()
681 - spdk_json_object_first()
682 - spdk_json_array_first()
687 Blobstore I/O operations are now based on io_units, instead of blobstore page size.
688 The io_unit size is now the same as the underlying block device's block size.
689 Logical volumes built on a block device with 512B block size can now be used as boot devices
694 The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental.
695 Support for the iSCSI and NVMe-oF targets has been added.
701 A new public header file bdev_module.h has been introduced to facilitate the
702 development of new bdev modules. This header includes an interface for the
703 spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of
704 multiple virtual bdevs on top of a single base bdev and should act as the
705 primary API for module authors.
707 spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide
710 A mechanism for handling out of memory condition errors (ENOMEM) returned from
711 I/O submission requests at the bdev layer has been added. See
712 spdk_bdev_queue_io_wait().
714 The spdk_bdev_get_io_stat() function now returns cumulative totals instead of
715 resetting on each call. This allows multiple callers to query I/O statistics
716 without conflicting with each other. Existing users will need to adjust their
717 code to record the previous I/O statistics to calculate the delta between calls.
719 I/O queue depth tracking and samples options have been added. See
720 spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and
721 spdk_bdev_set_qd_sampling_period().
724 A new bdev module called "raid" has been added as experimental module which
725 aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note
726 that vhost will not work with this module because it does not yet have support
727 for multi-element io vectors.
731 The debug log component flag available on several SPDK applications has been
732 renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the
733 option to be added to tools that already use `-t` to mean something else.
737 A new function, spdk_bs_dump(), has been added that dumps all of the contents of
738 a blobstore to a file pointer. This includes the metadata and is very useful for
741 Two new operations have been added for thin-provisioned blobs.
742 spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions
743 of the blob and populate them with the correct data by reading from the backing
744 blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate
745 clusters that correspond to data in the blob's immediate parent. Clusters
746 allocated to grandparents or that aren't allocated at all will remain
751 Changed the return type of spdk_file_truncate() from void to int to allow the
752 propagation of `ENOMEM` errors.
756 The new API functions spdk_nvme_qpair_add_cmd_error_injection() and
757 spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error
758 emulation. Users can set a specified command to fail with a particular error
761 Changed the name `timeout_sec` parameter to `timeout_us` in
762 spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from
763 uint32_t to uint64_t. This will give users more fine-grained control over the
766 Basic support for Open Channel SSDs was added. See nvme_ocssd.h
768 ### NVMe Over Fabrics
770 The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback
773 spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs.
775 spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed
776 number of namespaces for a given subsystem.
780 The build system now generates a combined shared library (libspdk.so) that may
781 be used in place of the individual static libraries (libspdk_*.a). The combined
782 library includes all components of SPDK and is intended to make linking against
783 SPDK easier. The static libraries are also still provided for users that prefer
784 to link only the minimal set of components required.
786 ### git pre-commit and pre-push hooks
788 The pre-commit hook will run `scripts/check_format.sh` and verify there are no
789 formating errors before allowing `git commit` to run. The pre-push hook runs
790 `make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and
791 clang compiler before allowing `git push` to run. Following each DEBUG build
792 `test/unit/unittest.sh` is run and verified. Results are recorded in the
795 To enable type: 'git config core.hooksPath .githooks'. To override after
796 configuration use the `git --no-verify` flag.
800 The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node
801 rather than always returning `true`.
805 The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports
806 dynamic memory allocation, but due to some issues found after the DPDK 18.05 release,
807 that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use
808 the legacy memory allocation model. The plan is to enable dynamic memory allocation
809 after the DPDK 18.08 release which should fix these issues.
811 ### Environment Abstraction Layer and Event Framework
813 The spdk_mem_map_translate() function now takes a size parameter to indicate the size of
814 the memory region. This can be used by environment implementations to validate the
815 requested translation.
817 The I/O Channel implementation has been moved to its own library - lib/thread. The
818 public API that was previously in spdk/io_channel.h is now in spdk/thread.h The
819 file spdk/io_channel.h remains and includes spdk/thread.h.
821 spdk_reactor_get_tsc_stats was added to return interesting statistics for each
826 IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable
827 option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is
828 now deprecated and will be removed in a future release.
830 ## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP
834 The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the
835 DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059).
836 Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059)
837 for additional information on the DPDK vulnerability.
839 Workarounds have been added to ensure vhost compatibility with QEMU 2.12.
841 EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the
842 [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details.
844 ### Unified Target Application
846 A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the
847 functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target,
848 and vhost target. The new application can be managed through the existing configuration file and
849 [JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods.
853 spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk().
855 New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added.
856 These new functions have a `flags` parameter that allows the user to specify whether the allocated
857 memory needs to be suitable for DMA and whether it should be shared across processes with the same
858 shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will
859 eventually be deprecated and removed.
863 A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules
864 when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require
865 notification that the set of initialization examine() calls is complete.
867 The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with
868 the spdk_bdev_get_uuid() function.
870 Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the
871 [set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation
876 The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed.
877 Users should switch to the `-r` command-line parameter instead.
879 The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as
880 needed; previously, the response was limited to 32 kilobytes.
884 EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available.
885 See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details.
889 EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is
890 experimental pending a functional allocator to free and reallocate CMB buffers.
892 spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available.
894 New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns()
895 have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if
896 a namespace ID is active.
900 Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters
901 to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes
902 the backing bdev's UUID as the namespace UUID when available.
904 spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion.
908 A number of functions have been renamed:
910 - spdk_bs_io_write_blob() => spdk_blob_io_write()
911 - spdk_bs_io_read_blob() => spdk_blob_io_read()
912 - spdk_bs_io_writev_blob() => spdk_blob_io_writev()
913 - spdk_bs_io_readv_blob() => spdk_blob_io_readv()
914 - spdk_bs_io_unmap_blob() => spdk_blob_io_unmap()
915 - spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes()
917 The old names still exist but are deprecated. They will be removed in the v18.07 release.
919 spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O
920 are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still
921 required to sync the updated metadata to disk.
925 A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added.
927 Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination.
929 New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols
930 and Clones of existing Snapshots.
931 See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation
934 Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method.
938 A set of changes were made in the SPDK's lib code altering
939 instances of calls to `exit()` and `abort()` to return a failure instead
940 wherever reasonably possible.
942 spdk_app_start() no longer exit()'s on an internal failure, but
943 instead returns a non-zero error status.
945 spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid
946 option, but instead returns SPDK_APP_PARSE_ARGS_HELP and
947 SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS
950 spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07.
954 The prototype for spdk_poller_fn() has been modified; it now returns a value indicating
955 whether or not the poller did any work. Existing pollers will need to be updated to
960 The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace
961 TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more
966 An iSCSI initiator bdev module has been added to SPDK. This module should be considered
967 experimental pending additional features and tests. More details can be found in
968 lib/bdev/iscsi/README.
972 The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of
973 `--with-nvml`. This reflects the renaming of the persistent memory library from NVML to
976 ### Virtio Block driver
978 A userspace driver for Virtio Block devices has been added. It was built on top of the
979 [Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to
980 the Virtio SCSI driver. See the
981 [Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for
984 ### Virtio with 2MB hugepages
986 The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option
987 enables SPDK Virtio to work with 2MB hugepages.
988 See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details.
990 ## v18.01: Blobstore Thin Provisioning
994 The build system now includes a `make install` rule, including support for the common
995 `DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix
996 may be set via the configure `--prefix` option. Example: `make install prefix=/usr`.
1000 A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/run/spdk.sock.
1001 A -r option command line option has been added to enable an alternative UNIX domain socket location,
1002 or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file
1003 section is now deprecated and will be removed in the v18.04 release.
1007 spdk_poller_register() and spdk_poller_unregister() were moved from the event
1008 framework (include/spdk/event.h) to the I/O channel library
1009 (include/spdk/io_channel.h). This allows code that doesn't depend on the event
1010 framework to request registration and unregistration of pollers.
1012 spdk_for_each_channel() now allows asynchronous operations during iteration.
1013 Instead of immediately continuing the interation upon returning from the iteration
1014 callback, the user must call spdk_for_each_channel_continue() to resume iteration.
1016 ### Block Device Abstraction Layer (bdev)
1018 The poller abstraction was removed from the bdev layer. There is now a general purpose
1019 abstraction for pollers available in include/spdk/io_channel.h
1023 A set of changes were made in the SPDK's lib code altering,
1024 instances of calls to `exit()` and `abort()` to return a failure instead
1025 wherever reasonably possible. This has resulted in return type changes of
1028 - spdk_env_init() from type `void` to `int`.
1029 - spdk_mem_map_init() from type `void` to `int`.
1031 Applications making use of these APIs should be modified to check for
1032 a non-zero return value instead of relying on them to fail without return.
1036 SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind:
1037 Only physical removal events are supported; removing devices via the sysfs `remove` file will not work.
1041 Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available
1042 cores round-robin. The "Core" option in the configuration file has been removed.
1046 A number of functions have been renamed:
1048 - spdk_bs_md_resize_blob() => spdk_blob_resize()
1049 - spdk_bs_md_sync_blob() => spdk_blob_sync_md()
1050 - spdk_bs_md_close_blob() => spdk_blob_close()
1051 - spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names()
1052 - spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value()
1053 - spdk_blob_md_set_xattr() => spdk_blob_set_xattr()
1054 - spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr()
1055 - spdk_bs_md_create_blob() => spdk_bs_create_blob()
1056 - spdk_bs_md_open_blob() => spdk_bs_open_blob()
1057 - spdk_bs_md_delete_blob() => spdk_bs_delete_blob()
1058 - spdk_bs_md_iter_first() => spdk_bs_iter_first()
1059 - spdk_bs_md_iter_next() => spdk_bs_iter_next()
1061 The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument
1062 rather than struct spdk_blob **.
1064 The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument
1065 rather than struct spdk_blob **.
1067 Thin provisioning support has been added to the blobstore. It can be enabled by setting the
1068 `thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext().
1072 The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be
1073 achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method.
1074 See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details.
1078 SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped,
1079 although it still remains to work for now. The new FIO contains huge amount of bugfixes and
1080 it's recommended to do an update.
1084 Previously a part of the bdev_virtio module, now a separate library. Virtio is now available
1085 via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing
1086 new Virtio backends, namely Virtio-BLK.
1090 The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated
1091 to an epoll/kqueue descriptor on the master core when idle.
1093 ## v17.10: Logical Volumes
1095 ### New dependencies
1097 libuuid was added as new dependency for logical volumes.
1099 libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08.
1101 ### Block Device Abstraction Layer (bdev)
1103 An [fio](http://github.com/axboe/fio) plugin was added that can route
1104 I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/)
1105 for more information.
1107 spdk_bdev_unmap() was modified to take an offset and a length in bytes as
1108 arguments instead of requiring the user to provide an array of SCSI
1109 unmap descriptors. This limits unmaps to a single contiguous range.
1111 spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out.
1112 If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using
1115 New API functions that accept I/O parameters in units of blocks instead of bytes
1117 - spdk_bdev_read_blocks(), spdk_bdev_readv_blocks()
1118 - spdk_bdev_write_blocks(), spdk_bdev_writev_blocks()
1119 - spdk_bdev_write_zeroes_blocks()
1120 - spdk_bdev_unmap_blocks()
1122 The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be
1127 The AIO bdev now allows the user to override the auto-detected block size.
1131 The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field.
1132 NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take
1133 into account when splitting I/O requests.
1135 The HotplugEnable option in `[Nvme]` sections of the configuration file is now
1136 "No" by default. It was previously "Yes".
1138 The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the
1139 NVMe Controller associated with a given namespace.
1141 The NVMe library now allows the user to specify a host identifier when attaching
1142 to a controller. The host identifier is used as part of the Reservations feature,
1143 as well as in the NVMe-oF Connect command. The default host ID is also now a
1144 randomly-generated UUID, and the default host NQN uses the host ID to generate
1147 spdk_nvme_connect() was added to allow the user to connect directly to a single
1148 NVMe or NVMe-oF controller.
1150 ### NVMe-oF Target (nvmf_tgt)
1152 The NVMe-oF target no longer requires any in-capsule data buffers to run, and
1153 the feature is now entirely optional. Previously, at least 4 KiB in-capsule
1154 data buffers were required.
1156 NVMe-oF subsytems have a new configuration option, AllowAnyHost, to control
1157 whether the host NQN whitelist is enforced when accepting new connections.
1158 If no Host options have been specified and AllowAnyHost is disabled, the
1159 connection will be denied; this is a behavior change from previous releases,
1160 which allowed any host NQN to connect if the Host list was empty.
1161 AllowAnyHost is disabled by default.
1163 NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number
1164 of namespaces per subsystem is no longer limited.
1166 The NVMe-oF target now supports the Write Zeroes command.
1168 ### Environment Abstraction Layer
1170 A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide
1171 additional clarity when constructing spdk_mempools. Previously, -1 could be
1172 passed and the library would choose a reasonable default, but this new value
1173 makes it explicit that the default is being used.
1177 The blobstore super block now contains a bstype field to identify the type of the blobstore.
1178 Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load().
1180 spdk_bs_destroy() was added to allow destroying blobstore on device
1181 with an initialized blobstore.
1183 spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable
1186 A CLI tool for blobstore has been added, allowing basic operations through either command
1187 line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli)
1188 documentation for more details.
1192 The ability to set a thread name, previously only used by the reactor code, is
1193 now part of the spdk_thread_allocate() API. Users may specify a thread name
1194 which will show up in tools like `gdb`.
1198 The spdk_trace_dump() function now takes a new parameter to allow the caller to
1199 specify an output file handle (stdout or stderr, for example).
1203 Logical volumes library built on top of SPDK blobstore has been added.
1204 It is possible to create logical volumes on top of other devices using RPC.
1206 See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information.
1208 ### Persistent Memory
1210 A new persistent memory bdev type has been added.
1211 The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/).
1212 It is possible to create pmem devices on top of pmem pool files using RPC.
1214 See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information.
1216 ### Virtio SCSI driver
1218 A userspace driver for Virtio SCSI devices has been added.
1219 The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application.
1221 See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and [Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information.
1225 The vhost target application now supports live migration between QEMU instances.
1228 ## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev
1232 A `configure` script has been added to simplify the build configuration process.
1233 The existing CONFIG file and `make CONFIG_...` options are also still supported.
1234 Run `./configure --help` for information about available configuration options.
1236 A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk`
1237 option is specified to configure, the SPDK build system will automatically build a
1238 known-good configuration of DPDK with the minimal options enabled. See the Building
1239 section of README.md for more information.
1241 A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to
1242 develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant
1243 section of README.md for more information.
1245 ### Userspace vhost-blk target
1247 The vhost library and example app have been updated to support the vhost-blk
1248 protocol in addition to the existing vhost-scsi protocol.
1249 See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details.
1251 ### Block device abstraction layer (bdev)
1253 A GPT virtual block device has been added, which automatically exposes GPT partitions
1254 with a special SPDK-specific partition type as bdevs.
1255 See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for
1260 The NVMe driver has been updated to support recent Intel SSDs, including the Intel®
1261 Optaneâ„¢ SSD DC P4800X series.
1263 A workaround has been added for devices that failed to recognize register writes
1264 during controller reset.
1266 The NVMe driver now allocates request tracking objects on a per-queue basis. The
1267 number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by
1268 modifying `io_queue_requests` in the opts structure.
1270 The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`).
1272 spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override
1273 controller-level options for each individual I/O queue pair.
1274 Existing callers with qprio == 0 can be updated to:
1276 ... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);
1278 Callers that need to specify a non-default qprio should be updated to:
1280 struct spdk_nvme_io_qpair_opts opts;
1281 spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts));
1282 opts.qprio = SPDK_NVME_QPRIO_...;
1283 ... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts));
1286 ### Environment Abstraction Layer
1288 The environment abstraction layer has been updated to include several new functions
1289 in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the
1290 current set of functions.
1292 ### SPDK Performance Analysis with Intel® VTune™ Amplifier
1294 Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018.
1296 This analysis provides:
1297 - I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.)
1298 - Tuning insights on the interplay of I/O and compute devices by estimating how many cores
1299 would be reasonable to provide for SPDK to keep up with a current storage workload.
1301 See the VTune Amplifier documentation for more information.
1304 ## v17.03: Blobstore and userspace vhost-scsi target
1306 ### Blobstore and BlobFS
1308 The blobstore is a persistent, power-fail safe block allocator designed to be
1309 used as the local storage system backing a higher-level storage service.
1310 See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details.
1312 BlobFS adds basic filesystem functionality like filenames on top of the blobstore.
1313 This release also includes a RocksDB Env implementation using BlobFS in place of the
1315 See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details.
1317 ### Userspace vhost-scsi target
1319 A userspace implementation of the QEMU vhost-scsi protocol has been added.
1320 The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices.
1321 See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details.
1325 The overhead of the main reactor event loop was reduced by optimizing the number of
1326 calls to spdk_get_ticks() per iteration.
1330 The NVMe library will now automatically split readv/writev requests with scatter-gather
1331 lists that do not map to valid PRP lists when the NVMe controller does not natively
1334 The `identify` and `perf` NVMe examples were modified to add a consistent format for
1335 specifying remote NVMe over Fabrics devices via the `-r` option.
1336 This is implemented using the new `spdk_nvme_transport_id_parse()` function.
1340 The [Nvme] section of the configuration file was modified to remove the `BDF` directive
1341 and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF)
1342 devices can now be specified as the backing block device. A script to generate an
1343 entire [Nvme] section based on the local NVMe devices attached was added at
1344 `scripts/gen_nvme.sh`.
1348 The [Nvme] section of the configuration file was modified to remove the `BDF` directive
1349 and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF)
1350 devices can now be specified as the backing block device. A script to generate an
1351 entire [Nvme] section based on the local NVMe devices attached was added at
1352 `scripts/gen_nvme.sh`.
1354 ## v16.12: NVMe over Fabrics host, hotplug, and multi-process
1358 The NVMe library has been changed to create its own request memory pool rather than
1359 requiring the user to initialize the global `request_mempool` variable. Apps can be
1360 updated by simply removing the initialization of `request_mempool`. Since the NVMe
1361 library user no longer needs to know the size of the internal NVMe request
1362 structure to create the pool, the `spdk_nvme_request_size()` function was also removed.
1364 The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become
1365 `spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe
1366 Dataset Management command's parameters. Existing callers can be updated to use
1367 `spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the
1370 The NVMe library SGL callback prototype has been changed to return virtual addresses
1371 rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and
1372 `spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match.
1374 The NVMe library now supports NVMe over Fabrics devices in addition to the existing
1375 support for local PCIe-attached NVMe devices. For an example of how to enable
1376 NVMe over Fabrics support in an application, see `examples/nvme/identify` and
1377 `examples/nvme/perf`.
1379 Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug
1380 support, an application should call the `spdk_nvme_probe()` function on a regular
1381 basis to probe for new devices (reported via the existing `probe_cb` callback) and
1382 removed devices (reported via a new `remove_cb` callback). Hotplug is currently
1383 only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe
1384 devices must be bound to `uio_pci_generic` by an external script or tool.
1386 Multiple processes may now coordinate and use a single NVMe device simultaneously
1387 using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html).
1389 ### NVMe over Fabrics target (`nvmf_tgt`)
1391 The `nvmf_tgt` configuration file format has been updated significantly to enable
1392 new features. See the example configuration file `etc/spdk/nvmf.conf.in` for
1393 more details on the new and changed options.
1395 The NVMe over Fabrics target now supports virtual mode subsystems, which allow the
1396 user to export devices from the SPDK block device abstraction layer as NVMe over
1397 Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported,
1398 and a single `nvmf_tgt` may export both types of subsystems simultaneously.
1400 ### Block device abstraction layer (bdev)
1402 The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe
1403 blockdev driver has been updated to support scatter/gather. Apps can use the
1404 new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()`
1407 The bdev status returned from each I/O has been extended to pass through NVMe
1408 or SCSI status codes directly in cases where the underlying device can provide
1409 a more specific status code.
1411 A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the
1412 `iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or
1417 `libpciaccess` has been removed as a dependency and DPDK PCI enumeration is
1418 used instead. Prior to DPDK 16.07 enumeration by class code was not supported,
1419 so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered
1420 by the NVMe library.
1422 The `env` environment abstraction library has been introduced, and a default
1423 DPDK-based implementation is provided as part of SPDK. The goal of the `env`
1424 layer is to enable use of alternate user-mode memory allocation and PCI access
1425 libraries. See `doc/porting.md` for more details.
1427 The build process has been modified to produce all of the library files in the
1428 `build/lib` directory. This is intended to simplify the use of SPDK from external
1429 projects, which can now link to SPDK libraries by adding the `build/lib` directory
1430 to the library path via `-L` and linking the SPDK libraries by name (for example,
1431 `-lspdk_nvme -lspdk_log -lspdk_util`).
1433 `nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user
1434 to query and modify the configuration at runtime. The RPC service is disabled by
1435 default, since it currently does not provide any authentication or security
1436 mechanisms; it should only be enabled on systems with controlled user access
1437 behind a firewall. An example RPC client implemented in Python is provided in
1440 ## v16.08: iSCSI target, NVMe over Fabrics maturity
1442 This release adds a userspace iSCSI target. The iSCSI target is capable of exporting
1443 NVMe devices over a network using the iSCSI protocol. The application is located
1444 in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in.
1446 This release also significantly improves the existing NVMe over Fabrics target.
1447 - The configuration file format was changed, which will require updates to
1448 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`):
1449 - `SubsystemGroup` was renamed to `Subsystem`.
1450 - `AuthFile` was removed (it was unimplemented).
1451 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names)
1452 when naming subsystems. The default node name was changed to reflect this;
1453 it is now "nqn.2016-06.io.spdk".
1454 - `Port` and `Host` sections were merged into the `Subsystem` section
1455 - Global options to control max queue depth, number of queues, max I/O
1456 size, and max in-capsule data size were added.
1457 - The Nvme section was removed. Now a list of devices is specified by
1458 bus/device/function directly in the Subsystem section.
1459 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt
1460 to future-proof the interface, so the only mode supported by this release
1462 - Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library.
1463 - The target now supports discovery.
1465 This release also adds one new feature and provides some better examples and tools
1466 for the NVMe driver.
1467 - The Weighted Round Robin arbitration method is now supported. This allows
1468 the user to specify different priorities on a per-I/O-queue basis. To
1469 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`.
1470 - A simplified "Hello World" example was added to show the proper way to use
1471 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`.
1472 - A test for measuring software overhead was added. See `test/lib/nvme/overhead`.
1474 ## v16.06: NVMf userspace target
1476 This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the
1477 newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe
1478 devices from a host machine over the network via RDMA. Currently, the target is
1479 limited to directly exporting physical NVMe devices, and the discovery subsystem
1482 This release includes a general API cleanup, including renaming all declarations
1483 in public headers to include a `spdk` prefix to prevent namespace clashes with
1487 - The `nvme_attach()` API was reworked into a new probe/attach model, which
1488 moves device detection into the NVMe library. The new API also allows
1489 parallel initialization of NVMe controllers, providing a major reduction in
1490 startup time when using multiple controllers.
1491 - I/O queue allocation was changed to be explicit in the API. Each function
1492 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`)
1493 argument, and I/O queues may be allocated using
1494 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of
1495 queue pairs than the previous model, which only allowed a single queue
1496 per thread and limited the total number of I/O queues to the lowest number
1497 supported on any attached controller.
1498 - Added support for the Write Zeroes command.
1499 - `examples/nvme/perf` can now report I/O command latency from the
1500 the controller's viewpoint using the Intel vendor-specific read/write latency
1502 - Added namespace reservation command support, which can be used to coordinate
1503 sharing of a namespace between multiple hosts.
1504 - Added hardware SGL support, which enables use of scattered buffers that
1505 don't conform to the PRP list alignment and length requirements on supported
1507 - Added end-to-end data protection support, including the ability to write and
1508 read metadata in extended LBA (metadata appended to each block of data in the
1509 buffer) and separate metadata buffer modes.
1510 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()`
1513 - The DMA block fill feature is now exposed via the `ioat_submit_fill()`
1514 function. This is functionally similar to `memset()`, except the memory is
1515 filled with an 8-byte repeating pattern instead of a single byte like memset.
1517 - Added support for using DPDK for PCI device mapping in addition to the
1518 existing libpciaccess option. Using the DPDK PCI support also allows use of
1519 the Linux VFIO driver model, which means that SPDK userspace drivers will work
1520 with the IOMMU enabled. Additionally, SPDK applications may be run as an
1521 unprivileged user with access restricted to a specific set of PCIe devices.
1522 - The PCI library API was made more generic to abstract away differences
1523 between the underlying PCI access implementations.
1525 ## v1.2.0: IOAT user-space driver
1527 This release adds a user-space driver with support for the Intel I/O Acceleration Technology (I/OAT, also known as "Crystal Beach") DMA offload engine.
1530 - New user-space driver supporting DMA memory copy offload
1531 - Example programs `ioat/perf` and `ioat/verify`
1532 - Kernel-mode DMA engine test driver `kperf` for performance comparison
1534 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry
1535 - Public API for retrieving log pages
1536 - Reservation register/acquire/release/report command support
1537 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks
1538 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features
1539 - Updated to support DPDK 2.2.0
1542 ## v1.0.0: NVMe user-space driver
1544 This is the initial open source release of the Storage Performance Development Kit (SPDK).
1547 - NVMe user-space driver
1548 - NVMe example programs
1549 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver
1550 - `examples/nvme/identify` displays NVMe controller information in a human-readable format
1551 - Linux and FreeBSD support