docs/nvdimm.txt

   1 QEMU Virtual NVDIMM
   2 ===================
   3
   4 This document explains the usage of virtual NVDIMM (vNVDIMM) feature
   5 which is available since QEMU v2.6.0.
   6
   7 The current QEMU only implements the persistent memory mode of vNVDIMM
   8 device and not the block window mode.
   9
  10 Basic Usage
  11 -----------
  12
  13 The storage of a vNVDIMM device in QEMU is provided by the memory
  14 backend (i.e. memory-backend-file and memory-backend-ram). A simple
  15 way to create a vNVDIMM device at startup time is done via the
  16 following command line options:
  17
  18  -machine pc,nvdimm=on
  19  -m $RAM_SIZE,slots=$N,maxmem=$MAX_SIZE
  20  -object memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE,readonly=off
  21  -device nvdimm,id=nvdimm1,memdev=mem1,unarmed=off
  22
  23 Where,
  24
  25  - the "nvdimm" machine option enables vNVDIMM feature.
  26
  27  - "slots=$N" should be equal to or larger than the total amount of
  28    normal RAM devices and vNVDIMM devices, e.g. $N should be >= 2 here.
  29
  30  - "maxmem=$MAX_SIZE" should be equal to or larger than the total size
  31    of normal RAM devices and vNVDIMM devices, e.g. $MAX_SIZE should be
  32    >= $RAM_SIZE + $NVDIMM_SIZE here.
  33
  34  - "object memory-backend-file,id=mem1,share=on,mem-path=$PATH,
  35    size=$NVDIMM_SIZE,readonly=off" creates a backend storage of size
  36    $NVDIMM_SIZE on a file $PATH. All accesses to the virtual NVDIMM device go
  37    to the file $PATH.
  38
  39    "share=on/off" controls the visibility of guest writes. If
  40    "share=on", then guest writes will be applied to the backend
  41    file. If another guest uses the same backend file with option
  42    "share=on", then above writes will be visible to it as well. If
  43    "share=off", then guest writes won't be applied to the backend
  44    file and thus will be invisible to other guests.
  45
  46    "readonly=on/off" controls whether the file $PATH is opened read-only or
  47    read/write (default).
  48
  49  - "device nvdimm,id=nvdimm1,memdev=mem1,unarmed=off" creates a read/write
  50    virtual NVDIMM device whose storage is provided by above memory backend
  51    device.
  52
  53    "unarmed" controls the ACPI NFIT NVDIMM Region Mapping Structure "NVDIMM
  54    State Flags" Bit 3 indicating that the device is "unarmed" and cannot accept
  55    persistent writes. Linux guest drivers set the device to read-only when this
  56    bit is present. Set unarmed to on when the memdev has readonly=on.
  57
  58 Multiple vNVDIMM devices can be created if multiple pairs of "-object"
  59 and "-device" are provided.
  60
  61 For above command line options, if the guest OS has the proper NVDIMM
  62 driver (e.g. "CONFIG_ACPI_NFIT=y" under Linux), it should be able to
  63 detect a NVDIMM device which is in the persistent memory mode and whose
  64 size is $NVDIMM_SIZE.
  65
  66 Note:
  67
  68 1. Prior to QEMU v2.8.0, if memory-backend-file is used and the actual
  69    backend file size is not equal to the size given by "size" option,
  70    QEMU will truncate the backend file by ftruncate(2), which will
  71    corrupt the existing data in the backend file, especially for the
  72    shrink case.
  73
  74    QEMU v2.8.0 and later check the backend file size and the "size"
  75    option. If they do not match, QEMU will report errors and abort in
  76    order to avoid the data corruption.
  77
  78 2. QEMU v2.6.0 only puts a basic alignment requirement on the "size"
  79    option of memory-backend-file, e.g. 4KB alignment on x86.  However,
  80    QEMU v.2.7.0 puts an additional alignment requirement, which may
  81    require a larger value than the basic one, e.g. 2MB on x86. This
  82    change breaks the usage of memory-backend-file that only satisfies
  83    the basic alignment.
  84
  85    QEMU v2.8.0 and later remove the additional alignment on non-s390x
  86    architectures, so the broken memory-backend-file can work again.
  87
  88 Label
  89 -----
  90
  91 QEMU v2.7.0 and later implement the label support for vNVDIMM devices.
  92 To enable label on vNVDIMM devices, users can simply add
  93 "label-size=$SZ" option to "-device nvdimm", e.g.
  94
  95  -device nvdimm,id=nvdimm1,memdev=mem1,label-size=128K
  96
  97 Note:
  98
  99 1. The minimal label size is 128KB.
 100
 101 2. QEMU v2.7.0 and later store labels at the end of backend storage.
 102    If a memory backend file, which was previously used as the backend
 103    of a vNVDIMM device without labels, is now used for a vNVDIMM
 104    device with label, the data in the label area at the end of file
 105    will be inaccessible to the guest. If any useful data (e.g. the
 106    meta-data of the file system) was stored there, the latter usage
 107    may result guest data corruption (e.g. breakage of guest file
 108    system).
 109
 110 Hotplug
 111 -------
 112
 113 QEMU v2.8.0 and later implement the hotplug support for vNVDIMM
 114 devices. Similarly to the RAM hotplug, the vNVDIMM hotplug is
 115 accomplished by two monitor commands "object_add" and "device_add".
 116
 117 For example, the following commands add another 4GB vNVDIMM device to
 118 the guest:
 119
 120  (qemu) object_add memory-backend-file,id=mem2,share=on,mem-path=new_nvdimm.img,size=4G
 121  (qemu) device_add nvdimm,id=nvdimm2,memdev=mem2
 122
 123 Note:
 124
 125 1. Each hotplugged vNVDIMM device consumes one memory slot. Users
 126    should always ensure the memory option "-m ...,slots=N" specifies
 127    enough number of slots, i.e.
 128      N >= number of RAM devices +
 129           number of statically plugged vNVDIMM devices +
 130           number of hotplugged vNVDIMM devices
 131
 132 2. The similar is required for the memory option "-m ...,maxmem=M", i.e.
 133      M >= size of RAM devices +
 134           size of statically plugged vNVDIMM devices +
 135           size of hotplugged vNVDIMM devices
 136
 137 Alignment
 138 ---------
 139
 140 QEMU uses mmap(2) to maps vNVDIMM backends and aligns the mapping
 141 address to the page size (getpagesize(2)) by default. However, some
 142 types of backends may require an alignment different than the page
 143 size. In that case, QEMU v2.12.0 and later provide 'align' option to
 144 memory-backend-file to allow users to specify the proper alignment.
 145 For device dax (e.g., /dev/dax0.0), this alignment needs to match the
 146 alignment requirement of the device dax. The NUM of 'align=NUM' option
 147 must be larger than or equal to the 'align' of device dax.
 148 We can use one of the following commands to show the 'align' of device dax.
 149
 150     ndctl list -X
 151     daxctl list -R
 152
 153 In order to get the proper 'align' of device dax, you need to install
 154 the library 'libdaxctl'.
 155
 156 For example, device dax require the 2 MB alignment, so we can use
 157 following QEMU command line options to use it (/dev/dax0.0) as the
 158 backend of vNVDIMM:
 159
 160  -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4G,align=2M
 161  -device nvdimm,id=nvdimm1,memdev=mem1
 162
 163 Guest Data Persistence
 164 ----------------------
 165
 166 Though QEMU supports multiple types of vNVDIMM backends on Linux,
 167 the only backend that can guarantee the guest write persistence is:
 168
 169 A. DAX device (e.g., /dev/dax0.0, ) or
 170 B. DAX file(mounted with dax option)
 171
 172 When using B (A file supporting direct mapping of persistent memory)
 173 as a backend, write persistence is guaranteed if the host kernel has
 174 support for the MAP_SYNC flag in the mmap system call (available
 175 since Linux 4.15 and on certain distro kernels) and additionally
 176 both 'pmem' and 'share' flags are set to 'on' on the backend.
 177
 178 If these conditions are not satisfied i.e. if either 'pmem' or 'share'
 179 are not set, if the backend file does not support DAX or if MAP_SYNC
 180 is not supported by the host kernel, write persistence is not
 181 guaranteed after a system crash. For compatibility reasons, these
 182 conditions are ignored if not satisfied. Currently, no way is
 183 provided to test for them.
 184 For more details, please reference mmap(2) man page:
 185 http://man7.org/linux/man-pages/man2/mmap.2.html.
 186
 187 When using other types of backends, it's suggested to set 'unarmed'
 188 option of '-device nvdimm' to 'on', which sets the unarmed flag of the
 189 guest NVDIMM region mapping structure.  This unarmed flag indicates
 190 guest software that this vNVDIMM device contains a region that cannot
 191 accept persistent writes. In result, for example, the guest Linux
 192 NVDIMM driver, marks such vNVDIMM device as read-only.
 193
 194 Backend File Setup Example
 195 --------------------------
 196
 197 Here are two examples showing how to setup these persistent backends on
 198 linux using the tool ndctl [3].
 199
 200 A. DAX device
 201
 202 Use the following command to set up /dev/dax0.0 so that the entirety of
 203 namespace0.0 can be exposed as an emulated NVDIMM to the guest:
 204
 205     ndctl create-namespace -f -e namespace0.0 -m devdax
 206
 207 The /dev/dax0.0 could be used directly in "mem-path" option.
 208
 209 B. DAX file
 210
 211 Individual files on a DAX host file system can be exposed as emulated
 212 NVDIMMS.  First an fsdax block device is created, partitioned, and then
 213 mounted with the "dax" mount option:
 214
 215     ndctl create-namespace -f -e namespace0.0 -m fsdax
 216     (partition /dev/pmem0 with name pmem0p1)
 217     mount -o dax /dev/pmem0p1 /mnt
 218     (create or copy a disk image file with qemu-img(1), cp(1), or dd(1)
 219      in /mnt)
 220
 221 Then the new file in /mnt could be used in "mem-path" option.
 222
 223 NVDIMM Persistence
 224 ------------------
 225
 226 ACPI 6.2 Errata A added support for a new Platform Capabilities Structure
 227 which allows the platform to communicate what features it supports related to
 228 NVDIMM data persistence.  Users can provide a persistence value to a guest via
 229 the optional "nvdimm-persistence" machine command line option:
 230
 231     -machine pc,accel=kvm,nvdimm,nvdimm-persistence=cpu
 232
 233 There are currently two valid values for this option:
 234
 235 "mem-ctrl" - The platform supports flushing dirty data from the memory
 236              controller to the NVDIMMs in the event of power loss.
 237
 238 "cpu"      - The platform supports flushing dirty data from the CPU cache to
 239              the NVDIMMs in the event of power loss.  This implies that the
 240              platform also supports flushing dirty data through the memory
 241              controller on power loss.
 242
 243 If the vNVDIMM backend is in host persistent memory that can be accessed in
 244 SNIA NVM Programming Model [1] (e.g., Intel NVDIMM), it's suggested to set
 245 the 'pmem' option of memory-backend-file to 'on'. When 'pmem' is 'on' and QEMU
 246 is built with libpmem [2] support (configured with --enable-libpmem), QEMU
 247 will take necessary operations to guarantee the persistence of its own writes
 248 to the vNVDIMM backend(e.g., in vNVDIMM label emulation and live migration).
 249 If 'pmem' is 'on' while there is no libpmem support, qemu will exit and report
 250 a "lack of libpmem support" message to ensure the persistence is available.
 251 For example, if we want to ensure the persistence for some backend file,
 252 use the QEMU command line:
 253
 254     -object memory-backend-file,id=nv_mem,mem-path=/XXX/yyy,size=4G,pmem=on
 255
 256 References
 257 ----------
 258
 259 [1] NVM Programming Model (NPM)
 260         Version 1.2
 261     https://www.snia.org/sites/default/files/technical_work/final/NVMProgrammingModel_v1.2.pdf
 262 [2] Persistent Memory Development Kit (PMDK), formerly known as NVML project, home page:
 263     http://pmem.io/pmdk/
 264 [3] ndctl-create-namespace - provision or reconfigure a namespace
 265     http://pmem.io/ndctl/ndctl-create-namespace.html