ceph/doc/rbd/rbd-config-ref.rst

   1 =======================
   2  Config Settings
   3 =======================
   4
   5 See `Block Device`_ for additional details.
   6
   7 Generic IO Settings
   8 ===================
   9
  10 ``rbd compression hint``
  11
  12 :Description: Hint to send to the OSDs on write operations. If set to `compressible` and the OSD `bluestore compression mode` setting is `passive`, the OSD will attempt to compress the data. If set to `incompressible` and the OSD compression setting is `aggressive`, the OSD will not attempt to compress the data.
  13 :Type: Enum
  14 :Required: No
  15 :Default: ``none``
  16 :Values: ``none``, ``compressible``, ``incompressible``
  17
  18 ``rbd read from replica policy``
  19
  20 :Description: policy for determining which OSD will receive read operations. If set to `default`, the primary OSD will always be used for read operations. If set to `balance`, read operations will be sent to a randomly selected OSD within the replica set. If set to `localize`, read operations will be sent to the closest OSD as determined by the CRUSH map. Note: this feature requires the cluster to be configured with a minimum compatible OSD release of Octopus.
  21 :Type: Enum
  22 :Required: No
  23 :Default: ``default``
  24 :Values: ``default``, ``balance``, ``localize``
  25
  26 Cache Settings
  27 =======================
  28
  29 .. sidebar:: Kernel Caching
  30
  31         The kernel driver for Ceph block devices can use the Linux page cache to
  32         improve performance.
  33
  34 The user space implementation of the Ceph block device (i.e., ``librbd``) cannot
  35 take advantage of the Linux page cache, so it includes its own in-memory
  36 caching, called "RBD caching." RBD caching behaves just like well-behaved hard
  37 disk caching.  When the OS sends a barrier or a flush request, all dirty data is
  38 written to the OSDs. This means that using write-back caching is just as safe as
  39 using a well-behaved physical hard disk with a VM that properly sends flushes
  40 (i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU)
  41 algorithm, and in write-back mode it can coalesce contiguous requests for
  42 better throughput.
  43
  44 The librbd cache is enabled by default and supports three different cache
  45 policies: write-around, write-back, and write-through. Writes return
  46 immediately under both the write-around and write-back policies, unless there
  47 are more than ``rbd cache max dirty`` unwritten bytes to the storage cluster.
  48 The write-around policy differs from the write-back policy in that it does
  49 not attempt to service read requests from the cache, unlike the write-back
  50 policy, and is therefore faster for high performance write workloads. Under the
  51 write-through policy, writes return only when the data is on disk on all
  52 replicas, but reads may come from the cache.
  53
  54 Prior to receiving a flush request, the cache behaves like a write-through cache
  55 to ensure safe operation for older operating systems that do not send flushes to
  56 ensure crash consistent behavior.
  57
  58 If the librbd cache is disabled, writes and
  59 reads go directly to the storage cluster, and writes return only when the data
  60 is on disk on all replicas.
  61
  62 .. note::
  63    The cache is in memory on the client, and each RBD image has
  64    its own.  Since the cache is local to the client, there's no coherency
  65    if there are others accessing the image. Running GFS or OCFS on top of
  66    RBD will not work with caching enabled.
  67
  68
  69 The ``ceph.conf`` file settings for RBD should be set in the ``[client]``
  70 section of your configuration file. The settings include:
  71
  72
  73 ``rbd cache``
  74
  75 :Description: Enable caching for RADOS Block Device (RBD).
  76 :Type: Boolean
  77 :Required: No
  78 :Default: ``true``
  79
  80
  81 ``rbd cache policy``
  82
  83 :Description: Select the caching policy for librbd.
  84 :Type: Enum
  85 :Required: No
  86 :Default: ``writearound``
  87 :Values: ``writearound``, ``writeback``, ``writethrough``
  88
  89
  90 ``rbd cache writethrough until flush``
  91
  92 :Description: Start out in write-through mode, and switch to write-back after the first flush request is received. Enabling this is a conservative but safe setting in case VMs running on rbd are too old to send flushes, like the virtio driver in Linux before 2.6.32.
  93 :Type: Boolean
  94 :Required: No
  95 :Default: ``true``
  96
  97
  98 ``rbd cache size``
  99
 100 :Description: The RBD cache size in bytes.
 101 :Type: 64-bit Integer
 102 :Required: No
 103 :Default: ``32 MiB``
 104 :Policies: write-back and write-through
 105
 106
 107 ``rbd cache max dirty``
 108
 109 :Description: The ``dirty`` limit in bytes at which the cache triggers write-back.  If ``0``, uses write-through caching.
 110 :Type: 64-bit Integer
 111 :Required: No
 112 :Constraint: Must be less than ``rbd cache size``.
 113 :Default: ``24 MiB``
 114 :Policies: write-around and write-back
 115
 116
 117 ``rbd cache target dirty``
 118
 119 :Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache.
 120 :Type: 64-bit Integer
 121 :Required: No
 122 :Constraint: Must be less than ``rbd cache max dirty``.
 123 :Default: ``16 MiB``
 124 :Policies: write-back
 125
 126
 127 ``rbd cache max dirty age``
 128
 129 :Description: The number of seconds dirty data is in the cache before writeback starts.
 130 :Type: Float
 131 :Required: No
 132 :Default: ``1.0``
 133 :Policies: write-back
 134
 135
 136 .. _Block Device: ../../rbd
 137
 138
 139 Read-ahead Settings
 140 =======================
 141
 142 librbd supports read-ahead/prefetching to optimize small, sequential reads.
 143 This should normally be handled by the guest OS in the case of a VM,
 144 but boot loaders may not issue efficient reads. Read-ahead is automatically
 145 disabled if caching is disabled or if the policy is write-around.
 146
 147
 148 ``rbd readahead trigger requests``
 149
 150 :Description: Number of sequential read requests necessary to trigger read-ahead.
 151 :Type: Integer
 152 :Required: No
 153 :Default: ``10``
 154
 155
 156 ``rbd readahead max bytes``
 157
 158 :Description: Maximum size of a read-ahead request.  If zero, read-ahead is disabled.
 159 :Type: 64-bit Integer
 160 :Required: No
 161 :Default: ``512 KiB``
 162
 163
 164 ``rbd readahead disable after bytes``
 165
 166 :Description: After this many bytes have been read from an RBD image, read-ahead is disabled for that image until it is closed.  This allows the guest OS to take over read-ahead once it is booted.  If zero, read-ahead stays enabled.
 167 :Type: 64-bit Integer
 168 :Required: No
 169 :Default: ``50 MiB``
 170
 171
 172 Image Features
 173 ==============
 174
 175 RBD supports advanced features which can be specified via the command line when creating images or the default features can be specified via Ceph config file via 'rbd_default_features = <sum of feature numeric values>' or 'rbd_default_features = <comma-delimited list of CLI values>'
 176
 177 ``Layering``
 178
 179 :Description: Layering enables you to use cloning.
 180 :Internal value: 1
 181 :CLI value: layering
 182 :Added in: v0.52 (Bobtail)
 183 :KRBD support: since v3.10
 184 :Default: yes
 185
 186 ``Striping v2``
 187
 188 :Description: Striping spreads data across multiple objects. Striping helps with parallelism for sequential read/write workloads.
 189 :Internal value: 2
 190 :CLI value: striping
 191 :Added in: v0.55 (Bobtail)
 192 :KRBD support: since v3.10 (default striping only, "fancy" striping added in v4.17)
 193 :Default: yes
 194
 195 ``Exclusive locking``
 196
 197 :Description: When enabled, it requires a client to get a lock on an object before making a write. Exclusive lock should only be enabled when a single client is accessing an image at the same time.
 198 :Internal value: 4
 199 :CLI value: exclusive-lock
 200 :Added in: v0.92 (Hammer)
 201 :KRBD support: since v4.9
 202 :Default: yes
 203
 204 ``Object map``
 205
 206 :Description: Object map support depends on exclusive lock support. Block devices are thin provisioned—meaning, they only store data that actually exists. Object map support helps track which objects actually exist (have data stored on a drive). Enabling object map support speeds up I/O operations for cloning; importing and exporting a sparsely populated image; and deleting.
 207 :Internal value: 8
 208 :CLI value: object-map
 209 :Added in: v0.93 (Hammer)
 210 :KRBD support: since v5.3
 211 :Default: yes
 212
 213
 214 ``Fast-diff``
 215
 216 :Description: Fast-diff support depends on object map support and exclusive lock support. It adds another property to the object map, which makes it much faster to generate diffs between snapshots of an image, and the actual data usage of a snapshot much faster.
 217 :Internal value: 16
 218 :CLI value: fast-diff
 219 :Added in: v9.0.1 (Infernalis)
 220 :KRBD support: since v5.3
 221 :Default: yes
 222
 223
 224 ``Deep-flatten``
 225
 226 :Description: Deep-flatten makes rbd flatten work on all the snapshots of an image, in addition to the image itself. Without it, snapshots of an image will still rely on the parent, so the parent will not be delete-able until the snapshots are deleted. Deep-flatten makes a parent independent of its clones, even if they have snapshots.
 227 :Internal value: 32
 228 :CLI value: deep-flatten
 229 :Added in: v9.0.2 (Infernalis)
 230 :KRBD support: since v5.1
 231 :Default: yes
 232
 233
 234 ``Journaling``
 235
 236 :Description: Journaling support depends on exclusive lock support. Journaling records all modifications to an image in the order they occur. RBD mirroring utilizes the journal to replicate a crash consistent image to a remote cluster.
 237 :Internal value: 64
 238 :CLI value: journaling
 239 :Added in: v10.0.1 (Jewel)
 240 :KRBD support: no
 241 :Default: no
 242
 243
 244 ``Data pool``
 245
 246 :Description: On erasure-coded pools, the image data block objects need to be stored on a separate pool from the image metadata.
 247 :Internal value: 128
 248 :Added in: v11.1.0 (Kraken)
 249 :KRBD support: since v4.11
 250 :Default: no
 251
 252
 253 ``Operations``
 254
 255 :Description: Used to restrict older clients from performing certain maintenance operations against an image (e.g. clone, snap create).
 256 :Internal value: 256
 257 :Added in: v13.0.2 (Mimic)
 258 :KRBD support: since v4.16
 259
 260
 261 ``Migrating``
 262
 263 :Description: Used to restrict older clients from opening an image when it is in migration state.
 264 :Internal value: 512
 265 :Added in: v14.0.1 (Nautilus)
 266 :KRBD support: no
 267
 268 ``Non-primary``
 269
 270 :Description: Used to restrict changes to non-primary images using snapshot-based mirroring.
 271 :Internal value: 1024
 272 :Added in: v15.2.0 (Octopus)
 273 :KRBD support: no
 274
 275
 276 QOS Settings
 277 ============
 278
 279 librbd supports limiting per image IO, controlled by the following
 280 settings.
 281
 282 ``rbd qos iops limit``
 283
 284 :Description: The desired limit of IO operations per second.
 285 :Type: Unsigned Integer
 286 :Required: No
 287 :Default: ``0``
 288
 289
 290 ``rbd qos bps limit``
 291
 292 :Description: The desired limit of IO bytes per second.
 293 :Type: Unsigned Integer
 294 :Required: No
 295 :Default: ``0``
 296
 297
 298 ``rbd qos read iops limit``
 299
 300 :Description: The desired limit of read operations per second.
 301 :Type: Unsigned Integer
 302 :Required: No
 303 :Default: ``0``
 304
 305
 306 ``rbd qos write iops limit``
 307
 308 :Description: The desired limit of write operations per second.
 309 :Type: Unsigned Integer
 310 :Required: No
 311 :Default: ``0``
 312
 313
 314 ``rbd qos read bps limit``
 315
 316 :Description: The desired limit of read bytes per second.
 317 :Type: Unsigned Integer
 318 :Required: No
 319 :Default: ``0``
 320
 321
 322 ``rbd qos write bps limit``
 323
 324 :Description: The desired limit of write bytes per second.
 325 :Type: Unsigned Integer
 326 :Required: No
 327 :Default: ``0``
 328
 329
 330 ``rbd qos iops burst``
 331
 332 :Description: The desired burst limit of IO operations.
 333 :Type: Unsigned Integer
 334 :Required: No
 335 :Default: ``0``
 336
 337
 338 ``rbd qos bps burst``
 339
 340 :Description: The desired burst limit of IO bytes.
 341 :Type: Unsigned Integer
 342 :Required: No
 343 :Default: ``0``
 344
 345
 346 ``rbd qos read iops burst``
 347
 348 :Description: The desired burst limit of read operations.
 349 :Type: Unsigned Integer
 350 :Required: No
 351 :Default: ``0``
 352
 353
 354 ``rbd qos write iops burst``
 355
 356 :Description: The desired burst limit of write operations.
 357 :Type: Unsigned Integer
 358 :Required: No
 359 :Default: ``0``
 360
 361
 362 ``rbd qos read bps burst``
 363
 364 :Description: The desired burst limit of read bytes.
 365 :Type: Unsigned Integer
 366 :Required: No
 367 :Default: ``0``
 368
 369
 370 ``rbd qos write bps burst``
 371
 372 :Description: The desired burst limit of write bytes.
 373 :Type: Unsigned Integer
 374 :Required: No
 375 :Default: ``0``
 376
 377
 378 ``rbd qos schedule tick min``
 379
 380 :Description: The minimum schedule tick (in milliseconds) for QoS.
 381 :Type: Unsigned Integer
 382 :Required: No
 383 :Default: ``50``