[ceph.git] / ceph / doc / rbd / rbd-config-ref.rst

=======================
 librbd Settings
=======================

See `Block Device`_ for additional details.

Generic IO Settings
===================

``rbd compression hint``

:Description: Hint to send to the OSDs on write operations. If set to `compressible` and the OSD `bluestore compression mode` setting is `passive`, the OSD will attempt to compress the data. If set to `incompressible` and the OSD compression setting is `aggressive`, the OSD will not attempt to compress the data.
:Type: Enum
:Required: No
:Default: ``none``
:Values: ``none``, ``compressible``, ``incompressible``

Cache Settings
=======================

.. sidebar:: Kernel Caching

	The kernel driver for Ceph block devices can use the Linux page cache to 
	improve performance.

The user space implementation of the Ceph block device (i.e., ``librbd``) cannot
take advantage of the Linux page cache, so it includes its own in-memory
caching, called "RBD caching." RBD caching behaves just like well-behaved hard
disk caching.  When the OS sends a barrier or a flush request, all dirty data is
written to the OSDs. This means that using write-back caching is just as safe as
using a well-behaved physical hard disk with a VM that properly sends flushes
(i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU)
algorithm, and in write-back mode it  can coalesce contiguous requests for
better throughput.

.. versionadded:: 0.46

Ceph supports write-back caching for RBD. To enable it, add  ``rbd cache =
true`` to the ``[client]`` section of your ``ceph.conf`` file. By default
``librbd`` does not perform any caching. Writes and reads go directly to the
storage cluster, and writes return only when the data is on disk on all
replicas. With caching enabled, writes return immediately, unless there are more
than ``rbd cache max dirty`` unflushed bytes. In this case, the write triggers
writeback and blocks until enough bytes are flushed.

.. versionadded:: 0.47

Ceph supports write-through caching for RBD. You can set the size of
the cache, and you can set targets and limits to switch from
write-back caching to write through caching. To enable write-through
mode, set ``rbd cache max dirty`` to 0. This means writes return only
when the data is on disk on all replicas, but reads may come from the
cache. The cache is in memory on the client, and each RBD image has
its own.  Since the cache is local to the client, there's no coherency
if there are others accessing the image. Running GFS or OCFS on top of
RBD will not work with caching enabled.

The ``ceph.conf`` file settings for RBD should be set in the ``[client]``
section of your configuration file. The settings include: 


``rbd cache``

:Description: Enable caching for RADOS Block Device (RBD).
:Type: Boolean
:Required: No
:Default: ``true``


``rbd cache size``

:Description: The RBD cache size in bytes.
:Type: 64-bit Integer
:Required: No
:Default: ``32 MiB``


``rbd cache max dirty``

:Description: The ``dirty`` limit in bytes at which the cache triggers write-back.  If ``0``, uses write-through caching.
:Type: 64-bit Integer
:Required: No
:Constraint: Must be less than ``rbd cache size``.
:Default: ``24 MiB``


``rbd cache target dirty``

:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache.
:Type: 64-bit Integer
:Required: No
:Constraint: Must be less than ``rbd cache max dirty``.
:Default: ``16 MiB``


``rbd cache max dirty age``

:Description: The number of seconds dirty data is in the cache before writeback starts. 
:Type: Float
:Required: No
:Default: ``1.0``

.. versionadded:: 0.60

``rbd cache writethrough until flush``

:Description: Start out in write-through mode, and switch to write-back after the first flush request is received. Enabling this is a conservative but safe setting in case VMs running on rbd are too old to send flushes, like the virtio driver in Linux before 2.6.32.
:Type: Boolean
:Required: No
:Default: ``true``

.. _Block Device: ../../rbd


Read-ahead Settings
=======================

.. versionadded:: 0.86

RBD supports read-ahead/prefetching to optimize small, sequential reads.
This should normally be handled by the guest OS in the case of a VM,
but boot loaders may not issue efficient reads.
Read-ahead is automatically disabled if caching is disabled.


``rbd readahead trigger requests``

:Description: Number of sequential read requests necessary to trigger read-ahead.
:Type: Integer
:Required: No
:Default: ``10``


``rbd readahead max bytes``

:Description: Maximum size of a read-ahead request.  If zero, read-ahead is disabled.
:Type: 64-bit Integer
:Required: No
:Default: ``512 KiB``


``rbd readahead disable after bytes``

:Description: After this many bytes have been read from an RBD image, read-ahead is disabled for that image until it is closed.  This allows the guest OS to take over read-ahead once it is booted.  If zero, read-ahead stays enabled.
:Type: 64-bit Integer
:Required: No
:Default: ``50 MiB``


RBD Features
============

RBD supports advanced features which can be specified via the command line when creating images or the default features can be specified via Ceph config file via 'rbd_default_features = <sum of feature numeric values>' or 'rbd_default_features = <comma-delimited list of CLI values>'

``Layering``

:Description: Layering enables you to use cloning.
:Internal value: 1
:CLI value: layering
:Added in: v0.70 (Emperor)
:KRBD support: since v3.10
:Default: yes

``Striping v2``

:Description: Striping spreads data across multiple objects. Striping helps with parallelism for sequential read/write workloads.
:Internal value: 2
:CLI value: striping
:Added in: v0.70 (Emperor)
:KRBD support: since v3.10
:Default: yes

``Exclusive locking``

:Description: When enabled, it requires a client to get a lock on an object before making a write. Exclusive lock should only be enabled when a single client is accessing an image at the same time. 
:Internal value: 4
:CLI value: exclusive-lock
:Added in: v0.92 (Hammer)
:KRBD support: since v4.9
:Default: yes

``Object map``

:Description: Object map support depends on exclusive lock support. Block devices are thin provisioned—meaning, they only store data that actually exists. Object map support helps track which objects actually exist (have data stored on a drive). Enabling object map support speeds up I/O operations for cloning; importing and exporting a sparsely populated image; and deleting.
:Internal value: 8
:CLI value: object-map
:Added in: v0.93 (Hammer)
:KRBD support: no
:Default: yes


``Fast-diff``

:Description: Fast-diff support depends on object map support and exclusive lock support. It adds another property to the object map, which makes it much faster to generate diffs between snapshots of an image, and the actual data usage of a snapshot much faster.
:Internal value: 16
:CLI value: fast-diff
:Added in: v9.0.1 (Infernalis)
:KRBD support: no
:Default: yes


``Deep-flatten``

:Description: Deep-flatten makes rbd flatten work on all the snapshots of an image, in addition to the image itself. Without it, snapshots of an image will still rely on the parent, so the parent will not be delete-able until the snapshots are deleted. Deep-flatten makes a parent independent of its clones, even if they have snapshots.
:Internal value: 32
:CLI value: deep-flatten
:Added in: v9.0.2 (Infernalis)
:KRBD support: no
:Default: yes


``Journaling``

:Description: Journaling support depends on exclusive lock support. Journaling records all modifications to an image in the order they occur. RBD mirroring utilizes the journal to replicate a crash consistent image to a remote cluster.
:Internal value: 64
:CLI value: journaling
:Added in: v10.0.1 (Jewel)
:KRBD support: no
:Default: no


``Data pool``

:Description: On erasure-coded pools, the image data block objects need to be stored on a separate pool from the image metadata.
:Internal value: 128
:Added in: v11.1.0 (Kraken)
:KRBD support: since v4.11
:Default: no


``Operations``

:Description: Used to restrict older clients from performing certain maintenance operations against an image (e.g. clone, snap create).
:Internal value: 256
:Added in: v13.0.2 (Mimic)
:KRBD support: since v4.16


``Migrating``

:Description: Used to restrict older clients from opening an image when it is in migration state.
:Internal value: 512
:Added in: v14.0.1 (Nautilus)
:KRBD support: no


RBD QOS Settings
================

RBD supports limiting per image IO, controlled by the following
settings.

``rbd qos iops limit``

:Description: The desired limit of IO operations per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos bps limit``

:Description: The desired limit of IO bytes per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos read iops limit``

:Description: The desired limit of read operations per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos write iops limit``

:Description: The desired limit of write operations per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos read bps limit``

:Description: The desired limit of read bytes per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos write bps limit``

:Description: The desired limit of write bytes per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos iops burst``

:Description: The desired burst limit of IO operations.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos bps burst``

:Description: The desired burst limit of IO bytes.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos read iops burst``

:Description: The desired burst limit of read operations.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos write iops burst``

:Description: The desired burst limit of write operations.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos read bps burst``

:Description: The desired burst limit of read bytes.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos write bps burst``

:Description: The desired burst limit of write bytes.
:Type: Unsigned Integer
:Required: No
:Default: ``0``


``rbd qos schedule tick min``

:Description: The minimum schedule tick (in milliseconds) for QoS.
:Type: Unsigned Integer
:Required: No
:Default: ``50``
Commit	Line	Data
7c673cae FG	1	=======================
	2	librbd Settings
	3	=======================
	4
	5	See `Block Device`_ for additional details.
	6
92f5a8d4 TL	7	Generic IO Settings
	8	===================
	9
	10	``rbd compression hint``
	11
	12	:Description: Hint to send to the OSDs on write operations. If set to `compressible` and the OSD `bluestore compression mode` setting is `passive`, the OSD will attempt to compress the data. If set to `incompressible` and the OSD compression setting is `aggressive`, the OSD will not attempt to compress the data.
	13	:Type: Enum
	14	:Required: No
	15	:Default: ``none``
	16	:Values: ``none``, ``compressible``, ``incompressible``
	17
7c673cae FG	18	Cache Settings
	19	=======================
	20
	21	.. sidebar:: Kernel Caching
	22
	23	The kernel driver for Ceph block devices can use the Linux page cache to
	24	improve performance.
	25
	26	The user space implementation of the Ceph block device (i.e., ``librbd``) cannot
	27	take advantage of the Linux page cache, so it includes its own in-memory
	28	caching, called "RBD caching." RBD caching behaves just like well-behaved hard
	29	disk caching. When the OS sends a barrier or a flush request, all dirty data is
	30	written to the OSDs. This means that using write-back caching is just as safe as
	31	using a well-behaved physical hard disk with a VM that properly sends flushes
	32	(i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU)
	33	algorithm, and in write-back mode it can coalesce contiguous requests for
	34	better throughput.
	35
	36	.. versionadded:: 0.46
	37
	38	Ceph supports write-back caching for RBD. To enable it, add ``rbd cache =
	39	true`` to the ``[client]`` section of your ``ceph.conf`` file. By default
	40	``librbd`` does not perform any caching. Writes and reads go directly to the
	41	storage cluster, and writes return only when the data is on disk on all
	42	replicas. With caching enabled, writes return immediately, unless there are more
	43	than ``rbd cache max dirty`` unflushed bytes. In this case, the write triggers
	44	writeback and blocks until enough bytes are flushed.
	45
	46	.. versionadded:: 0.47
	47
	48	Ceph supports write-through caching for RBD. You can set the size of
	49	the cache, and you can set targets and limits to switch from
	50	write-back caching to write through caching. To enable write-through
	51	mode, set ``rbd cache max dirty`` to 0. This means writes return only
	52	when the data is on disk on all replicas, but reads may come from the
	53	cache. The cache is in memory on the client, and each RBD image has
	54	its own. Since the cache is local to the client, there's no coherency
	55	if there are others accessing the image. Running GFS or OCFS on top of
	56	RBD will not work with caching enabled.
	57
	58	The ``ceph.conf`` file settings for RBD should be set in the ``[client]``
	59	section of your configuration file. The settings include:
	60
	61
	62	``rbd cache``
	63
	64	:Description: Enable caching for RADOS Block Device (RBD).
	65	:Type: Boolean
	66	:Required: No
	67	:Default: ``true``
	68
	69
	70	``rbd cache size``
	71
	72	:Description: The RBD cache size in bytes.
	73	:Type: 64-bit Integer
	74	:Required: No
	75	:Default: ``32 MiB``
	76
	77
	78	``rbd cache max dirty``
	79
	80	:Description: The ``dirty`` limit in bytes at which the cache triggers write-back. If ``0``, uses write-through caching.
	81	:Type: 64-bit Integer
82	:Required: No
83	:Constraint: Must be less than ``rbd cache size``.
84	:Default: ``24 MiB``
85
86
87	``rbd cache target dirty``
88
89	:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache.
90	:Type: 64-bit Integer
91	:Required: No
92	:Constraint: Must be less than ``rbd cache max dirty``.
93	:Default: ``16 MiB``
94
95
96	``rbd cache max dirty age``
97
98	:Description: The number of seconds dirty data is in the cache before writeback starts.
99	:Type: Float
100	:Required: No
101	:Default: ``1.0``
102
103	.. versionadded:: 0.60
104
105	``rbd cache writethrough until flush``
106
107	:Description: Start out in write-through mode, and switch to write-back after the first flush request is received. Enabling this is a conservative but safe setting in case VMs running on rbd are too old to send flushes, like the virtio driver in Linux before 2.6.32.
108	:Type: Boolean
109	:Required: No
110	:Default: ``true``
111
d2e6a577	112	.. _Block Device: ../../rbd
7c673cae FG	113
	114
	115	Read-ahead Settings
	116	=======================
	117
	118	.. versionadded:: 0.86
	119
	120	RBD supports read-ahead/prefetching to optimize small, sequential reads.
	121	This should normally be handled by the guest OS in the case of a VM,
	122	but boot loaders may not issue efficient reads.
	123	Read-ahead is automatically disabled if caching is disabled.
	124
	125
	126	``rbd readahead trigger requests``
	127
	128	:Description: Number of sequential read requests necessary to trigger read-ahead.
	129	:Type: Integer
	130	:Required: No
	131	:Default: ``10``
	132
	133
	134	``rbd readahead max bytes``
	135
	136	:Description: Maximum size of a read-ahead request. If zero, read-ahead is disabled.
	137	:Type: 64-bit Integer
	138	:Required: No
	139	:Default: ``512 KiB``
	140
	141
	142	``rbd readahead disable after bytes``
	143
	144	:Description: After this many bytes have been read from an RBD image, read-ahead is disabled for that image until it is closed. This allows the guest OS to take over read-ahead once it is booted. If zero, read-ahead stays enabled.
	145	:Type: 64-bit Integer
	146	:Required: No
	147	:Default: ``50 MiB``
11fdf7f2 TL	148
	149
	150	RBD Features
	151	============
	152
	153	RBD supports advanced features which can be specified via the command line when creating images or the default features can be specified via Ceph config file via 'rbd_default_features = <sum of feature numeric values>' or 'rbd_default_features = <comma-delimited list of CLI values>'
	154
	155	``Layering``
	156
	157	:Description: Layering enables you to use cloning.
	158	:Internal value: 1
	159	:CLI value: layering
	160	:Added in: v0.70 (Emperor)
	161	:KRBD support: since v3.10
	162	:Default: yes
	163
	164	``Striping v2``
	165
	166	:Description: Striping spreads data across multiple objects. Striping helps with parallelism for sequential read/write workloads.
	167	:Internal value: 2
	168	:CLI value: striping
	169	:Added in: v0.70 (Emperor)
	170	:KRBD support: since v3.10
	171	:Default: yes
	172
	173	``Exclusive locking``
	174
	175	:Description: When enabled, it requires a client to get a lock on an object before making a write. Exclusive lock should only be enabled when a single client is accessing an image at the same time.
	176	:Internal value: 4
	177	:CLI value: exclusive-lock
	178	:Added in: v0.92 (Hammer)
	179	:KRBD support: since v4.9
	180	:Default: yes
	181
	182	``Object map``
	183
	184	:Description: Object map support depends on exclusive lock support. Block devices are thin provisioned—meaning, they only store data that actually exists. Object map support helps track which objects actually exist (have data stored on a drive). Enabling object map support speeds up I/O operations for cloning; importing and exporting a sparsely populated image; and deleting.
	185	:Internal value: 8
	186	:CLI value: object-map
	187	:Added in: v0.93 (Hammer)
	188	:KRBD support: no
	189	:Default: yes
	190
	191
	192	``Fast-diff``
	193
	194	:Description: Fast-diff support depends on object map support and exclusive lock support. It adds another property to the object map, which makes it much faster to generate diffs between snapshots of an image, and the actual data usage of a snapshot much faster.
	195	:Internal value: 16
	196	:CLI value: fast-diff
	197	:Added in: v9.0.1 (Infernalis)
	198	:KRBD support: no
	199	:Default: yes
	200
	201
	202	``Deep-flatten``
	203
	204	:Description: Deep-flatten makes rbd flatten work on all the snapshots of an image, in addition to the image itself. Without it, snapshots of an image will still rely on the parent, so the parent will not be delete-able until the snapshots are deleted. Deep-flatten makes a parent independent of its clones, even if they have snapshots.
	205	:Internal value: 32
	206	:CLI value: deep-flatten
	207	:Added in: v9.0.2 (Infernalis)
	208	:KRBD support: no
	209	:Default: yes
	210
	211
212	``Journaling``
213
214	:Description: Journaling support depends on exclusive lock support. Journaling records all modifications to an image in the order they occur. RBD mirroring utilizes the journal to replicate a crash consistent image to a remote cluster.
215	:Internal value: 64
216	:CLI value: journaling
217	:Added in: v10.0.1 (Jewel)
218	:KRBD support: no
219	:Default: no
220
221
222	``Data pool``
223
224	:Description: On erasure-coded pools, the image data block objects need to be stored on a separate pool from the image metadata.
225	:Internal value: 128
226	:Added in: v11.1.0 (Kraken)
227	:KRBD support: since v4.11
228	:Default: no
229
230
231	``Operations``
232
233	:Description: Used to restrict older clients from performing certain maintenance operations against an image (e.g. clone, snap create).
234	:Internal value: 256
235	:Added in: v13.0.2 (Mimic)
236	:KRBD support: since v4.16
237
238
239	``Migrating``
240
241	:Description: Used to restrict older clients from opening an image when it is in migration state.
242	:Internal value: 512
243	:Added in: v14.0.1 (Nautilus)
244	:KRBD support: no
245
246
247	RBD QOS Settings
248	================
249
250	RBD supports limiting per image IO, controlled by the following
251	settings.
252
253	``rbd qos iops limit``
254
255	:Description: The desired limit of IO operations per second.
256	:Type: Unsigned Integer
257	:Required: No
258	:Default: ``0``
259
260
261	``rbd qos bps limit``
262
263	:Description: The desired limit of IO bytes per second.
264	:Type: Unsigned Integer
265	:Required: No
266	:Default: ``0``
267
268
269	``rbd qos read iops limit``
270
271	:Description: The desired limit of read operations per second.
272	:Type: Unsigned Integer
273	:Required: No
274	:Default: ``0``
275
276
277	``rbd qos write iops limit``
278
279	:Description: The desired limit of write operations per second.
280	:Type: Unsigned Integer
281	:Required: No
282	:Default: ``0``
283
284
285	``rbd qos read bps limit``
286
287	:Description: The desired limit of read bytes per second.
288	:Type: Unsigned Integer
289	:Required: No
290	:Default: ``0``
291
292
293	``rbd qos write bps limit``
294
295	:Description: The desired limit of write bytes per second.
296	:Type: Unsigned Integer
297	:Required: No
298	:Default: ``0``
299
300
301	``rbd qos iops burst``
302
303	:Description: The desired burst limit of IO operations.
304	:Type: Unsigned Integer
305	:Required: No
306	:Default: ``0``
307
308
309	``rbd qos bps burst``
310
311	:Description: The desired burst limit of IO bytes.
312	:Type: Unsigned Integer
313	:Required: No
314	:Default: ``0``
315
316
317	``rbd qos read iops burst``
318
319	:Description: The desired burst limit of read operations.
320	:Type: Unsigned Integer
321	:Required: No
322	:Default: ``0``
323
324
325	``rbd qos write iops burst``
326
327	:Description: The desired burst limit of write operations.
328	:Type: Unsigned Integer
329	:Required: No
330	:Default: ``0``
331
332
333	``rbd qos read bps burst``
334
335	:Description: The desired burst limit of read bytes.
336	:Type: Unsigned Integer
337	:Required: No
338	:Default: ``0``
339
340
341	``rbd qos write bps burst``
342
343	:Description: The desired burst limit of write bytes.
344	:Type: Unsigned Integer
345	:Required: No
346	:Default: ``0``
347
348
349	``rbd qos schedule tick min``
350
351	:Description: The minimum schedule tick (in milliseconds) for QoS.
352	:Type: Unsigned Integer
353	:Required: No
354	:Default: ``50``