Monitor and manager daemon memory usage generally scales with the size of the
cluster. Note that at boot-time and during topology changes and recovery these
daemons will need more RAM than they do during steady-state operation, so plan
-for peak usage. For very small clusters, 32 GB suffices. For
-clusters of up to, say, 300 OSDs go with 64GB. For clusters built with (or
-which will grow to) even more OSDS you should provision
-128GB. You may also want to consider tuning settings like ``mon_osd_cache_size``
-or ``rocksdb_cache_size`` after careful research.
+for peak usage. For very small clusters, 32 GB suffices. For clusters of up to,
+say, 300 OSDs go with 64GB. For clusters built with (or which will grow to)
+even more OSDs you should provision 128GB. You may also want to consider
+tuning the following settings:
+
+* :confval:`mon_osd_cache_size`
+* :confval:`rocksdb_cache_size`
+
Metadata servers (ceph-mds)
---------------------------
The metadata daemon memory utilization depends on how much memory its cache is
configured to consume. We recommend 1 GB as a minimum for most systems. See
-``mds_cache_memory``.
+:confval:`mds_cache_memory_limit`.
Memory
this may cause extremely slow performance).
- Setting the memory target between 2GB and 4GB typically works but may result
- in degraded performance as metadata may be read from disk during IO unless the
+ in degraded performance: metadata may be read from disk during IO unless the
active data set is relatively small.
- 4GB is the current default :confval:`osd_memory_target` size. This default
performance when there many (small) objects or when large (256GB/OSD
or more) data sets are processed.
-.. important:: The OSD memory autotuning is "best effort". While the OSD may
+.. important:: OSD memory autotuning is "best effort". Although the OSD may
unmap memory to allow the kernel to reclaim it, there is no guarantee that
the kernel will actually reclaim freed memory within a specific time
frame. This applies especially in older versions of Ceph, where transparent
huge pages can prevent the kernel from reclaiming memory that was freed from
fragmented huge pages. Modern versions of Ceph disable transparent huge
- pages at the application level to avoid this, though that still does not
- guarantee that the kernel will immediately reclaim unmapped memory. The OSD
- may still at times exceed it's memory target. We recommend budgeting around
- 20% extra memory on your system to prevent OSDs from going OOM during
- temporary spikes or due to any delay in reclaiming freed pages by the
- kernel. That value may be more or less than needed depending on the exact
- configuration of the system.
+ pages at the application level to avoid this, but that does not
+ guarantee that the kernel will immediately reclaim unmapped memory. The OSD
+ may still at times exceed its memory target. We recommend budgeting
+ approximately 20% extra memory on your system to prevent OSDs from going OOM
+ (**O**\ut **O**\f **M**\emory) during temporary spikes or due to delay in
+ the kernel reclaiming freed pages. That 20% value might be more or less than
+ needed, depending on the exact configuration of the system.
When using the legacy FileStore back end, the page cache is used for caching
data, so no tuning is normally needed. When using the legacy FileStore backend,
Plan your data storage configuration carefully. There are significant cost and
performance tradeoffs to consider when planning for data storage. Simultaneous
-OS operations, and simultaneous request for read and write operations from
-multiple daemons against a single drive can slow performance considerably.
+OS operations and simultaneous requests from multiple daemons for read and
+write operations against a single drive can slow performance.
Hard Disk Drives
----------------
-OSDs should have plenty of hard disk drive space for object data. We recommend a
-minimum hard disk drive size of 1 terabyte. Consider the cost-per-gigabyte
-advantage of larger disks. We recommend dividing the price of the hard disk
-drive by the number of gigabytes to arrive at a cost per gigabyte, because
-larger drives may have a significant impact on the cost-per-gigabyte. For
-example, a 1 terabyte hard disk priced at $75.00 has a cost of $0.07 per
-gigabyte (i.e., $75 / 1024 = 0.0732). By contrast, a 3 terabyte hard disk priced
-at $150.00 has a cost of $0.05 per gigabyte (i.e., $150 / 3072 = 0.0488). In the
-foregoing example, using the 1 terabyte disks would generally increase the cost
-per gigabyte by 40%--rendering your cluster substantially less cost efficient.
+OSDs should have plenty of storage drive space for object data. We recommend a
+minimum disk drive size of 1 terabyte. Consider the cost-per-gigabyte advantage
+of larger disks. We recommend dividing the price of the disk drive by the
+number of gigabytes to arrive at a cost per gigabyte, because larger drives may
+have a significant impact on the cost-per-gigabyte. For example, a 1 terabyte
+hard disk priced at $75.00 has a cost of $0.07 per gigabyte (i.e., $75 / 1024 =
+0.0732). By contrast, a 3 terabyte disk priced at $150.00 has a cost of $0.05
+per gigabyte (i.e., $150 / 3072 = 0.0488). In the foregoing example, using the
+1 terabyte disks would generally increase the cost per gigabyte by
+40%--rendering your cluster substantially less cost efficient.
.. tip:: Running multiple OSDs on a single SAS / SATA drive
is **NOT** a good idea. NVMe drives, however, can achieve
.. tip:: Running an OSD and a monitor or a metadata server on a single
drive is also **NOT** a good idea.
+.. tip:: With spinning disks, the SATA and SAS interface increasingly
+ becomes a bottleneck at larger capacities. See also the `Storage Networking
+ Industry Association's Total Cost of Ownership calculator`_.
+
+
Storage drives are subject to limitations on seek time, access time, read and
write times, as well as total throughput. These physical limitations affect
overall system performance--especially during recovery. We recommend using a
dedicated (ideally mirrored) drive for the operating system and software, and
one drive for each Ceph OSD Daemon you run on the host (modulo NVMe above).
-Many "slow OSD" issues not attributable to hardware failure arise from running
-an operating system and multiple OSDs on the same drive. Since the cost of troubleshooting performance issues on a small cluster likely exceeds the cost of the extra disk drives, you can optimize your cluster design planning by avoiding the temptation to overtax the OSD storage drives.
+Many "slow OSD" issues (when they are not attributable to hardware failure)
+arise from running an operating system and multiple OSDs on the same drive.
+
+It is technically possible to run multiple Ceph OSD Daemons per SAS / SATA
+drive, but this will lead to resource contention and diminish overall
+throughput.
-You may run multiple Ceph OSD Daemons per SAS / SATA drive, but this will likely
-lead to resource contention and diminish the overall throughput.
+To get the best performance out of Ceph, run the following on separate drives:
+(1) operating systems, (2) OSD data, and (3) BlueStore db. For more
+information on how to effectively use a mix of fast drives and slow drives in
+your Ceph cluster, see the `block and block.db`_ section of the Bluestore
+Configuration Reference.
Solid State Drives
------------------
-One opportunity for performance improvement is to use solid-state drives (SSDs)
-to reduce random access time and read latency while accelerating throughput.
-SSDs often cost more than 10x as much per gigabyte when compared to a hard disk
-drive, but SSDs often exhibit access times that are at least 100x faster than a
-hard disk drive.
+Ceph performance can be improved by using solid-state drives (SSDs). This
+reduces random access time and reduces latency while accelerating throughput.
-SSDs do not have moving mechanical parts so they are not necessarily subject to
-the same types of limitations as hard disk drives. SSDs do have significant
+SSDs cost more per gigabyte than do hard disk drives, but SSDs often offer
+access times that are, at a minimum, 100 times faster than hard disk drives.
+SSDs avoid hotspot issues and bottleneck issues within busy clusters, and
+they may offer better economics when TCO is evaluated holistically.
+
+SSDs do not have moving mechanical parts, so they are not necessarily subject
+to the same types of limitations as hard disk drives. SSDs do have significant
limitations though. When evaluating SSDs, it is important to consider the
performance of sequential reads and writes.
.. important:: We recommend exploring the use of SSDs to improve performance.
However, before making a significant investment in SSDs, we **strongly
- recommend** both reviewing the performance metrics of an SSD and testing the
- SSD in a test configuration to gauge performance.
+ recommend** reviewing the performance metrics of an SSD and testing the
+ SSD in a test configuration in order to gauge performance.
Relatively inexpensive SSDs may appeal to your sense of economy. Use caution.
-Acceptable IOPS are not enough when selecting an SSD for use with Ceph.
+Acceptable IOPS are not the only factor to consider when selecting an SSD for
+use with Ceph.
-SSDs have historically been cost prohibitive for object storage, though
-emerging QLC drives are closing the gap. HDD OSDs may see a significant
+SSDs have historically been cost prohibitive for object storage, but emerging
+QLC drives are closing the gap, offering greater density with lower power
+consumption and less power spent on cooling. HDD OSDs may see a significant
performance improvement by offloading WAL+DB onto an SSD.
-One way Ceph accelerates CephFS file system performance is to segregate the
-storage of CephFS metadata from the storage of the CephFS file contents. Ceph
-provides a default ``metadata`` pool for CephFS metadata. You will never have to
-create a pool for CephFS metadata, but you can create a CRUSH map hierarchy for
-your CephFS metadata pool that points only to a host's SSD storage media. See
-:ref:`CRUSH Device Class<crush-map-device-class>` for details.
+To get a better sense of the factors that determine the cost of storage, you
+might use the `Storage Networking Industry Association's Total Cost of
+Ownership calculator`_
+
+Partition Alignment
+~~~~~~~~~~~~~~~~~~~
+
+When using SSDs with Ceph, make sure that your partitions are properly aligned.
+Improperly aligned partitions suffer slower data transfer speeds than do
+properly aligned partitions. For more information about proper partition
+alignment and example commands that show how to align partitions properly, see
+`Werner Fischer's blog post on partition alignment`_.
+
+CephFS Metadata Segregation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+One way that Ceph accelerates CephFS file system performance is by segregating
+the storage of CephFS metadata from the storage of the CephFS file contents.
+Ceph provides a default ``metadata`` pool for CephFS metadata. You will never
+have to create a pool for CephFS metadata, but you can create a CRUSH map
+hierarchy for your CephFS metadata pool that points only to SSD storage media.
+See :ref:`CRUSH Device Class<crush-map-device-class>` for details.
Controllers
-----------
Disk controllers (HBAs) can have a significant impact on write throughput.
-Carefully consider your selection to ensure that they do not create
-a performance bottleneck. Notably RAID-mode (IR) HBAs may exhibit higher
-latency than simpler "JBOD" (IT) mode HBAs, and the RAID SoC, write cache,
-and battery backup can substantially increase hardware and maintenance
-costs. Some RAID HBAs can be configured with an IT-mode "personality".
+Carefully consider your selection of HBAs to ensure that they do not create a
+performance bottleneck. Notably, RAID-mode (IR) HBAs may exhibit higher latency
+than simpler "JBOD" (IT) mode HBAs. The RAID SoC, write cache, and battery
+backup can substantially increase hardware and maintenance costs. Some RAID
+HBAs can be configured with an IT-mode "personality".
.. tip:: The `Ceph blog`_ is often an excellent source of information on Ceph
performance issues. See `Ceph Write Throughput 1`_ and `Ceph Write
The larger the Ceph cluster, the more common OSD failures will be.
The faster that a placement group (PG) can recover from a ``degraded`` state to
an ``active + clean`` state, the better. Notably, fast recovery minimizes
-the liklihood of multiple, overlapping failures that can cause data to become
+the likelihood of multiple, overlapping failures that can cause data to become
temporarily unavailable or even lost. Of course, when provisioning your
network, you will have to balance price against performance.
===============
A failure domain is any failure that prevents access to one or more OSDs. That
-could be a stopped daemon on a host; a hard disk failure, an OS crash, a
-malfunctioning NIC, a failed power supply, a network outage, a power outage, and
-so forth. When planning out your hardware needs, you must balance the
+could be a stopped daemon on a host; a disk failure, an OS crash, a
+malfunctioning NIC, a failed power supply, a network outage, a power outage,
+and so forth. When planning out your hardware needs, you must balance the
temptation to reduce costs by placing too many responsibilities into too few
failure domains, and the added costs of isolating every potential failure
domain.
projects / ceph.git / blobdiff