ceph/doc/start/hardware-recommendations.rst

   1 .. _hardware-recommendations:
   2
   3 ==========================
   4  Hardware Recommendations
   5 ==========================
   6
   7 Ceph was designed to run on commodity hardware, which makes building and
   8 maintaining petabyte-scale data clusters economically feasible.
   9 When planning out your cluster hardware, you will need to balance a number
  10 of considerations, including failure domains and potential performance
  11 issues. Hardware planning should include distributing Ceph daemons and
  12 other processes that use Ceph across many hosts. Generally, we recommend
  13 running Ceph daemons of a specific type on a host configured for that type
  14 of daemon. We recommend using other hosts for processes that utilize your
  15 data cluster (e.g., OpenStack, CloudStack, etc).
  16
  17
  18 .. tip:: Check out the `Ceph blog`_ too.
  19
  20
  21 CPU
  22 ===
  23
  24 CephFS metadata servers are CPU intensive, so they should have significant
  25 processing power (e.g., quad core or better CPUs) and benefit from higher clock
  26 rate (frequency in GHz). Ceph OSDs run the :term:`RADOS` service, calculate
  27 data placement with :term:`CRUSH`, replicate data, and maintain their own copy of the
  28 cluster map. Therefore, OSD nodes should have a reasonable amount of processing
  29 power. Requirements vary by use-case; a starting point might be one core per
  30 OSD for light / archival usage, and two cores per OSD for heavy workloads such
  31 as RBD volumes attached to VMs.  Monitor / manager nodes do not have heavy CPU
  32 demands so a modest processor can be chosen for them.  Also consider whether the
  33 host machine will run CPU-intensive processes in addition to Ceph daemons. For
  34 example, if your hosts will run computing VMs (e.g., OpenStack Nova), you will
  35 need to ensure that these other processes leave sufficient processing power for
  36 Ceph daemons. We recommend running additional CPU-intensive processes on
  37 separate hosts to avoid resource contention.
  38
  39
  40 RAM
  41 ===
  42
  43 Generally, more RAM is better.  Monitor / manager nodes for a modest cluster
  44 might do fine with 64GB; for a larger cluster with hundreds of OSDs 128GB
  45 is a reasonable target.  There is a memory target for BlueStore OSDs that
  46 defaults to 4GB.  Factor in a prudent margin for the operating system and
  47 administrative tasks (like monitoring and metrics) as well as increased
  48 consumption during recovery:  provisioning ~8GB per BlueStore OSD
  49 is advised.
  50
  51 Monitors and managers (ceph-mon and ceph-mgr)
  52 ---------------------------------------------
  53
  54 Monitor and manager daemon memory usage generally scales with the size of the
  55 cluster.  Note that at boot-time and during topology changes and recovery these
  56 daemons will need more RAM than they do during steady-state operation, so plan
  57 for peak usage.  For very small clusters, 32 GB suffices.  For
  58 clusters of up to, say, 300 OSDs go with 64GB.  For clusters built with (or
  59 which will grow to) even more OSDS you should provision
  60 129GB.  You may also want to consider tuning settings like ``mon_osd_cache_size``
  61 or ``rocksdb_cache_size`` after careful research.
  62
  63 Metadata servers (ceph-mds)
  64 ---------------------------
  65
  66 The metadata daemon memory utilization depends on how much memory its cache is
  67 configured to consume.  We recommend 1 GB as a minimum for most systems.  See
  68 ``mds_cache_memory``.
  69
  70 OSDs (ceph-osd)
  71 ---------------
  72
  73 Memory
  74 ======
  75
  76 Bluestore uses its own memory to cache data rather than relying on the
  77 operating system page cache.  In bluestore you can adjust the amount of memory
  78 the OSD attempts to consume with the ``osd_memory_target`` configuration
  79 option.
  80
  81 - Setting the osd_memory_target below 2GB is typically not recommended (it may
  82   fail to keep the memory that low and may also cause extremely slow performance.
  83
  84 - Setting the memory target between 2GB and 4GB typically works but may result
  85   in degraded performance as metadata may be read from disk during IO unless the
  86   active data set is relatively small.
  87
  88 - 4GB is the current default osd_memory_target size and was set that way to try
  89   and balance memory requirements and OSD performance for typical use cases.
  90
  91 - Setting the osd_memory_target higher than 4GB may improve performance when
  92   there are many (small) objects or large (256GB/OSD or more) data sets being
  93   processed.
  94
  95 .. important:: The OSD memory autotuning is "best effort".  While the OSD may
  96    unmap memory to allow the kernel to reclaim it, there is no guarantee that
  97    the kernel will actually reclaim freed memory within any specific time
  98    frame.  This is especially true in older versions of Ceph where transparent
  99    huge pages can prevent the kernel from reclaiming memory freed from
 100    fragmented huge pages. Modern versions of Ceph disable transparent huge
 101    pages at the application level to avoid this, though that still does not
 102    guarantee that the kernel will immediately reclaim unmapped memory.  The OSD
 103    may still at times exceed it's memory target.  We recommend budgeting around
 104    20% extra memory on your system to prevent OSDs from going OOM during
 105    temporary spikes or due to any delay in reclaiming freed pages by the
 106    kernel.  That value may be more or less than needed depending on the exact
 107    configuration of the system.
 108
 109 When using the legacy FileStore backend, the page cache is used for caching
 110 data, so no tuning is normally needed, and the OSD memory consumption is
 111 generally related to the number of PGs per daemon in the system.
 112
 113
 114 Data Storage
 115 ============
 116
 117 Plan your data storage configuration carefully. There are significant cost and
 118 performance tradeoffs to consider when planning for data storage. Simultaneous
 119 OS operations, and simultaneous request for read and write operations from
 120 multiple daemons against a single drive can slow performance considerably.
 121
 122 .. important:: Since Ceph has to write all data to the journal (or WAL+DB)
 123    before it can ACK writes, having this metadata and OSD
 124    performance in balance is really important!
 125
 126
 127 Hard Disk Drives
 128 ----------------
 129
 130 OSDs should have plenty of hard disk drive space for object data. We recommend a
 131 minimum hard disk drive size of 1 terabyte. Consider the cost-per-gigabyte
 132 advantage of larger disks. We recommend dividing the price of the hard disk
 133 drive by the number of gigabytes to arrive at a cost per gigabyte, because
 134 larger drives may have a significant impact on the cost-per-gigabyte. For
 135 example, a 1 terabyte hard disk priced at $75.00 has a cost of $0.07 per
 136 gigabyte (i.e., $75 / 1024 = 0.0732). By contrast, a 3 terabyte hard disk priced
 137 at $150.00 has a cost of $0.05 per gigabyte (i.e., $150 / 3072 = 0.0488). In the
 138 foregoing example, using the 1 terabyte disks would generally increase the cost
 139 per gigabyte by 40%--rendering your cluster substantially less cost efficient.
 140
 141 .. tip:: Running multiple OSDs on a single SAS / SATA drive
 142    is **NOT** a good idea.  NVMe drives, however, can achieve
 143    improved performance by being split into two more more OSDs.
 144
 145 .. tip:: Running an OSD and a monitor or a metadata server on a single
 146    drive is also **NOT** a good idea.
 147
 148 Storage drives are subject to limitations on seek time, access time, read and
 149 write times, as well as total throughput. These physical limitations affect
 150 overall system performance--especially during recovery. We recommend using a
 151 dedicated (ideally mirrored) drive for the operating system and software, and
 152 one drive for each Ceph OSD Daemon you run on the host (modulo NVMe above).
 153 Many "slow OSD" issues not attributable to hardware failure arise from running
 154 an operating system, multiple OSDs, and/or multiple journals on the same drive.
 155 Since the cost of troubleshooting performance issues on a small cluster likely
 156 exceeds the cost of the extra disk drives, you can optimize your cluster
 157 design planning by avoiding the temptation to overtax the OSD storage drives.
 158
 159 You may run multiple Ceph OSD Daemons per SAS / SATA drive, but this will likely
 160 lead to resource contention and diminish the overall throughput. You may store a
 161 journal and object data on the same drive, but this may increase the time it
 162 takes to journal a write and ACK to the client. Ceph must write to the journal
 163 before it can ACK the write.
 164
 165 Ceph best practices dictate that you should run operating systems, OSD data and
 166 OSD journals on separate drives.
 167
 168
 169 Solid State Drives
 170 ------------------
 171
 172 One opportunity for performance improvement is to use solid-state drives (SSDs)
 173 to reduce random access time and read latency while accelerating throughput.
 174 SSDs often cost more than 10x as much per gigabyte when compared to a hard disk
 175 drive, but SSDs often exhibit access times that are at least 100x faster than a
 176 hard disk drive.
 177
 178 SSDs do not have moving mechanical parts so they are not necessarily subject to
 179 the same types of limitations as hard disk drives. SSDs do have significant
 180 limitations though. When evaluating SSDs, it is important to consider the
 181 performance of sequential reads and writes. An SSD that has 400MB/s sequential
 182 write throughput may have much better performance than an SSD with 120MB/s of
 183 sequential write throughput when storing multiple journals for multiple OSDs.
 184
 185 .. important:: We recommend exploring the use of SSDs to improve performance.
 186    However, before making a significant investment in SSDs, we **strongly
 187    recommend** both reviewing the performance metrics of an SSD and testing the
 188    SSD in a test configuration to gauge performance.
 189
 190 Since SSDs have no moving mechanical parts, it makes sense to use them in the
 191 areas of Ceph that do not use a lot of storage space (e.g., journals).
 192 Relatively inexpensive SSDs may appeal to your sense of economy. Use caution.
 193 Acceptable IOPS are not enough when selecting an SSD for use with Ceph. There
 194 are a few important performance considerations for journals and SSDs:
 195
 196 - **Write-intensive semantics:** Journaling involves write-intensive semantics,
 197   so you should ensure that the SSD you choose to deploy will perform equal to
 198   or better than a hard disk drive when writing data. Inexpensive SSDs may
 199   introduce write latency even as they accelerate access time, because
 200   sometimes high performance hard drives can write as fast or faster than
 201   some of the more economical SSDs available on the market!
 202
 203 - **Sequential Writes:** When you store multiple journals on an SSD you must
 204   consider the sequential write limitations of the SSD too, since they may be
 205   handling requests to write to multiple OSD journals simultaneously.
 206
 207 - **Partition Alignment:** A common problem with SSD performance is that
 208   people like to partition drives as a best practice, but they often overlook
 209   proper partition alignment with SSDs, which can cause SSDs to transfer data
 210   much more slowly. Ensure that SSD partitions are properly aligned.
 211
 212 SSDs have historically been cost prohibitive for object storage, though
 213 emerging QLC drives are closing the gap.  HDD OSDs may see a significant
 214 performance improvement by offloading WAL+DB onto an SSD.
 215
 216 One way Ceph accelerates CephFS file system performance is to segregate the
 217 storage of CephFS metadata from the storage of the CephFS file contents. Ceph
 218 provides a default ``metadata`` pool for CephFS metadata. You will never have to
 219 create a pool for CephFS metadata, but you can create a CRUSH map hierarchy for
 220 your CephFS metadata pool that points only to a host's SSD storage media. See
 221 :ref:`CRUSH Device Class<crush-map-device-class>` for details.
 222
 223
 224 Controllers
 225 -----------
 226
 227 Disk controllers (HBAs) can have a significant impact on write throughput.
 228 Carefully consider your selection to ensure that they do not create
 229 a performance bottleneck.  Notably RAID-mode (IR) HBAs may exhibit higher
 230 latency than simpler "JBOD" (IT) mode HBAs, and the RAID SoC, write cache,
 231 and battery backup can substantially increase hardware and maintenance
 232 costs.  Some RAID HBAs can be configured with an IT-mode "personality".
 233
 234 .. tip:: The `Ceph blog`_ is often an excellent source of information on Ceph
 235    performance issues. See `Ceph Write Throughput 1`_ and `Ceph Write
 236    Throughput 2`_ for additional details.
 237
 238
 239 Additional Considerations
 240 -------------------------
 241
 242 You typically will run multiple OSDs per host, but you should ensure that the
 243 aggregate throughput of your OSD drives doesn't exceed the network bandwidth
 244 required to service a client's need to read or write data. You should also
 245 consider what percentage of the overall data the cluster stores on each host. If
 246 the percentage on a particular host is large and the host fails, it can lead to
 247 problems such as exceeding the ``full ratio``,  which causes Ceph to halt
 248 operations as a safety precaution that prevents data loss.
 249
 250 When you run multiple OSDs per host, you also need to ensure that the kernel
 251 is up to date. See `OS Recommendations`_ for notes on ``glibc`` and
 252 ``syncfs(2)`` to ensure that your hardware performs as expected when running
 253 multiple OSDs per host.
 254
 255
 256 Networks
 257 ========
 258
 259 Provision at least 10Gbps+ networking in your racks. Replicating 1TB of data
 260 across a 1Gbps network takes 3 hours, and 10TBs takes 30 hours! By contrast,
 261 with a 10Gbps network, the replication times would be 20 minutes and 1 hour
 262 respectively. In a petabyte-scale cluster, failure of an OSD drive is an
 263 expectation, not an exception. System administrators will appreciate PGs
 264 recovering from a ``degraded`` state to an ``active + clean`` state as rapidly
 265 as possible, with price / performance tradeoffs taken into consideration.
 266 Additionally, some deployment tools employ VLANs to make  hardware and network
 267 cabling more manageable. VLANs using 802.1q protocol require VLAN-capable NICs
 268 and Switches. The added hardware expense may be offset by the operational cost
 269 savings for network setup and maintenance. When using VLANs to handle VM
 270 traffic between the cluster and compute stacks (e.g., OpenStack, CloudStack,
 271 etc.), there is additional value in using 10G Ethernet or better; 40Gb or
 272 25/50/100 Gb networking as of 2020 is common for production clusters.
 273
 274 Top-of-rack routers for each network also need to be able to communicate with
 275 spine routers that have even faster throughput, often 40Gbp/s or more.
 276
 277
 278 Your server hardware should have a Baseboard Management Controller (BMC).
 279 Administration and deployment tools may also use BMCs extensively, especially
 280 via IPMI or Redfish, so consider
 281 the cost/benefit tradeoff of an out-of-band network for administration.
 282 Hypervisor SSH access, VM image uploads, OS image installs, management sockets,
 283 etc. can impose significant loads on a network.  Running three networks may seem
 284 like overkill, but each traffic path represents a potential capacity, throughput
 285 and/or performance bottleneck that you should carefully consider before
 286 deploying a large scale data cluster.
 287
 288
 289 Failure Domains
 290 ===============
 291
 292 A failure domain is any failure that prevents access to one or more OSDs. That
 293 could be a stopped daemon on a host; a hard disk failure, an OS crash, a
 294 malfunctioning NIC, a failed power supply, a network outage, a power outage, and
 295 so forth. When planning out your hardware needs, you must balance the
 296 temptation to reduce costs by placing too many responsibilities into too few
 297 failure domains, and the added costs of isolating every potential failure
 298 domain.
 299
 300
 301 Minimum Hardware Recommendations
 302 ================================
 303
 304 Ceph can run on inexpensive commodity hardware. Small production clusters
 305 and development clusters can run successfully with modest hardware.
 306
 307 +--------------+----------------+-----------------------------------------+
 308 |  Process     | Criteria       | Minimum Recommended                     |
 309 +==============+================+=========================================+
 310 | ``ceph-osd`` | Processor      | - 1 core minimum                        |
 311 |              |                | - 1 core per 200-500 MB/s               |
 312 |              |                | - 1 core per 1000-3000 IOPS             |
 313 |              |                |                                         |
 314 |              |                | * Results are before replication.       |
 315 |              |                | * Results may vary with different       |
 316 |              |                |   CPU models and Ceph features.         |
 317 |              |                |   (erasure coding, compression, etc)    |
 318 |              |                | * ARM processors specifically may       |
 319 |              |                |   require additional cores.             |
 320 |              |                | * Actual performance depends on many    |
 321 |              |                |   factors including drives, net, and    |
 322 |              |                |   client throughput and latency.        |
 323 |              |                |   Benchmarking is highly recommended.   |
 324 |              +----------------+-----------------------------------------+
 325 |              | RAM            | - 4GB+ per daemon (more is better)      |
 326 |              |                | - 2-4GB often functions (may be slow)   |
 327 |              |                | - Less than 2GB not recommended         |
 328 |              +----------------+-----------------------------------------+
 329 |              | Volume Storage |  1x storage drive per daemon            |
 330 |              +----------------+-----------------------------------------+
 331 |              | DB/WAL         |  1x SSD partition per daemon (optional) |
 332 |              +----------------+-----------------------------------------+
 333 |              | Network        |  1x 1GbE+ NICs (10GbE+ recommended)     |
 334 +--------------+----------------+-----------------------------------------+
 335 | ``ceph-mon`` | Processor      | - 2 cores minimum                       |
 336 |              +----------------+-----------------------------------------+
 337 |              | RAM            |  24GB+ per daemon                       |
 338 |              +----------------+-----------------------------------------+
 339 |              | Disk Space     |  60 GB per daemon                       |
 340 |              +----------------+-----------------------------------------+
 341 |              | Network        |  1x 1GbE+ NICs                          |
 342 +--------------+----------------+-----------------------------------------+
 343 | ``ceph-mds`` | Processor      | - 2 cores minimum                       |
 344 |              +----------------+-----------------------------------------+
 345 |              | RAM            |  2GB+ per daemon                        |
 346 |              +----------------+-----------------------------------------+
 347 |              | Disk Space     |  1 MB per daemon                        |
 348 |              +----------------+-----------------------------------------+
 349 |              | Network        |  1x 1GbE+ NICs                          |
 350 +--------------+----------------+-----------------------------------------+
 351
 352 .. tip:: If you are running an OSD with a single disk, create a
 353    partition for your volume storage that is separate from the partition
 354    containing the OS. Generally, we recommend separate disks for the
 355    OS and the volume storage.
 356
 357
 358
 359
 360
 361 .. _Ceph blog: https://ceph.com/community/blog/
 362 .. _Ceph Write Throughput 1: http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/
 363 .. _Ceph Write Throughput 2: http://ceph.com/community/ceph-performance-part-2-write-throughput-without-ssd-journals/
 364 .. _Mapping Pools to Different Types of OSDs: ../../rados/operations/crush-map#placing-different-pools-on-different-osds
 365 .. _OS Recommendations: ../os-recommendations