ceph/doc/start/hardware-recommendations.rst

   1 ==========================
   2  Hardware Recommendations
   3 ==========================
   4
   5 Ceph was designed to run on commodity hardware, which makes building and
   6 maintaining petabyte-scale data clusters economically feasible.
   7 When planning out your cluster hardware, you will need to balance a number
   8 of considerations, including failure domains and potential performance
   9 issues. Hardware planning should include distributing Ceph daemons and
  10 other processes that use Ceph across many hosts. Generally, we recommend
  11 running Ceph daemons of a specific type on a host configured for that type
  12 of daemon. We recommend using other hosts for processes that utilize your
  13 data cluster (e.g., OpenStack, CloudStack, etc).
  14
  15
  16 .. tip:: Check out the `Ceph blog`_ too.
  17
  18
  19 CPU
  20 ===
  21
  22 Ceph metadata servers dynamically redistribute their load, which is CPU
  23 intensive. So your metadata servers should have significant processing power
  24 (e.g., quad core or better CPUs). Ceph OSDs run the :term:`RADOS` service, calculate
  25 data placement with :term:`CRUSH`, replicate data, and maintain their own copy of the
  26 cluster map. Therefore, OSDs should have a reasonable amount of processing power
  27 (e.g., dual core processors). Monitors simply maintain a master copy of the
  28 cluster map, so they are not CPU intensive. You must also consider whether the
  29 host machine will run CPU-intensive processes in addition to Ceph daemons. For
  30 example, if your hosts will run computing VMs (e.g., OpenStack Nova), you will
  31 need to ensure that these other processes leave sufficient processing power for
  32 Ceph daemons. We recommend running additional CPU-intensive processes on
  33 separate hosts.
  34
  35
  36 RAM
  37 ===
  38
  39 Generally, more RAM is better.
  40
  41 Monitors and managers (ceph-mon and ceph-mgr)
  42 ---------------------------------------------
  43
  44 Monitor and manager daemon memory usage generally scales with the size of the
  45 cluster.  For small clusters, 1-2 GB is generally sufficient.  For
  46 large clusters, you should provide more (5-10 GB).  You may also want
  47 to consider tuning settings like ``mon_osd_cache_size`` or
  48 ``rocksdb_cache_size``.
  49
  50 Metadata servers (ceph-mds)
  51 ---------------------------
  52
  53 The metadata daemon memory utilization depends on how much memory its cache is
  54 configured to consume.  We recommend 1 GB as a minimum for most systems.  See
  55 ``mds_cache_memory``.
  56
  57 OSDs (ceph-osd)
  58 ---------------
  59
  60 By default, OSDs that use the BlueStore backend require 3-5 GB of RAM.  You can
  61 adjust the amount of memory the OSD consumes with the ``osd_memory_target`` configuration option when BlueStore is in use.  When using the legacy FileStore backend, the operating system page cache is used for caching data, so no tuning is normally needed, and the OSD memory consumption is generally related to the number of PGs per daemon in the system.
  62
  63
  64 Data Storage
  65 ============
  66
  67 Plan your data storage configuration carefully. There are significant cost and
  68 performance tradeoffs to consider when planning for data storage. Simultaneous
  69 OS operations, and simultaneous request for read and write operations from
  70 multiple daemons against a single drive can slow performance considerably.
  71
  72 .. important:: Since Ceph has to write all data to the journal before it can
  73    send an ACK (for XFS at least), having the journal and OSD
  74    performance in balance is really important!
  75
  76
  77 Hard Disk Drives
  78 ----------------
  79
  80 OSDs should have plenty of hard disk drive space for object data. We recommend a
  81 minimum hard disk drive size of 1 terabyte. Consider the cost-per-gigabyte
  82 advantage of larger disks. We recommend dividing the price of the hard disk
  83 drive by the number of gigabytes to arrive at a cost per gigabyte, because
  84 larger drives may have a significant impact on the cost-per-gigabyte. For
  85 example, a 1 terabyte hard disk priced at $75.00 has a cost of $0.07 per
  86 gigabyte (i.e., $75 / 1024 = 0.0732). By contrast, a 3 terabyte hard disk priced
  87 at $150.00 has a cost of $0.05 per gigabyte (i.e., $150 / 3072 = 0.0488). In the
  88 foregoing example, using the 1 terabyte disks would generally increase the cost
  89 per gigabyte by 40%--rendering your cluster substantially less cost efficient.
  90 Also, the larger the storage drive capacity, the more memory per Ceph OSD Daemon
  91 you will need, especially during rebalancing, backfilling and recovery. A
  92 general rule of thumb is ~1GB of RAM for 1TB of storage space.
  93
  94 .. tip:: Running multiple OSDs on a single disk--irrespective of partitions--is
  95    **NOT** a good idea.
  96
  97 .. tip:: Running an OSD and a monitor or a metadata server on a single
  98    disk--irrespective of partitions--is **NOT** a good idea either.
  99
 100 Storage drives are subject to limitations on seek time, access time, read and
 101 write times, as well as total throughput. These physical limitations affect
 102 overall system performance--especially during recovery. We recommend using a
 103 dedicated drive for the operating system and software, and one drive for each
 104 Ceph OSD Daemon you run on the host. Most "slow OSD" issues arise due to running
 105 an operating system, multiple OSDs, and/or multiple journals on the same drive.
 106 Since the cost of troubleshooting performance issues on a small cluster likely
 107 exceeds the cost of the extra disk drives, you can accelerate your cluster
 108 design planning by avoiding the temptation to overtax the OSD storage drives.
 109
 110 You may run multiple Ceph OSD Daemons per hard disk drive, but this will likely
 111 lead to resource contention and diminish the overall throughput. You may store a
 112 journal and object data on the same drive, but this may increase the time it
 113 takes to journal a write and ACK to the client. Ceph must write to the journal
 114 before it can ACK the write.
 115
 116 Ceph best practices dictate that you should run operating systems, OSD data and
 117 OSD journals on separate drives.
 118
 119
 120 Solid State Drives
 121 ------------------
 122
 123 One opportunity for performance improvement is to use solid-state drives (SSDs)
 124 to reduce random access time and read latency while accelerating throughput.
 125 SSDs often cost more than 10x as much per gigabyte when compared to a hard disk
 126 drive, but SSDs often exhibit access times that are at least 100x faster than a
 127 hard disk drive.
 128
 129 SSDs do not have moving mechanical parts so they are not necessarily subject to
 130 the same types of limitations as hard disk drives. SSDs do have significant
 131 limitations though. When evaluating SSDs, it is important to consider the
 132 performance of sequential reads and writes. An SSD that has 400MB/s sequential
 133 write throughput may have much better performance than an SSD with 120MB/s of
 134 sequential write throughput when storing multiple journals for multiple OSDs.
 135
 136 .. important:: We recommend exploring the use of SSDs to improve performance.
 137    However, before making a significant investment in SSDs, we **strongly
 138    recommend** both reviewing the performance metrics of an SSD and testing the
 139    SSD in a test configuration to gauge performance.
 140
 141 Since SSDs have no moving mechanical parts, it makes sense to use them in the
 142 areas of Ceph that do not use a lot of storage space (e.g., journals).
 143 Relatively inexpensive SSDs may appeal to your sense of economy. Use caution.
 144 Acceptable IOPS are not enough when selecting an SSD for use with Ceph. There
 145 are a few important performance considerations for journals and SSDs:
 146
 147 - **Write-intensive semantics:** Journaling involves write-intensive semantics,
 148   so you should ensure that the SSD you choose to deploy will perform equal to
 149   or better than a hard disk drive when writing data. Inexpensive SSDs may
 150   introduce write latency even as they accelerate access time, because
 151   sometimes high performance hard drives can write as fast or faster than
 152   some of the more economical SSDs available on the market!
 153
 154 - **Sequential Writes:** When you store multiple journals on an SSD you must
 155   consider the sequential write limitations of the SSD too, since they may be
 156   handling requests to write to multiple OSD journals simultaneously.
 157
 158 - **Partition Alignment:** A common problem with SSD performance is that
 159   people like to partition drives as a best practice, but they often overlook
 160   proper partition alignment with SSDs, which can cause SSDs to transfer data
 161   much more slowly. Ensure that SSD partitions are properly aligned.
 162
 163 While SSDs are cost prohibitive for object storage, OSDs may see a significant
 164 performance improvement by storing an OSD's journal on an SSD and the OSD's
 165 object data on a separate hard disk drive. The ``osd journal`` configuration
 166 setting defaults to ``/var/lib/ceph/osd/$cluster-$id/journal``. You can mount
 167 this path to an SSD or to an SSD partition so that it is not merely a file on
 168 the same disk as the object data.
 169
 170 One way Ceph accelerates CephFS filesystem performance is to segregate the
 171 storage of CephFS metadata from the storage of the CephFS file contents. Ceph
 172 provides a default ``metadata`` pool for CephFS metadata. You will never have to
 173 create a pool for CephFS metadata, but you can create a CRUSH map hierarchy for
 174 your CephFS metadata pool that points only to a host's SSD storage media. See
 175 `Mapping Pools to Different Types of OSDs`_ for details.
 176
 177
 178 Controllers
 179 -----------
 180
 181 Disk controllers also have a significant impact on write throughput. Carefully,
 182 consider your selection of disk controllers to ensure that they do not create
 183 a performance bottleneck.
 184
 185 .. tip:: The `Ceph blog`_ is often an excellent source of information on Ceph
 186    performance issues. See `Ceph Write Throughput 1`_ and `Ceph Write
 187    Throughput 2`_ for additional details.
 188
 189
 190 Additional Considerations
 191 -------------------------
 192
 193 You may run multiple OSDs per host, but you should ensure that the sum of the
 194 total throughput of your OSD hard disks doesn't exceed the network bandwidth
 195 required to service a client's need to read or write data. You should also
 196 consider what percentage of the overall data the cluster stores on each host. If
 197 the percentage on a particular host is large and the host fails, it can lead to
 198 problems such as exceeding the ``full ratio``,  which causes Ceph to halt
 199 operations as a safety precaution that prevents data loss.
 200
 201 When you run multiple OSDs per host, you also need to ensure that the kernel
 202 is up to date. See `OS Recommendations`_ for notes on ``glibc`` and
 203 ``syncfs(2)`` to ensure that your hardware performs as expected when running
 204 multiple OSDs per host.
 205
 206 Hosts with high numbers of OSDs (e.g., > 20) may spawn a lot of threads,
 207 especially during recovery and rebalancing. Many Linux kernels default to
 208 a relatively small maximum number of threads (e.g., 32k). If you encounter
 209 problems starting up OSDs on hosts with a high number of OSDs, consider
 210 setting ``kernel.pid_max`` to a higher number of threads. The theoretical
 211 maximum is 4,194,303 threads. For example, you could add the following to
 212 the ``/etc/sysctl.conf`` file::
 213
 214         kernel.pid_max = 4194303
 215
 216
 217 Networks
 218 ========
 219
 220 We recommend that each host have at least two 1Gbps network interface
 221 controllers (NICs). Since most commodity hard disk drives have a throughput of
 222 approximately 100MB/second, your NICs should be able to handle the traffic for
 223 the OSD disks on your host. We recommend a minimum of two NICs to account for a
 224 public (front-side) network and a cluster (back-side) network. A cluster network
 225 (preferably not connected to the internet) handles the additional load for data
 226 replication and helps stop denial of service attacks that prevent the cluster
 227 from achieving ``active + clean`` states for placement groups as OSDs replicate
 228 data across the cluster. Consider starting with a 10Gbps network in your racks.
 229 Replicating 1TB of data across a 1Gbps network takes 3 hours, and 3TBs (a
 230 typical drive configuration) takes 9 hours. By contrast, with a 10Gbps network,
 231 the  replication times would be 20 minutes and 1 hour respectively. In a
 232 petabyte-scale cluster, failure of an OSD disk should be an expectation, not an
 233 exception. System administrators will appreciate PGs recovering from a
 234 ``degraded`` state to an ``active + clean`` state as rapidly as possible, with
 235 price / performance tradeoffs taken into consideration. Additionally, some
 236 deployment tools  (e.g., Dell's Crowbar) deploy with five different networks,
 237 but employ VLANs to make hardware and network cabling more manageable. VLANs
 238 using 802.1q protocol require VLAN-capable NICs and Switches. The added hardware
 239 expense may be offset by the operational cost savings for network setup and
 240 maintenance. When using VLANs to handle VM traffic between the cluster
 241 and compute stacks (e.g., OpenStack, CloudStack, etc.), it is also worth
 242 considering using 10G Ethernet. Top-of-rack routers for each network also need
 243 to be able to communicate with spine routers that have even faster
 244 throughput--e.g.,  40Gbps to 100Gbps.
 245
 246 Your server hardware should have a Baseboard Management Controller (BMC).
 247 Administration and deployment tools may also use BMCs extensively, so consider
 248 the cost/benefit tradeoff of an out-of-band network for administration.
 249 Hypervisor SSH access, VM image uploads, OS image installs, management sockets,
 250 etc. can impose significant loads on a network.  Running three networks may seem
 251 like overkill, but each traffic path represents a potential capacity, throughput
 252 and/or performance bottleneck that you should carefully consider before
 253 deploying a large scale data cluster.
 254
 255
 256 Failure Domains
 257 ===============
 258
 259 A failure domain is any failure that prevents access to one or more OSDs. That
 260 could be a stopped daemon on a host; a hard disk failure,  an OS crash, a
 261 malfunctioning NIC, a failed power supply, a network outage, a power outage, and
 262 so forth. When planning out your hardware needs, you must balance the
 263 temptation to reduce costs by placing too many responsibilities into too few
 264 failure domains, and the added costs of isolating every potential failure
 265 domain.
 266
 267
 268 Minimum Hardware Recommendations
 269 ================================
 270
 271 Ceph can run on inexpensive commodity hardware. Small production clusters
 272 and development clusters can run successfully with modest hardware.
 273
 274 +--------------+----------------+-----------------------------------------+
 275 |  Process     | Criteria       | Minimum Recommended                     |
 276 +==============+================+=========================================+
 277 | ``ceph-osd`` | Processor      | - 1x 64-bit AMD-64                      |
 278 |              |                | - 1x 32-bit ARM dual-core or better     |
 279 |              +----------------+-----------------------------------------+
 280 |              | RAM            |  ~1GB for 1TB of storage per daemon     |
 281 |              +----------------+-----------------------------------------+
 282 |              | Volume Storage |  1x storage drive per daemon            |
 283 |              +----------------+-----------------------------------------+
 284 |              | Journal        |  1x SSD partition per daemon (optional) |
 285 |              +----------------+-----------------------------------------+
 286 |              | Network        |  2x 1GB Ethernet NICs                   |
 287 +--------------+----------------+-----------------------------------------+
 288 | ``ceph-mon`` | Processor      | - 1x 64-bit AMD-64                      |
 289 |              |                | - 1x 32-bit ARM dual-core or better     |
 290 |              +----------------+-----------------------------------------+
 291 |              | RAM            |  1 GB per daemon                        |
 292 |              +----------------+-----------------------------------------+
 293 |              | Disk Space     |  10 GB per daemon                       |
 294 |              +----------------+-----------------------------------------+
 295 |              | Network        |  2x 1GB Ethernet NICs                   |
 296 +--------------+----------------+-----------------------------------------+
 297 | ``ceph-mds`` | Processor      | - 1x 64-bit AMD-64 quad-core            |
 298 |              |                | - 1x 32-bit ARM quad-core               |
 299 |              +----------------+-----------------------------------------+
 300 |              | RAM            |  1 GB minimum per daemon                |
 301 |              +----------------+-----------------------------------------+
 302 |              | Disk Space     |  1 MB per daemon                        |
 303 |              +----------------+-----------------------------------------+
 304 |              | Network        |  2x 1GB Ethernet NICs                   |
 305 +--------------+----------------+-----------------------------------------+
 306
 307 .. tip:: If you are running an OSD with a single disk, create a
 308    partition for your volume storage that is separate from the partition
 309    containing the OS. Generally, we recommend separate disks for the
 310    OS and the volume storage.
 311
 312
 313 Production Cluster Examples
 314 ===========================
 315
 316 Production clusters for petabyte scale data storage may also use commodity
 317 hardware, but should have considerably more memory, processing power and data
 318 storage to account for heavy traffic loads.
 319
 320 Dell Example
 321 ------------
 322
 323 A recent (2012) Ceph cluster project is using two fairly robust hardware
 324 configurations for Ceph OSDs, and a lighter configuration for monitors.
 325
 326 +----------------+----------------+------------------------------------+
 327 |  Configuration | Criteria       | Minimum Recommended                |
 328 +================+================+====================================+
 329 | Dell PE R510   | Processor      |  2x 64-bit quad-core Xeon CPUs     |
 330 |                +----------------+------------------------------------+
 331 |                | RAM            |  16 GB                             |
 332 |                +----------------+------------------------------------+
 333 |                | Volume Storage |  8x 2TB drives. 1 OS, 7 Storage    |
 334 |                +----------------+------------------------------------+
 335 |                | Client Network |  2x 1GB Ethernet NICs              |
 336 |                +----------------+------------------------------------+
 337 |                | OSD Network    |  2x 1GB Ethernet NICs              |
 338 |                +----------------+------------------------------------+
 339 |                | Mgmt. Network  |  2x 1GB Ethernet NICs              |
 340 +----------------+----------------+------------------------------------+
 341 | Dell PE R515   | Processor      |  1x hex-core Opteron CPU           |
 342 |                +----------------+------------------------------------+
 343 |                | RAM            |  16 GB                             |
 344 |                +----------------+------------------------------------+
 345 |                | Volume Storage |  12x 3TB drives. Storage           |
 346 |                +----------------+------------------------------------+
 347 |                | OS Storage     |  1x 500GB drive. Operating System. |
 348 |                +----------------+------------------------------------+
 349 |                | Client Network |  2x 1GB Ethernet NICs              |
 350 |                +----------------+------------------------------------+
 351 |                | OSD Network    |  2x 1GB Ethernet NICs              |
 352 |                +----------------+------------------------------------+
 353 |                | Mgmt. Network  |  2x 1GB Ethernet NICs              |
 354 +----------------+----------------+------------------------------------+
 355
 356
 357
 358
 359 .. _Ceph blog: https://ceph.com/community/blog/
 360 .. _Ceph Write Throughput 1: http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/
 361 .. _Ceph Write Throughput 2: http://ceph.com/community/ceph-performance-part-2-write-throughput-without-ssd-journals/
 362 .. _Mapping Pools to Different Types of OSDs: ../../rados/operations/crush-map#placing-different-pools-on-different-osds
 363 .. _OS Recommendations: ../os-recommendations